Training and Serving the Claygent Model Router

Introduction

Applied Compute and Clay partnered to train a specialized model router for Claygent⌝, Clay’s agent for GTM research. Claygent has processed over 5 billion queries which range in difficulty, from simple contact lookups to more difficult deep company research.

Claygent doesn’t need to use the most capable model for every query since smaller models can be more cost effective and lower latency. We trained a specialized router to pick the right model for each task, cutting inference costs by up to 47% with minimal impact on answer quality.

Applied Compute cut our Claygent inference costs by up to 47% while maintaining quality. Their team built evals, trained the router, and served it in production on AC2. Collaborating with them felt like working with a world class research and infrastructure team, and AC2 gave us the tooling to move from experiment to production in weeks.

Jeff Barg

Head of AI at Clay

Claygent Background

Claygent is one of Clay’s highest-volume inference workloads - it has processed over 5 billion queries to date. Claygent researches the web to answer GTM questions about companies and people. A user writes a question in plain language, and Claygent answers it for every company or person in their list, from a quick classification to multi-step research

Before auto-mode, users manually selected a model for each Claygent, and every task ran on it. Now, auto-mode uses a router to dynamically select the model based on the query.

Claygent tasks typically involve web search queries. Here are a few examples of potential Claygent queries segmented into a few example categories:

• Company classification: “Based on {company_domain}, is this a B2B company that sells products or services to other businesses? Answer only Yes or No.”
• Contact lookup: “Find a professional email address for {full_name} who works at {company_name}. Return only the email address.”
• Deep company research: “Analyze {domain} for AI adoption signals. Check job postings, blog, and press releases from the last 12 months. Return adoption level, confidence, and key excerpts.”

Benefits of Claygent Auto-mode Routing

Claygent Auto-mode automatically selects the appropriate model for each query, so every task gets the right amount of intelligence. More difficult queries can be routed to more capable models, while simpler queries can use faster, more efficient models without sacrificing performance. This gives users better results on the queries that need it, while reducing latency and cost on the queries that don’t. Users get these benefits automatically, without needing to understand the model landscape or manually choose the right model for each query.

Applied Compute’s router performed at frontier quality while saving a significant amount against our default mode and our live A/B test while maintaining performance on key guardrail metrics.

Stefanie Tignor

Head of Data at Clay

Training a Specialized Query Router

Applied Compute post-trained Qwen 3.6 35B A3B. Relative to other routing policies, such as heuristic routers based on task complexity, the trained router is calibrated to GTM task difficulty, learning the nuances of which queries actually require stronger models.

We generated a synthetic set of representative Claygent queries and paired them with representative public entity data to construct Claygent-style tasks. Each task was run across candidate models, and an LLM judge scored the outputs against a correctness rubric to estimate each model’s performance. These scores were then used as training data for the router.

The trained router’s input is the enrichment query plus the list of candidate model configurations. The model’s output is a single structured object: the query category, and a predicted quality score in $[0, 1]$ for each candidate model.

We trained using GRPO with a reward that optimizes a linear combination of score prediction accuracy with category classification.

At serving time, we select a desired score operating point (e.g. 95% accuracy) by supplying a cost penalty $\lambda$, which trades off score for lower cost. Using this, we select the model that maximizes the utility between the routing candidates. Each operating point is additionally constrained so that no individual query class falls below a floor, which stops the policy from paying for its aggregate savings out of one category.

We estimate each model's average cost within a query class using reference rollout data, expressed as $\bar{C}(c, m)$, where $c$ is a query class and the model is $m$:

In the above, $C(q', m)$ is the measured cost of running candidate model $m$ on reference query $q'$, and $\mathcal{D}_c$ is the set of reference queries belonging to class $c$.

The router then balances predicted quality against the average cost for the predicted query class. The utility of a candidate model is expressed as:

Here, $\hat{s}(q,m)$ denotes the router’s predicted quality score for running candidate model $m$ on query $q$, and $\hat{c}(q)$ is the router’s predicted class for the query. The utility function is parameterized by $\lambda \geq 0$, which controls the tradeoff between predicted quality and cost. Larger values favor cheaper models. When cost is measured in dollars, $\lambda$ has units of quality score per dollar. The selected model, $m^*(q)$, maximizes this utility for a particular query:

Synthetic Example Query

Here is an example routing query using a decision-maker lookup for a fictional company, ACME Corp.

find the top most decision maker of the company and return
full name, job title, and LinkedIn URL — ACME Corp

After running Claygent searches for each of the candidate models, each finds the correct response.

This means routing to the cheapest model (5.4 Nano) is optimal. Here, the router output the following:

{
  "category": "Decision-maker / executive lookup",
  "scores": {
    "gpt-5.4-nano_medium": 0.75,
    "gpt-5.4-mini_medium": 0.85,
    "gpt-5.6-sol_medium": 0.95
  }
}

The router predicts high quality for all candidates. The predicted quality gap is small enough that the cost penalty tips the utility toward the cheapest model:

utility(Nano) = 0.75 − 7.63 × $0.00802 = 0.689   ← selected
utility(Mini) = 0.85 − 7.63 × $0.02884 = 0.630
utility(Sol)  = 0.95 − 7.63 × $0.03516 = 0.682

In this illustrative example, the router saved 77% of the downstream cost on this query by routing to Nano instead of Sol.

Offline Evaluation

We compared the trained router (AC-001) against other router policies in offline evaluations:

• Score based routers: Selects model routing based on an overall utility function (AC-001 falls in this category)
  • Qwen 3.6 35B (base): the same model before post-training, with the same prompt and structured output format as AC-001
  • GPT-5.6 Luna (no reasoning): the same prompt and structured output format as AC-001
• Heuristic based routers: Outputs a single model routing decision
  • Difficulty heuristic: a prompted binary classifier that labels queries “easy” or “hard” and routes “easy” to Mini, “hard” to Sol
  • 3-class heuristic router: a prompted GPT-5.6 Sol classifier that directly selects Nano, Mini, or Sol per query, based on prompt optimizing against the training set for query categories that have best score at lowest cost
• Deterministic policies: always route to a fixed model, such as GPT-5.6 Sol or GPT-5.4 Mini

The performance of these router policies is shown below, with different router model performance at different operating points. The performance and cost axes are shown relative to a baseline enrichment model (5.6 Sol).

The fine-tuned model out performs other router policies on offline evaluations. At the 92.5% quality point, it saves 48.2% relative to always routing to Sol.

Model Serving Optimizations

Claygent processes a high volume of queries often in large bursts, so we optimized the router’s serving configuration to handle this serving demand while keeping routing fast and inexpensive.

We optimized the router to serve more requests while meeting our latency targets. We tuned the serving topology and parallelism configurations to favor more independent replicas, allowing more requests to be processed concurrently and increasing throughput for the available GPUs.

We also tuned how requests enter the model. Under load, immediately processing each arriving prompt can lead to many small prefill batches. We introduced a short, bounded delay before prefill. Existing requests can continue decoding while new arrivals wait. Larger prefill batches spread fixed scheduling and execution costs across more requests. This is particularly useful for the router’s short outputs, where frequent prefills can account for a substantial share of serving overhead. The processing time saved can outweigh the initial delay, increasing throughput and keeping queues shorter.

Below is an example of the routing algorithm and serving configuration. Queries and metrics are approximate for illustrative purposes. The image shows example queries arriving, from prefill and decoding to model selection based on predicted quality and cost. We simulate requests using both even request spacing and a fitted poisson distribution, reflected in the request timeline. The routing graph shows prefill queues which reduce end-to-end latency via batched prefills, requests load balanced across GPUs running multiple model replicas, and alternating prefill and decode. The bottom section shows example queries, the configurable $\lambda$ cost-performance tradeoff, and the final model utility which determines the routing. Sliding the $\lambda$ parameter shows less cost sensitivity at lower values, resulting in more queries routed to Sol.

Production Performance

The model is served on the Applied Compute platform (AC2) and can route hundreds of millions of queries monthly with typical request routing in a fraction of a second at a substantially lower cost than alternative routing systems using frontier models.

In an A/B test using live Claygent production traffic, the fine tuned model’s performance reflected offline evaluations: the router reduced costs by 47% relative to the baseline of routing all requests to Sol while also maintaining performance on primary guardrail metrics.

Conclusion

When a single instruction fans out across thousands of rows, small per-query improvements in cost and quality compound quickly. By training a router to predict calibrated quality scores rather than classify difficulty, and by investing in evidence-based grading that surfaces real gaps between candidates, we built a model that significantly reduces downstream inference cost while maintaining answer quality.

Try out Claygent auto-mode⌝ on Clay today!