Posting a Spring AI architectural pattern I just shipped, because it solves a real problem most production AI teams hit eventually.
Setup: I have a code review service using Claude Opus 4.7. Most requests are trivial ("does this method handle null?", "is this variable named well?"). Opus is overkill for those and the price reflects it — $75 per million output tokens.
Solution: route based on query complexity. Two ChatClient beans — local Gemma 4 e2b via LM Studio for simple queries, Claude Opus for complex ones. A QueryRouter decides per request.
The dispatch is one method:
public RoutedResponse route(String prompt) {
RoutingDecision decision = router.route(prompt);
ChatClient client = (decision.tier() == ModelTier.LOCAL)
? localClient
: cloudClient;
ChatResponse response = client.prompt(prompt).call().chatResponse();
long[] tokens = extractTokens(response, prompt, text);
tracker.record(decision, tokens[0], tokens[1]);
return new RoutedResponse(decision, text);
}
Router rules (intentionally simple — transparent and debuggable):
Prompt longer than 500 chars → cloud
Contains one of {architecture, design, refactor, security, performance, scalability, tradeoff, compare, analyze, best practice} → cloud
Otherwise → local
Spring AI autoconfiguration handles the local client (pointed at LM Studio via Anthropic protocol). An explicit u/Configuration class adds the cloud bean by qualifier. Two ChatClient beans, different names, no conflict.
Real measurements from the demo:
- 10 queries through the router: 7 local (free), 3 cloud → $0.25
- 10 queries through Opus only: $0.48
- Same answers on the easy 7. 48% cheaper overall.
Per-query cloud costs (the three that escalated):
- Tradeoff comparison: ~$0.10 (most expensive — structured comparison runs long, 1,100+ output tokens)
- Security review: ~$0.08 (Opus enumerates everything that could go wrong with the auth flow — 1,200+ output tokens)
- Architecture review: ~$0.07 (cheapest — usually one or two real issues, gets to the point, ~800 output tokens)
The pattern: cost scales with output tokens, and output tokens scale with how much there is to say. The verbose analytical genres (tradeoff, security) cost more than the concise ones (architecture). Output is ~5x the price of input on Opus, so the response length is what moves the bill.
One observation worth flagging: the 7 routed-away queries would have cost ~$0.23 collectively on cloud, almost matching the $0.25 from the 3 cloud queries. Cheap individually, expensive in aggregate. The savings come from removing the long tail of trivial queries from cloud, not from avoiding premium prices on premium queries.
Three things that aren't obvious until you ship this:
Anthropic requires max_tokens on every request. Without it, Spring AI uses a low default and Opus responses truncate mid-sentence. Set AnthropicChatOptions.maxTokens(4096) explicitly.
Opus regularly takes 15-45 seconds per response. Spring AI's underlying Reactor Netty has a shorter default timeout — you'll get ReadTimeoutException. Pass a custom RestClient.Builder with responseTimeout(Duration.ofSeconds(300)) to AnthropicApi.builder().
Token usage metadata is reliable for Anthropic, sometimes null for local models depending on LM Studio model loadout. Build a fallback path (character/4 estimate) so the dashboard never shows mysterious zeros.
Full demo with the live cost dashboard: https://youtu.be/ziMzlY9Szvs
Repo with code: https://github.com/DmitryFinashkin/spring-ai
