Insights
Field notes from building and operating AI systems in production: what breaks, why it happens, and how to test for it.
LLM tool calling in production 5
What breaks when open-weight models call tools in production, and how to test for it.
- The model answered. We threw the answer away. A model gave complete, valid answers and our system discarded them as errors. How we found it, how we fixed it safely, and the bug we found in our own fix.
- Catalog metadata is not a health check A provider catalog listed our model as healthy with function calling while every real call failed, and a typo passed lint. Only a live probe tells the truth.
- The 400 that only happens on the tool-result turn A model handled a single tool call but returned HTTP 400 when we sent the tool result back, and looked fast because it failed fast. Test the continuation.
- The eval that scored 100% with zero tool calls A tool-calling benchmark reported a perfect score for a model that failed every request. An empty-denominator bug, and how to gate model selection properly.
- Our fallback hid an outage for weeks A cheap model failed every request, a fallback to a stronger model kept users happy, and only an unrelated code path noticed. What to monitor and how to test.
Evaluating models 10
Qualifying models and building evals and tests you can trust.
- We made five model picks in one day Four ad hoc tests gave four model picks, and a fifth fell to our own scoring. The single replay protocol we use now, and the order we read its results in.
- Half our replay sample tested the wrong prompt We rebuilt production requests for a model eval and paired nearly half with answers to a different prompt. How it showed, what we withdrew, and the guards.
- What broke when we tested 20+ models on one tool call We ran more than twenty models through one structured-decision task with a forced tool call. About half failed on liveness. The failure types, and what passed.
- Recording the bad day: replay tapes for rate limits and retries Our record and replay setup could not hold a 429 followed by a 200. What a sequence recorder needs, and how to capture failures no host gives on demand.
- Testing tool calling on Hugging Face Inference Providers A method for qualifying tool calling across Hugging Face Inference Providers: the endpoint, provider pinning, fixture design and the traps. No measured results.
- A stub LLM that always returns clean JSON hides your parsing bug Our tests passed because the stub returned tidy JSON and the check only looked at the status code. Real models add fences and prose. Test the dirty case.
- Record and replay for LLM tests Record against live APIs once, replay in CI for fast, cheap, deterministic tests. Plus a static test that stops placeholder values reaching the model as facts.
- Using an independent LLM judge, and its limits A method note on LLM-as-judge: keep the judge independent, keep samples honest, treat a past production answer as a baseline and not as truth.
- Grade the tool call, not the text Our first benchmark run scored zero for every model because the grader read message text while the answer was in tool-call arguments. How to grade it right.
- How to qualify an LLM router model: hard gates before cost Sorting candidate models by price first surfaces broken ones. Disqualify on failures, then rank on cost and latency. The gates, the process and the traps.
Changing models and providers 6
When a provider retires your model: replacements, other hosts for the same weights, and routing.
- Open weights are not open availability A provider retired a model that worked, its successor failed our tests, and so did about half the alternatives. What running open models in production costs.
- Our provider retired a model. The API said nothing. An inference provider retired our production model with nine days of notice, posted only on a changelog. How we missed it, and what we watch now.
- The newer model was worse for our job We tried to replace DeepSeek V3.2 with its V4 Flash successors on one provider and lost a working confidence threshold. What to test before a swap.
- Same model ID, different hosts, different answers After a retirement we found the same open weights on other hosts. Through OpenRouter, some of those hosts behaved like a different model on our task.
- Pinning providers on OpenRouter: what we verified How provider order and allow_fallbacks behave on OpenRouter, the shared rate limits behind a pin, and the checks that keep a pinned route honest.
- SSE keep-alive comments broke our stream parser Slow streams on OpenRouter open with comment lines before any data. Our reader took them for broken JSON and reported the host as failed. Fix and tests.
Agent design patterns 5
General patterns for agent loops, tools and tiered models.
- MCP servers versus an internal tool bridge What an MCP server can replace between an agent and your app, what we kept on an internal bridge and why, and a safe way to migrate one tool category at a time.
- Which tools do you show the model? Sending every tool definition costs prompt tokens on each call. Showing a subset saves them, but keyword routers and selector models each have blind spots.
- Cheap router model plus stronger fallback: design and blind spots A tiered LLM setup saves money until the fallback hides failures or quietly spends more. Design rules for tiers, and the blind spots that break them.
- Confirmation-gated write tools and idempotency Pause agent write actions for user approval and make retries safe: race conditions, declined and expired approvals, and idempotency keys against double writes.
- A bounded agent loop that fails safely Iteration caps, typed tool errors, a final tool-less summary step and cost budgets that fail closed: stop an agent loop from running away or leaking output.
LLM operations 5
Cost tracking, alerting, monitoring and reliability for LLM features.
- The retry that could never run We built a deadline-aware retry for rate-limited LLM calls. On our main path the callers' deadlines were too short for it to start. List the callers first.
- Built but never wired: monitoring that had no callers Our quality scoring and request logging were finished and read by dashboards, but no production code called them. How an audit found it and what to test.
- Your anomaly detector is relative, so it misses slow ramps Baseline-relative cost alerts stay quiet when spend climbs slowly, and they can be noisy at low volume. Add absolute limits, floors and a provider health check.
- Cache-aware LLM cost accounting Cached-token discounts are easy to model wrong. The tolerant parsing, the zero-default trap and the tests that keep per-call LLM cost honest.
- Your LLM cost tracking is probably wrong in both directions Our recorded LLM cost did not match the provider invoice. Untracked trigger calls, cached-input discounts and missing prices each skew it. How to reconcile.
Output guardrails 5
Validating, containing and caching what models return.
- Principles for caching LLM responses What belongs in an LLM cache key, what must never be shared across users, and which results must not be cached at all. General principles from real near-misses.
- Refusals that return HTTP 200 A model that refuses or has nothing to analyze still returns success. Treat a refusal as its own outcome, retry once, and keep it out of accuracy stats.
- Detecting degenerate model output Repetition loops and garbled text can burn tokens and reach users. Detect them mid-stream, cancel, retry once at lower temperature, then degrade gracefully.
- Drop, don't error: sanity checks on model output When one field of a model's answer is out of range, discard that field and count it, instead of failing the whole request. How, and when not to clamp.
- Structured output with an abstain option Models over-commit to an answer when the right response is insufficient data. Make abstain a first-class value with a required reason, and route it to a human.
Retrieval 1
Retrieval and reranking for search and RAG pipelines.