Insights

Field notes from building and operating AI systems in production: what breaks, why it happens, and how to test for it.

Evaluating models 11

Qualifying models and building evals and tests you can trust.

  1. Another team's winning model lost on our workload A sister product picked a new model after a careful test. We ran the same model on our own recorded requests and kept the models we had. What decided it.
  2. We made five model picks in one day Four ad hoc tests gave four model picks, and a fifth fell to our own scoring. The single replay protocol we use now, and the order we read its results in.
  3. Half our replay sample tested the wrong prompt We rebuilt production requests for a model eval and paired nearly half with answers to a different prompt. How it showed, what we withdrew, and the guards.
  4. What broke when we tested 20+ models on one tool call We ran more than twenty models through one structured-decision task with a forced tool call. About half failed on liveness. The failure types, and what passed.
  5. Recording the bad day: replay tapes for rate limits and retries Our record and replay setup could not hold a 429 followed by a 200. What a sequence recorder needs, and how to capture failures no host gives on demand.
  6. Testing tool calling on Hugging Face Inference Providers A method for qualifying tool calling across Hugging Face Inference Providers: the endpoint, provider pinning, fixture design and the traps. No measured results.
  7. A stub LLM that always returns clean JSON hides your parsing bug Our tests passed because the stub returned tidy JSON and the check only looked at the status code. Real models add fences and prose. Test the dirty case.
  8. Record and replay for LLM tests Record against live APIs once, replay in CI for fast, cheap, deterministic tests. Plus a static test that stops placeholder values reaching the model as facts.
  9. Using an independent LLM judge, and its limits A method note on LLM-as-judge: keep the judge independent, keep samples honest, treat a past production answer as a baseline and not as truth.
  10. Grade the tool call, not the text Our first benchmark run scored zero for every model because the grader read message text while the answer was in tool-call arguments. How to grade it right.
  11. How to qualify an LLM router model: hard gates before cost Sorting candidate models by price first surfaces broken ones. Disqualify on failures, then rank on cost and latency. The gates, the process and the traps.

Changing models and providers 7

When a provider retires your model: replacements, other hosts for the same weights, and routing.

  1. A model test is a snapshot. Hosts keep changing. Our best replacement passed thousands of calls on three hosts. It was fast only because those hosts skip a step its maker documents. So we built a trial.
  2. Open weights are not open availability A provider retired a model that worked, its successor failed our tests, and so did about half the alternatives. What running open models in production costs.
  3. Our provider retired a model. The API said nothing. An inference provider retired our production model with nine days of notice, posted only on a changelog. How we missed it, and what we watch now.
  4. The newer model was worse for our job We tried to replace DeepSeek V3.2 with its V4 Flash successors on one provider and lost a working confidence threshold. What to test before a swap.
  5. Same model ID, different hosts, different answers After a retirement we found the same open weights on other hosts. Through OpenRouter, some of those hosts behaved like a different model on our task.
  6. Pinning providers on OpenRouter: what we verified How provider order and allow_fallbacks behave on OpenRouter, the shared rate limits behind a pin, and the checks that keep a pinned route honest.
  7. SSE keep-alive comments broke our stream parser Slow streams on OpenRouter open with comment lines before any data. Our reader took them for broken JSON and reported the host as failed. Fix and tests.