Insights - Privorum — AI, Automation & Backend Engineering

Insights

Field notes from building and operating AI systems in production: what breaks, why it happens, and how to test for it.

Your LLM cost tracking is probably wrong in both directions LLM operations · Our recorded LLM cost did not match the provider invoice. Untracked trigger calls, cached-input discounts and missing prices each skew it. How to reconcile.
Catalog metadata is not a health check LLM tool calling in production · A provider catalog listed our model as healthy with function calling while every real call failed, and a typo passed lint. Only a live probe tells the truth.
The 400 that only happens on the tool-result turn LLM tool calling in production · A model handled a single tool call but returned HTTP 400 when we sent the tool result back, and looked fast because it failed fast. Test the continuation.
The eval that scored 100% with zero tool calls LLM tool calling in production · A tool-calling benchmark reported a perfect score for a model that failed every request. An empty-denominator bug, and how to gate model selection properly.
Our fallback hid an outage for weeks LLM tool calling in production · A cheap model failed every request, a fallback to a stronger model kept users happy, and only an unrelated code path noticed. What to monitor and how to test.