The first time I wired a language model into a production system, the hardest part was not the model. It was the same problem that makes every integration hard: the data shape was different in staging than in prod, nobody owned the contract between the two systems, and there was no test path that used real traffic. Custom LLM integration layers two failure modes on top of that unchanged list. If you design for the model but not for the list under it, you will fail at the parts that have nothing to do with AI.
What it actually means to wire a language model in
There is a version of this that is just a feature: your code sends text, the model returns text, you display it somewhere. That is fine for what it is.
An integration is different. The model sits in a workflow. Its input comes from a data shape your system produces, which means someone owns that shape, and changes to it break the model call. Its output goes to downstream code that expects a specific format, which means when the model returns something unexpected, you need an error path that handles it cleanly, not an exception that surfaces three days later in prod.
Most of the work, when someone asks me to put a language model into their system, happens before I touch the model. What does the system produce, in the format it actually produces in prod. What does downstream code need. Where does the model sit between those two surfaces. Those questions take longer than the model call. The integration is the answers to those questions. The model call is the last piece, not the first.
A useful check: if you pulled the model out and replaced it with a person reading the same input, would the workflow still function. If not, because downstream code depends on the model's specific output quirks rather than a defined schema, the integration is already fragile. This is the same problem as any other system-to-system interface, as I found directly when wiring Claude Code access into a live infrastructure via MCP: the model's role in the workflow only becomes stable when the contract on both sides is explicit.
Why LLM integrations fail, and it is not new
Three patterns drive most failures. None of them started with language models.
The interface to the legacy system does not match what you were told it would. A field that returns ISO dates in staging returns formatted strings in prod. A schema documented in one place is not the schema the API actually uses. You find this out when real data flows.
Nobody owns the data contract. Two teams built toward the same interface from opposite sides, and when the production shape does not match what either team expected, neither takes the ticket.
There is no test path that uses real traffic. You build against fixtures through the whole development cycle, deploy, and discover that prod data has edge cases your fixtures never covered.
I have seen all three in custom ERP and back-office automation work with no AI involved at all. They are not failure modes of AI systems. They are failure modes of integration. When an LLM integration fails, these three are worth checking before you look at the model.
The two things LLM adds that integration never had before
Two failure modes need explicit design before the first real query. They do not exist in most non-AI integrations, which means they are not on a team's radar when they first do this work.
The first is nondeterminism. Structured outputs and JSON-schema enforcement at the API layer solve the format-drift problem reliably. The failure that remains is semantic drift: schema valid, fields typed correctly, validation passed, and the model returned an invented fact, a missed business rule, or a figure that looks right and is off. That does not trip a parser.
The design decision is made before the first real call: where a wrong answer has low cost, fall back to a deterministic rule and log the case. Where the decision involves money, a contract, or a compliance obligation, route it to a human-review queue. Where neither fits, refuse the operation. This failure mode is documented in my notes on agent breakdowns, where semantic drift is consistently the hardest failure to catch in testing.
The second is cost drift. Unlike a fixed-price API call, token consumption in an LLM varies with every response: the model's output length is stochastic, context windows grow as prior calls or retrieved documents accumulate, and reasoning-heavy paths can produce far more output tokens for the same input than a simpler prompt would. Standard monitoring stacks do not aggregate token use without instrumentation you add explicitly. A prompt change or a heavier load period can spike spend in ways that are invisible until the billing cycle closes. Instrumentation on the model boundary, live before you go to prod, is the only way to see what is actually happening in token spend.
Both require decisions made before the first real call. Leave them to the debugging phase and the debugging phase is very expensive.
Where the time actually goes
A working custom LLM integration for a bounded, isolated scope takes about a week. That timeline assumes access to the relevant code and data, and a contact who can confirm what the production data contract actually is. When those conditions are not yet in place, the timeline stretches in proportion to how much of that discovery still needs to happen.
Here is what the week looks like when those conditions are met.
Day one is the data contract. What the system produces, in the format it actually produces in prod, with the edge cases already visible. What downstream code needs. That contract is written out before anything is built. It is also how the three classic failures surface early: if the interface to the legacy system does not match what you were told, you find it on day one, not after the model is in the workflow.
Mid-week is the legacy interface. If the data lives in a system I need to read from, I build that layer first, separately from the model call. The model call goes in last. The reason is the same: if the interface has a problem, I want to find it before a nondeterministic component sits on top of it.
By end of week the model is in the workflow. Output schema enforcement runs at the call site. Token instrumentation is live. A test path sends real prod data through a staging copy of the integration. That test path gets handed over with documentation and instructions for running it without me.
The week is ordered this way because each failure mode surfaces earliest at the layer where it lives. Data-contract work goes first because that is where the three classic failures are found. Legacy interface work follows because problems there are cleanest when isolated. The model goes in last for the same reason: a nondeterministic component on top of a broken interface produces failures that are hard to attribute.
What to ask before you hire anyone for this
Five questions. If the person you are talking to cannot answer these specifically, they will bill you for a proof-of-concept and leave.
One: how do you handle the data contract. Not "we write documentation." How do you establish it, who confirms it, and what happens when prod violates it.
Two: how do you handle nondeterminism in the output. What enforcement runs at the call site, and what does the error path look like when the model returns something outside the expected schema.
Three: what is the test path in prod, and do I have access to it when the engagement ends. I want something I can run myself against real traffic, in test mode on the real system, after you leave.
Four: how do you instrument token use. What monitoring is in place before the integration goes live, and what does it take for me to see cost broken down by call path or prompt version.
Five: who owns the code when you leave. The codebase, the test path, the instrumentation, running in my own infrastructure, readable and modifiable by my team without your involvement.
Those five questions map to the five failure modes above. The first covers the data-contract failure. The second covers nondeterminism. The third covers the missing test path. The fourth covers cost drift. The fifth determines whether the first four still hold three months after the engagement closes. If auditing your data contract or building the integration is the right next step, klim.expert does this work with one engineer in your codebase.
One senior engineer, your codebase, a working integration in about a week. You own every line of code. klim.expert
