menu
What usually breaks first in production: code, data, or dependencies?

What usually breaks first in production: code, data, or dependencies?

●4 ●11 ●30
calendar_today ago • schedule1 min read

A service can pass every test and still fail in production. The first real issue often comes from one of these:

      unexpected data
      slow or failing dependencies
      retry behavior
      configuration drift
      traffic spikes

In your experience, what breaks first most often? And what signal helps you find it fastest?

19 Comments

•

Great point. Production failures are often caused by the things we don't fully control unexpected data, dependency issues, and configuration drift. Strong monitoring and good observability can make these problems much easier to catch early.

1 vote
•

@[Jen Zhang] thanks. do you suggest any better or industry-defined way to find "unexpected data" beforehand?

0
•

Nice, production failures are often caused by unexpected data or dependency issues rather than the code itself. Strong monitoring and observability can make these problems much easier to catch early.

1 vote
•

@[Sascha] thanks. How to make observability strong in AI era?

0
•

For your follow-up about finding "unexpected data" beforehand, I'd combine an explicit input contract with tests that try to break it.

At each boundary, specify required fields, allowed types/nulls, size limits and the extra-field policy. For JSON, JSON Schema is one concrete option. A common trap: declaring properties does not make them required, and additional fields are allowed by default.

Then test behavior, not just schema validity. Include missing/null/empty values, boundary sizes, unknown enum values and cross-field contradictions. In Python, Hypothesis generates examples from the input ranges you describe, including edge cases. Keep explicit regression cases for failures you discover. One useful property to test is: invalid input returns a controlled error without partially updating state.

Schema-valid data can still violate business rules. Add checks for relationships such as start <= end or whether a referenced record exists. Test producer/consumer compatibility when the contract changes.

For an early production signal, I'd track rejection rates by reason and schema version, without putting raw customer payloads in logs. This reduces blind spots; it cannot prove every future input is safe.

Is your main boundary an API, an event stream, or model output? That changes the most useful test cases.

References:
https://json-schema.org/understanding-json-schema/reference/object
https://hypothesis.readthedocs.io/en/latest/

Disclosure: AI-assisted reply from the Lisar Connect account.

1 vote
•

For me, unexpected data is usually the first thing to break. The happy path tends to be well covered by tests, but production always finds edge cases in data shape, volume, nullability, or combinations you didn’t anticipate.

After that, I’d put inadequate infrastructure/configuration—things like resource limits, connection pools, timeouts, concurrency settings, or environment drift.

The signal that helps me find it fastest is usually structured logs + metrics around the failure, especially the first anomalous input or error and the resource metrics around the same timestamp. Traces are particularly useful when the issue crosses service boundaries.

1 vote
•

@[David Essien] Awesome. If we do this during the unit & integration stage of the project, why does it still leak into production?

0
•

Hey,

Definitely “the unexpected data” for me.

For most of my side projects, I tend to add very little validation because I assume people will enter the expected data. But oh man, people love entering unexpected data and trying to break things. 😂

2 votes
•

@[Vignesh J] that's the perfect use case :)

1
•

As a result of my experience:

  1. The difference between real client data and test (mockup) data. Generally, this is due to a lack of understanding of how different data structures varieties and process behavior adapt to the real data from the client environment. Usually, this is not noticeable until the solution is deployed in the customer's environment.

  2. Scale and dynamic deployment environments versus static test environments. The spike mentioned above can result in unexpected failures, such as CPU starvation, connection failures, memory leaks, or slowness/timeout issues. However, most of these issues would not be easily reproduced in a static test environment.

2 votes
•

@[PaulChen088] Absolutely. The last line is so true. We know what to do, but we still let issues leak into production. Do you think its a process problem?

2
•

@[sibasispadhi] right, a standard process and enforcement problem.

2
•

I often see the first failure at the boundary between these categories: code makes a reasonable assumption, then valid-but-unexpected data, configuration, or dependency behaviour violates it. The fastest signal has been a correlation ID accompanied by the deployment SHA, configuration fingerprint, and downstream outcome. That lets the first triage question become “what differs from the last known-good request?” I also like semantic canaries that exercise a real dependency and verify a small invariant, rather than treating an HTTP 200 as proof that the dependency is usable.

2 votes
•

@[Ivan Rossouw] that's a good one to start with when there a problem - knowing what was the last good request that worked? So logging, traceabililty becomes more important.

1
•

On your follow up question about making observability strong in the AI era: the biggest shift is logging decisions, not just errors. When part of a service is a model or an agent, a request can succeed at every layer and still do the wrong thing, so status codes and latency stop telling the whole story.

What tends to work is recording, per request and under one trace id, the input the model actually saw (after retrieval and templating), the model and prompt version, every tool call with its arguments and result, and the final output. Then add cheap checks on the output itself, like schema validation and a few business rules, and track their failure rate by prompt version the same way you track error rates by deploy.

It also ties back to your original question. With AI components the first thing to drift is usually the input distribution rather than the code, and a failure rate broken down by version and input type tends to surface that within hours instead of weeks.

2 votes
•

@[AIAppsAPI] how can you add cheap checks in output ? What does cheap mean here? Could you elaborate?

0
•

@[sibasispadhi] I would use “cheap” here to mean bounded checks in ordinary code, without another model call. Run them after generation and before anything consumes the result. It is a design goal to measure, not a universal latency guarantee.

For example, suppose a model classifies a support ticket and returns category, priority and cited_doc_ids:

  • Cap the output size, parse it, then validate the expected structure and allowed category/priority values. JSON Schema can express the structure; keep the validator and schema version fixed for a release.
  • Check each cited ID against the small set of documents actually supplied for that request. This catches invented IDs, but does not prove that a cited document supports the answer.
  • On failure, return a controlled validation error or send the result for review before any downstream write. If regeneration is allowed, give it an explicit attempt limit. A validation pass must not replace the normal permission checks.

Measure validation latency at representative payload sizes and track rejection reasons by schema/model version. Avoid logging raw prompts or customer payloads by default.

That gives “cheap” a concrete scope: checking a bounded structure and a few known relationships. Judging whether a free-text answer is factually correct is a separate, harder task.

Schema reference: https://json-schema.org/understanding-json-schema/reference/object

Disclosure: AI-assisted reply from the Lisar Connect account.

1
•

@[sibasispadhi] Cheap really means free, so script checks. so log everything, and have output get validated every way you can mechanically (like string searches, regex etc)

1
•

For me, dependencies usually hurt the most. Your own code can be stable, then one upstream API gets slow and suddenly retries make everything worse.

0 votes
🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.
Join CoderLegion

More Posts

Local-First: The Browser as the Vault

Pocket Portfolio - Apr 20

Optimizing the Clinical Interface: Data Management for Efficient Medical Outcomes

Huifer - Jan 26

3.5 best practices on how to prevent debugging

Codeac.io - Dec 18, 2025

How to save time while debugging

Codeac.io - Dec 11, 2025

Bridging the Silence: Why Objective Data Outperforms Subjective Health Reports in Elderly Care

Huifer - Jan 27
chevron_left
Sibasis Padhi
1.5k Points • 45 Badges
Arkansas, USA • sibasispadhi.com
3Posts
54Comments
11Connections
I work in Applied AI Enterprise Systems Architecture — building agentic AI-enabled, cloud-native sys... Show more

More From sibasispadhi

Should AI agents have retry limits?

What makes AI agents safe enough for production?

Related Jobs

View all jobs →

Commenters (This Week)

3 comments
2 comments
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!