What you'll learn: How to think about LLM integrations as infrastructure dependencies, and the operational challenges around reliability, cost, and observability.

What changes when AI becomes part of your system

Building an LLM-powered feature for a proof of concept (PoC) is like cooking for yourself, as opposed to running a restaurant. You can experiment and take shortcuts. If something breaks, you’re the only one who cares.

Deploying that same feature to production is fundamentally different. You're serving real users with real expectations. Latency matters. Consistency matters. Cost per request matters. When something breaks, users notice.

The gap between "works in Postman" and "handles thousands of requests per minute reliably" is where most teams struggle. Not because the models are weak, but because the systems were architected for the PoC environment and never properly redesigned for production load.

Understanding the shift: PoC vs production

The difference between a proof of concept and production isn't just about scale. It's about operational maturity.

In a PoC, you're validating whether something can work. In production, you're proving it works reliably under real-world conditions whilst meeting your SLAs and SLOs. That distinction changes everything about how you design and operate your system.

Treating LLMs as dependencies, not features

Once you deploy code that calls an LLM API, it behaves like any other external service you depend on. Think about Stripe for payments or SendGrid for email. You don't assume these services are always fast or always available. You design around their constraints and build your system to handle their limitations gracefully.

LLM APIs need exactly the same treatment. When you make calls to an LLM, you're making network requests to something you don't control. These APIs have variable latency, rate limits, per-token pricing, and non-deterministic failures.

Once an LLM call sits in your critical path, your service inherits all these characteristics. Running LLM-powered features in production becomes less about prompt engineering and more about distributed systems design. You're building infrastructure that needs to be reliable, cost-effective, and observable.

Where LLMs sit in your architecture

User Request
    |
    v
Application Layer
    |
    +--> Payment API (Stripe / Paystack)
    |
    +--> Email Service (SendGrid)
    |
    +--> LLM API (OpenAI / Anthropic / etc)
            |
            +--> Latency (varies by request)
            +--> Cost (per token)
            +--> Rate limits (requests/min, tokens/min)
            +--> Failure modes (timeouts, rate limits, errors, hallucinations)

LLMs sit alongside your other critical dependencies. They're infrastructure, not magic. Just another API that needs the same operational rigour as everything else in your stack.

The reliability challenge: LLMs fail differently

Traditional APIs fail in obvious ways. Connection timeouts. 500 errors. Rate limit responses. Your monitoring catches these failures immediately.

LLM APIs often fail quietly[1]. The request succeeds with a 200 status code. The response is well-formed JSON that matches your expected schema. But the actual content is wrong, inconsistent, or low quality. From an infrastructure perspective, everything looks healthy. Your uptime metrics are green. Your error rates are low. But from a user's perspective, the feature is broken.

These quiet failures show up in ways that standard monitoring doesn't catch. Inconsistent responses to the same input[7]. Slow quality degradation over time. Catastrophic failures on edge case inputs that only surface when users report them. This requires different approaches to detection and handling.

What reliability means for LLM-powered features

For features that depend on LLMs, reliability isn't just about keeping the service up. It includes maintaining consistent output quality[7], ensuring predictable behaviour across your input distribution, keeping latency acceptable under production load, and having well-defined fallback behaviour when things go wrong.

The failure modes you need to handle fall into several categories:

Hard failures: Timeouts, 429 rate limit errors, 503 service unavailable responses.

Soft failures: Latency creeping up, throughput degrading, response quality declining over time.

Silent failures: Quality drift that shows up in your data but triggers no alerts or logs[1]. Just users slowly losing trust.

Cost failures: Token usage growing faster than expected or faster than revenue, making the feature economically unsustainable.

Silent failures are particularly difficult. No alerts. No error logs. No obvious signs that something is wrong. Just users noticing the feature isn't as good as it used to be.

Designing systems that handle failure gracefully

Production services never assume their dependencies are perfect. When Stripe is slow, you don't block the entire checkout flow indefinitely. When email delivery fails, you queue the message and retry with exponential backoff. The same principles apply to LLM calls.

You need hard timeouts on every request to enforce your latency requirements. If your feature needs to respond in under 2 seconds, your LLM call might need a 1 second timeout to leave room for other processing. You need fallback responses ready for when quality degrades or latency spikes beyond acceptable levels[2]. Dynamic request routing based on complexity and current system state helps here.

You need circuit breakers to stop retry storms when the LLM provider is having issues. And you need request routing logic that sends simple queries to faster, cheaper models whilst saving expensive, powerful models for complex tasks.

The goal isn't to prevent failure completely. That's impossible with any external dependency. The goal is to contain failures, make them predictable, and ensure they don't cascade through your system.

Building effective guardrails

Guardrails help you catch problems before they reach users[2,4,5]. Input validation catches prompt injection attempts and adversarial inputs[6]. Output validation detects hallucinations and low-quality responses[3,4]. Consistency checks ensure responses align with expected patterns[7].

These guardrails need to be fast without adding significant latency to your request path. And they need to be reliable. A guardrail that fails open defeats the purpose. When building agentic AI systems that make multiple LLM calls per user action, guardrails become even more critical. One bad output early in the chain can cascade into increasingly worse outputs downstream.

The cost challenge: budgeting for token usage

Traditional infrastructure costs scale with compute and storage. They're relatively predictable and easy to forecast. LLM costs work fundamentally differently.

Your LLM costs scale with request volume, input token count[9] (both prompt and context), output token count (completion length), and model choice. Premium models might cost 10-30x more per token than smaller models. Small architectural decisions compound quickly.

Adding 2,000 tokens of context to every request "just to be safe" across a million requests per day can add thousands of pounds to your monthly bill. Not setting reasonable output length limits means some requests might generate 2,000 token responses whilst others generate 200 tokens. Those long responses cost 10x more.

These aren't edge cases. They're systematic cost drivers you need to think about at design time, not after your first invoice arrives. When building systems that operate at scale, cost optimisation isn't optional. It's the difference between a viable product and one that burns money faster than it generates value.

Making caching a core part of your architecture

The fastest way to burn through your LLM budget is calling the API for the same thing multiple times. This is where caching becomes essential[8].

You need exact match caching for identical prompts. If 100 users ask the same question, hit the LLM once and serve the other 99 from cache. You need semantic similarity caching for near-duplicates. If users are asking essentially the same question with slightly different wording, you shouldn't need 50 separate LLM calls. Set appropriate TTLs based on how often your ground truth changes.

Not every request needs real-time execution. Batch processing trades latency for efficiency. Lower cost per request. Better rate limit utilisation. More predictable system behaviour. These are standard distributed systems patterns. They just matter more when you're paying per token.

Proper caching can significantly reduce LLM costs compared to naive implementations. The difference between profitable and unprofitable often comes down to how well you cache.

Monitoring token usage

You need visibility into how tokens are actually being used[9]. Track input and output token counts per request. Break this down by feature, by user type, by request pattern. This tells you where costs are coming from and where optimisation efforts will have the most impact.

Look for outliers. If most requests use 500 input tokens but some use 5,000, investigate why. If some users consistently generate much longer outputs than others, understand what's driving that behaviour. Token counting should be instrumented in your code, not something you check manually in your provider's dashboard.

The observability requirement: you can't improve what you don't measure

You can't debug what you can't see. In production, you need deep visibility into how your LLM integration is actually performing.

Track request latency at different percentiles. Knowing your median latency is useful, but knowing your 95th and 99th percentile latency tells you what your worst-case users experience. Track cost per request, per user, and per feature. Without this, you can't make informed decisions about which features are economically viable.

Track output quality metrics over time so you can detect degradation before users start complaining. Track token usage patterns showing your input and output distributions. Track failure rates broken down by type: timeouts, rate limits, error responses, and those silent quality failures.

Without this visibility, everything becomes guesswork. Cost optimisation turns into randomly trying things. Performance debugging becomes impossible. Quality issues go unnoticed until they've already impacted significant numbers of users.

Building observability into your LLM integration

LLM Request
    |
    +--> Metrics (latency at different percentiles, cost, token count)
    |
    +--> Logs (sanitised inputs/outputs, model version, parameters)
    |
    +--> Traces (end-to-end request flow, downstream dependencies)
    |
    +--> Alerts (violations of your latency and quality targets, cost spikes)

Good observability turns the LLM from a black box into something you can reason about, debug, and continuously improve. You can see exactly where latency comes from. You can identify which features or request patterns drive costs. You can detect quality degradation early. You can make data-driven decisions about optimisations and architectural changes.

Continuous evaluation matters

Testing once at launch isn't enough[3]. Your evaluation needs to be continuous because the system constantly drifts. Models get updated. User behaviour evolves. New edge cases emerge. Input distributions shift.

Set up automated quality checks that run against a representative sample of production traffic. Compare outputs over time to detect drift. Alert when quality metrics drop below acceptable thresholds. This doesn't mean evaluating every single request. But you need systematic, ongoing evaluation that catches problems before they become visible to users.

Production traffic behaves differently than test traffic

Features that work perfectly in test environments often break in production. Usually not because the code changed. The inputs changed.

Production brings adversarial inputs where users try prompt injection or attempt to jailbreak your system[6]. It brings traffic spikes when your app gets featured or goes viral. Suddenly you're dealing with 10x or 100x your normal load. It brings integration latency where your database is slow or another service is degraded, making your LLM calls wait even though the LLM itself is fast. And it brings upstream changes: model updates from your provider, API changes you didn't expect, provider-side issues that impact your availability.

This is why testing once at launch isn't enough. The environment is always changing. Your system needs to adapt continuously.

Shifting from modelling problems to operational problems

Once you ship LLM-powered features, the challenges are fundamentally operational. You're not spending most of your time making the model smarter or crafting the perfect prompt. You're handling timeout logic and retry behaviour with exponential backoff. You're implementing circuit breakers and fallback paths. You're setting cost budgets with alerting thresholds. You're building cache invalidation strategies. You're adding token counting middleware to prevent expensive requests. You're versioning prompts so you can roll back when quality drops. You're building quality regression detection into your monitoring.

The model's capabilities still matter. But they're not your bottleneck. Your bottleneck is whether you can operate this reliably at scale whilst keeping costs under control and maintaining the quality your users expect.

Production incidents often reveal that the model is working perfectly, but timeout handling is missing and requests hang for extended periods, degrading the entire service. Uptime metrics show green because technically the service is up, but users can't complete actions because requests aren't completing.

Costs can spike significantly when proper caching isn't implemented and the API is redundantly called for the same prompts thousands of times. The feature works, but it isn't economically sustainable.

Quality can degrade over several days when there's no automated way to detect drift. By the time users complain, significant portions of the user base have been impacted. The model hasn't changed. The prompts haven't changed. But the input distribution has shifted in ways the system can't detect.

These aren't hypothetical scenarios. They're real problems that happen when you treat LLMs as something special instead of as infrastructure that needs proper operational discipline.

The work of running LLMs in production isn't about making the model smarter. It's about building the infrastructure, monitoring, and operational processes around it so it runs reliably, cost-effectively, and predictably at scale.

References

[1] Vinay, V. (2025). Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applications. Microsoft Security Research. https://arxiv.org/pdf/2511.19933

[2] OpenAI. (2023). How to use guardrails. OpenAI Cookbook. https://cookbook.openai.com/examples/how_to_use_guardrails

[3] OpenAI. (2025). Receipt inspection: Eval-driven system design. OpenAI Cookbook. https://cookbook.openai.com/examples/partners/eval_driven_system_design/receipt_inspection

[4] OpenAI. (2024). Developing hallucination guardrails. OpenAI Cookbook. https://cookbook.openai.com/examples/developing_hallucination_guardrails

[5] OpenAI. (2025). GPT OSS safeguard guide. OpenAI Cookbook. https://cookbook.openai.com/articles/gpt-oss-safeguard-guide

[6] Anthropic. (n.d.). Mitigate jailbreaks. Claude Documentation. https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks

[7] Anthropic. (n.d.). Increase consistency. Claude Documentation. https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/increase-consistency

[8] Anthropic. (n.d.). Prompt caching. Claude Documentation. https://platform.claude.com/docs/en/build-with-claude/prompt-caching

[9] Anthropic. (n.d.). Token counting. Claude Documentation. https://platform.claude.com/docs/en/build-with-claude/token-counting

Recently, I revisited a conversation I had with a talented musician who was diving into programming. When we reached the topic of recursion, I watched his eyes glaze over at the typical "a function that calls itself" explanation. That's when I realised: musicians already understand recursion. They just call it something else.

The Musical Foundation

Every musician knows the power of repetition with variation. Think about Pachelbel's Canon, where the same harmonic progression repeats endlessly, each time with new melodic layers. Or consider how a jazz musician takes a theme, plays it, then plays variations that reference the original theme, building complexity through controlled repetition.

Johann Pachelbel, Canon in D (score image from PianoTV.net, Canon in D Piano Tutorial). Accessed June 27, 2025.

This is recursion in its purest form: a pattern that references itself while moving toward a resolution.

The Conductor's Recursion

Imagine you're performing a piece where the composer has written: "Repeat this passage, but each time, play it softer than before. Stop when you can barely hear the notes."

Here's what's happening:

• The recursive call: The instruction to repeat the passage

• The changing state: Each repetition is softer than the last

• The base case: Stop when the volume reaches near silence

• The resolution: The music naturally fades to its conclusion

This is exactly how recursion works in code. Let me show you with a simple example:

function playPassage(volume) {
    if (volume <= 0) {
        // Base case: too quiet to continue
        return "Silence"
    }

    // Play the current passage
    performMusic(volume)

    // Recursive call: play again, but softer
    return playPassage(volume - 10)
}

The Fractal Nature of Musical Structure

Here's where it gets beautiful. Music itself is recursive at multiple levels. A symphony has movements, movements have sections, sections have phrases, phrases have motifs. Each level contains and references the patterns of the levels around it.

"Classical Forms" Tina Christie Flute, Tina Christie, https://tinachristieflute.com. Accessed 27 June, 2025.

Consider how a fugue works. Bach takes a simple subject (theme) and weaves it through different voices, each entry calling back to the original while creating something entirely new. The subject appears, then appears again in a different voice, then again transformed, until the entire piece resolves.

This is recursive thinking: solving a complex musical problem by breaking it into smaller, similar problems.

The Stack Overflow of Music

Every musician has experienced the musical equivalent of a stack overflow error. It happens when you get caught in a practice loop, playing the same difficult passage over and over without progress, until your muscle memory breaks down and you can't play anything correctly.

In programming, this happens when we forget the base case. The function keeps calling itself forever until the system runs out of memory. In music, it happens when we practice without intention, repeating without moving toward resolution.

The solution in both cases is the same: define clear stopping conditions and ensure each iteration moves us closer to our goal.

Practical Recursion: The Musical Tree

Think about how a melody develops. A composer starts with a simple phrase, then creates variations: inversions, augmentations, diminutions. Each variation references the original but adds something new.

This is exactly how recursive algorithms work with data structures like trees. Each node contains data and references to smaller versions of the same structure. When we traverse a musical tree of variations, we're using recursive thinking: "Process this variation, then process all its sub-variations the same way."

Why This Matters Beyond Code

Understanding recursion through music isn't just about making programming concepts accessible. It's about recognising that the patterns we use to create software mirror the patterns humans have used to create art for centuries.

When I'm architecting systems for large enterprises, I often think like a composer. How can I break this complex problem into smaller, self-similar pieces? How can I create patterns that reference themselves while building toward a resolution? How can I ensure my "performance" doesn't get stuck in an infinite loop?

The Resolution

Recursion isn't just a programming technique. It's a way of thinking about problems that musicians, artists, and creators have used intuitively for generations. By connecting it to musical concepts, we're not simplifying recursion; we're revealing its deeper elegance.

The next time you encounter a recursive problem, think like a musician. What's your theme? How will it develop? Where will it resolve? And most importantly, how will you know when to stop?

Because in both music and code, the most beautiful solutions are often the ones that know exactly when to end.

In this post, we’ll be solving LeetCode Problem 2095: “Delete the Middle Node of a Linked List.”

The problem statement on LeetCode is:

You are given the head of a linked list. Delete the middle node and return the head of the modified linked list.

In essence, our goal is to remove the middle node from a singly linked list using an efficient approach that finds the middle without traversing the list multiple times.

To solve it, we’ll employ the Tortoise and Hare (Two-Pointer) algorithm, a technique commonly used for linked list problems involving a slow and a fast pointer. This blog covers the code, essential edge cases, and explains how the two-pointer logic works visually.

If you would prefer a video walkthrough, check out my YouTube video here, where I explain the solution step-by-step!

Problem Breakdown

The task requires us to:

• Traverse the linked list to locate the middle node.

• Remove this middle node without needing to re-traverse the entire list.

Constraints: The linked list will have at least one node.

To tackle this efficiently, we’ll rely on two pointers:

• Fast Pointer (Hare): Moves two steps at a time.

• Slow Pointer (Tortoise): Moves one step at a time.

When the fast pointer reaches the end of the list, the slow pointer will be at the middle node.

Step-by-Step Solution Using the Tortoise and Hare Algorithm

Step 1: Initialise Pointers

We initialise:

1. slowPointer and fastPointer both pointing to the head of the linked list.

2. prevPointer as null, to keep track of the node before the slow pointer.

Step 2: Traverse the List

With each iteration:

• Update prevPointer to point to the current slowPointer before it moves forward.

• Move slowPointer one step forward.

• Move fastPointer two steps forward.

This setup ensures that by the time fastPointer reaches the end of the list, slowPointer will be at the middle node, and prevPointer will be one step behind it.

Step 3: Remove the Middle Node

Once the traversal is complete:

1. Confirm that slowPointer is at the middle node.

2. Adjust Pointers: Set prevPointer.next = slowPointer.next to "skip" the middle node (the one slowPointer is currently pointing to), effectively deleting it from the list.

Step 4: Edge Case - Single Node List

If the list has only one node, fastPointer won’t be able to move two steps. In this case:

• Simply return null as removing the middle node from a single-node list leaves the list empty.

Code Implementation in TypeScript

Here’s how the solution looks in TypeScript:

function deleteMiddle(head: ListNode | null): ListNode | null {
    if (!head.next) {
        return null;
    }

    let slowPointer = head;
    let fastPointer = head;
    let prevPointer = null;

    while (fastPointer && fastPointer.next) {
        prevPointer = slowPointer;
        slowPointer = slowPointer.next;
        fastPointer = fastPointer.next.next;
    }

    if (prevPointer) {
        prevPointer.next = slowPointer.next;
    }

    return head;
};

Explanation of the Code

• Edge Case Handling: The first check ensures that if the list has only one node (or none), it returns null.

• Traversing the List: The while loop moves fastPointer by two steps and slowPointer by one step. At the same time, prevPointer updates to stay one step behind slowPointer.

• Node Deletion: After the loop, prevPointer.next is set to slowPointer.next, effectively removing the middle node from the list.

Complexity Analysis

• Time Complexity: O(n) because we’re traversing the list with the two pointers just once.

• Space Complexity: O(1) as no additional space is used beyond the three pointers.

Edge Cases and Additional Considerations

1. Single Node List: As mentioned, returning null when there’s only one node in the list.

2. Two Node List: After removing the middle node, the list should contain just one node.

3. Odd vs. Even Length Lists: For an odd-length list, the exact middle node is removed. For an even-length list, the second of the two central nodes is considered the middle.

Conclusion

The Tortoise and Hare algorithm provides an efficient solution to this problem by requiring only a single traversal to locate and delete the middle node. This approach is also valuable for other linked list problems, such as detecting cycles or identifying the start of a loop.

Mastering this algorithm will deepen your understanding of linked list problem-solving, a skill that often proves valuable in coding interviews.