The Code Citadel - The Great Fire

In the previous chronicle, we confronted the impossibility of perfect consistency across distributed ledgers. We learned that the Royal Truth of the Castle has been replaced by the Merchant’s ambiguity - a world of Sagas and Compensating Transactions. Lady Catherine’s bookings now succeed reliably through eventual consistency. The Royal Travel Service handles 50,000 bookings per day. But accepting eventual consistency is not enough. We must also prepare for the inevitable: catastrophic failure.

Prologue: The Baker’s Spark #

It began on Pudding Lane.

In the early hours of September 2, 1666, a small fire started in the bakery of Thomas Farriner. In a modern, fire-resistant city, this would have been a minor incident - a handled exception. A bucket of water, a stamped-out ember, a logged error.

But London was not a modern city. It was a Monolith of Thatch and Timber. The houses were built of wood and pitch, crowded so closely together that a man could shake hands with his neighbor across the street from the upper stories. This was Tight Coupling in its deadliest form. The city was a single, combustible mass. When the spark jumped from the bakery to the Star Inn, it did not just burn the Inn; it ignited a chain reaction that would consume more than 13,000 houses, 87 churches, and St. Paul’s Cathedral.

The Royal Travel Service’s Near-Death Experience: It was a Tuesday morning, three years after moving to the City. The platform was humming along nicely, processing thousands of bookings for journeys like Lady Catherine’s York to Rome trips. Then, at 9:47 AM, a junior engineer deployed a seemingly innocent change to the Image Resizing Service - a tiny service that optimized traveler profile pictures.

The change had a bug. A memory leak. For every image processed, the service leaked 10MB of memory. Within 15 minutes, the Image Resizing Service consumed all available memory and started thrashing, using 100% CPU trying to garbage collect.

In the Keep, under Master Clerk Edmund, this would have been contained - one clerk’s mistake wouldn’t affect the others. But in the distributed City, the services were tightly coupled through synchronous calls. Here’s what happened:

9:47 AM: Image Resizing Service deployed with memory leak 10:02 AM: Image Resizing Service starts thrashing (100% CPU, out of memory) 10:03 AM: Profile Service calls Image Resizing Service, requests timeout after 30 seconds 10:04 AM: All Profile Service threads are blocked waiting for Image Resizing Service 10:05 AM: Booking Service calls Profile Service (to show traveler info), requests timeout 10:06 AM: All Booking Service threads are blocked waiting for Profile Service 10:07 AM: Route Service calls Booking Service, requests timeout 10:08 AM: The entire platform is down

In the Code Citadel, we fear this above all else: Cascading Failure. A single microservice - perhaps a humble “Image Resizer” - stalls. It consumes memory. It blocks threads. Because our services are tightly coupled, the caller waits. And the caller’s caller waits. The fire spreads up the stack until the entire platform burns to the ground.

The Aftermath: The platform was completely down for 47 minutes. During that time:

• 12,000 travelers tried to book journeys (all failed)
• Lady Catherine tried to check her existing booking (failed)
• The competitor “Swift Journeys Ltd” gained 2,000 new customers
• The King sent a furious letter threatening to revoke the charter

The irony? The Image Resizing Service was non-critical. Profile pictures are nice to have, but not essential for bookings. Yet this tiny, non-critical service brought down the entire platform.

The CTO called an emergency meeting: “We cannot let a single service failure destroy the entire platform. We must design for catastrophe.”

To survive in the distributed city, we must learn the hard lessons of 1666. We must build not just for commerce, but for catastrophe.

The Firebreak: The Circuit Breaker Pattern #

The tragedy of the Great Fire was not the spark, but the delay in response. When the fire was still small, the Lord Mayor of London, Sir Thomas Bludworth, was summoned. His advisors begged him to pull down the neighboring houses - to create a gap that the fire could not cross. Bludworth hesitated. He worried about the cost. He worried about the legal complaints of the landlords. He famously dismissed the danger, saying, “A woman might piss it out”, and went back to bed.

He failed to Fail Fast. By the time the city realized the magnitude of the threat, it was too late. The fire was moving faster than men could run.

The Software Equivalent: The Hanging Thread #

In software, the “Fire” is often Latency. Imagine your Order Service calls your Payment Service. One day, the Payment Service database locks up. It doesn’t crash; it just hangs.

1. The Order Service sends a request. It waits.
2. The thread holding that request is now blocked. It is wood added to the fire.
3. More customers click “Buy”. More threads block.
4. Soon, the Order Service has zero threads available. It stops responding.
5. The Frontend calls the Order Service, and its threads block.

The entire application freezes because of one database lock. You are Lord Mayor Bludworth, watching the city burn while debating whether to act.

The Solution: The Circuit Breaker #

The only way to stop the Great Fire was to blow up the houses. On the King’s orders, the Navy used gunpowder to demolish entire streets, creating a Firebreak - a void of fuel that starved the flames.

In microservices, we use the Circuit Breaker pattern. This is a mechanism that sits between a service and its dependencies. It watches the error rate and latency.

• Closed State (Normal): Traffic flows freely.
• Open State (The Firebreak): The Breaker notices that the Payment Service has timed out 5 times in the last 10 seconds. SNAP. It opens the circuit.
• The Benefit: Now, when the Order Service tries to call the Payment Service, the Breaker immediately throws an error (or returns a cached response). It does not wait. It does not consume a thread. It fails fast.

The payment transaction fails, yes (the house is blown up), but the Order Service survives to serve other customers who might just be browsing the catalog. The city loses a district, but the capital stands.

The Royal Travel Service’s Circuit Breaker Implementation:

After the Image Resizing disaster, the engineers implemented circuit breakers on every service-to-service call. They wrapped every call with a circuit breaker that would open after 5 failures.

The Next Incident: Two months later, the Image Resizing Service had another problem (different bug, same symptoms). Marcus, the on-call engineer, was alerted. Here’s what happened this time:

2:15 PM: Image Resizing Service starts failing 2:15 PM: Profile Service calls Image Resizing Service 2:15 PM: First 5 requests timeout (2 seconds each) 2:15 PM: Circuit breaker opens: “Image Resizing Service is down” 2:15 PM: Subsequent requests fail immediately (no waiting) 2:15 PM: Profile Service returns profiles with default images 2:15 PM: Booking Service continues working normally 2:15 PM: Platform stays up!

The Result: Travelers saw default profile pictures instead of their custom ones. Not ideal, but infinitely better than a complete platform outage. Most travelers didn’t even notice. Lady Catherine booked her journey to Rome without any issues.

After 30 seconds, the circuit breaker tried again (half-open state). The Image Resizing Service was still broken, so the circuit stayed open. After 5 minutes, Marcus fixed the bug and redeployed. The circuit breaker detected the service was healthy again and closed automatically.

Total downtime: 0 minutes (for the platform) Degraded functionality: 5 minutes (profile pictures) Bookings lost: 0

The King sent a letter: “I noticed profile pictures were missing briefly this afternoon, but the booking system continued to work flawlessly. This is the resilience I expect.”

The Lazaretto: The Bulkhead Pattern #

If Fire is the enemy of the Monolith, Contagion is the enemy of the Crowd. In 1377, the Great Council of Ragusa (modern-day Dubrovnik) faced the Black Death. While other cities relied on prayer, Ragusa relied on architecture. They established the Lazaretto - the first quarantine station in history.

They declared that no ship could enter the city harbor until it had waited 30 days (a trentino, later quarantino) on the island of Mrkan. They physically partitioned the “infected” traffic from the “healthy” population.

The Software Equivalent: Resource Exhaustion #

In a naive microservice architecture, we often share resources. We use a single thread pool for all outgoing connections. Imagine an application handles both User Logins (critical, fast) and Report Generation (slow, heavy). If a user triggers a massive report that consumes all the threads, no one else can log in. The infection of the “Reporting” module has killed the “Login” module.

The Royal Travel Service’s Resource Exhaustion Crisis:

The Booking Service had a single thread pool of 200 threads for all operations:

• Booking journeys (critical, fast)
• Generating travel reports (non-critical, slow)
• Sending notifications (non-critical, medium)
• Updating analytics (non-critical, fast)

One day, a corporate client requested a massive report: “Show me all bookings for the past year, with full itinerary details, for all 5,000 employees.”

The report generation started. It was slow - each employee’s data took 5 seconds to compile. 5,000 employees × 5 seconds = 25,000 seconds = 7 hours of work.

The report generation consumed all 200 threads. For 7 hours, the Booking Service couldn’t process any actual bookings. Travelers saw error messages. The platform was effectively down for critical operations, even though the service was “healthy” (just busy generating a report).

The Solution: The Bulkhead #

We must learn from Ragusa. We apply the Bulkhead Pattern (named after the steel partitions in a ship’s hull). We divide our resources into isolated pools.

• Pool A (10 Threads): Reserved exclusively for the Payment Service.
• Pool B (50 Threads): Reserved for the Catalog Service.

If the Payment Service goes down and consumes all 10 threads in Pool A, the 50 threads in Pool B are unaffected. The Catalog continues to load lightning fast. We have quarantined the failure. The “Payment District” is plagued and boarded up, but the “Catalog District” is open for business.

The Royal Travel Service’s Bulkhead Implementation:

The team partitioned the Booking Service’s thread pools:

• Critical Pool (150 threads): Booking journeys, checking availability
• Report Pool (20 threads): Generating reports
• Notification Pool (20 threads): Sending emails
• Analytics Pool (10 threads): Updating dashboards

Now, when the massive report request came in:

• Report generation consumed all 20 threads in the Report Pool
• The report took 7 hours to complete
• But the Critical Pool (150 threads) was unaffected
• Travelers could still book journeys normally
• The platform stayed up for critical operations

The Trade-off: The report took longer to complete (7 hours instead of potentially faster with more threads). But this was acceptable - reports are not time-critical. Bookings are.

You also implemented queue depth limits:

• Report Pool queue: Maximum 10 pending reports
• If queue is full, reject new report requests with: “System busy, try again later”

This prevented a flood of report requests from overwhelming even the isolated pool.

The Result: The platform became resilient to resource exhaustion. Non-critical operations could no longer starve critical operations. The King’s bookings always went through, even when corporate clients were generating massive reports.

The Siege Timer: The Philosophy of Timeouts #

In medieval warfare, the Siege was a test of endurance. But it was rarely indefinite. When a castle was surrounded, the Castellan and the besieging General would often negotiate a specific type of truce: The Conditional Surrender. The Castellan would promise: “If my Lord King does not arrive with a Relief Force by the Feast of St. John (30 days hence), I will open the gates.”

This was a Timeout. Without this agreement, the garrison would starve to death waiting for a savior who might never come. The Timeout forced a resolution. It set a Latency Budget on hope.

The Fallacy of Infinite Patience #

In software, developers are often optimists. We set our default timeouts to infinity or 60 seconds. In a high-traffic system, waiting 60 seconds for a response is eternity. It is the equivalent of a 10-year siege. If a user is waiting for a page to load, and the backend is waiting 60 seconds for a database query, the user has already left. The connection is held open, uselessly consuming resources.

The Royal Travel Service’s Timeout Disaster:

In the early days in the City, the service had generous timeouts:

• Service-to-service calls: 60 seconds
• Database queries: 30 seconds
• External API calls: 90 seconds

These seemed reasonable. “Better to wait and succeed than to fail fast”, the engineers thought.

But during peak season, the Partner Service (which called external inn APIs) started experiencing slowness. The inn APIs were overloaded and taking 45 seconds to respond.

Here’s what happened:

• Booking Service calls Partner Service (60 second timeout)
• Partner Service calls Inn API (90 second timeout)
• Inn API is slow (45 seconds to respond)
• Partner Service waits 45 seconds, then responds to Booking Service
• Booking Service thread is blocked for 45 seconds
• With 200 threads and 1,000 requests/minute, all threads are blocked within 12 seconds
• Platform freezes

The generous timeouts meant threads were held for too long, causing thread exhaustion.

The Solution: Aggressive Timeouts #

We must adopt the ruthlessness of the Siege Commander.

• Set tight timeouts: If the Inventory Service normally replies in 50ms, set the timeout to 200ms, not 30 seconds. If it hasn’t replied in 200ms, it is likely sick. Cut the connection.
• Retries with Backoff: If the “Relief Force” didn’t arrive, do not immediately send another messenger. Wait. Give the system time to recover. Sending 1,000 retries per second to a struggling service is like firing arrows at your own fainting soldiers.

The Royal Travel Service’s Aggressive Timeout Strategy:

The team analyzed service latencies and set aggressive timeouts:

Service-to-Service Calls:

• Fast services (Profile, Auth): 500ms timeout
• Medium services (Booking, Payment): 2 second timeout
• Slow services (Report, Analytics): 5 second timeout
• External APIs (Partner inns): 3 second timeout

Database Queries:

• Simple queries (by ID): 100ms timeout
• Complex queries (reports): 5 second timeout

Retry Strategy:

• First retry: Immediate (maybe it was a transient network blip)
• Second retry: After 100ms (exponential backoff)
• Third retry: After 1 second
• After 3 failures: Open circuit breaker

The Partner Service Incident:

When the inn APIs became slow again:

• Partner Service calls Inn API (3 second timeout)
• Inn API doesn’t respond in 3 seconds
• Partner Service times out, returns error
• Booking Service receives error in 3 seconds (not 45 seconds)
• Booking Service thread is freed after 3 seconds
• With 200 threads and 3 second timeouts, you can handle 66 requests/second before exhaustion
• Platform stays up (though some bookings fail)

The Trade-off: Some bookings failed that might have succeeded if you’d waited longer. But the platform stayed up. You could show travelers: “Some partner inns are slow to respond. Try again or choose a different inn.”

This was better than the entire platform freezing.

The Result: The mean time to detect failures dropped from 60 seconds to 3 seconds. The platform became more responsive, even under partial failure conditions.

The Plague Cross: Semantic Monitoring #

Finally, how do we know which house to blow up? How do we know which district is sick?

In 1665, during the Great Plague of London, the authorities ordered that any infected house must paint a Red Cross on the door with the words “Lord Have Mercy Upon Us.”

This was a crude but effective form of Service Discovery and Health Checking. It signaled to the rest of the city: “This node is unhealthy. Do not route traffic here.”

In our Code Citadel, our Load Balancers and Circuit Breakers must constantly look for the Plague Cross. If a service instance starts returning HTTP 500 errors (The Red Cross), the Load Balancer must eject it from the pool immediately. It must not send new requests to a dying node.

The Royal Travel Service’s Health Checking:

The team implemented comprehensive health checks for every service:

Basic Health Check (The Pulse):

GET /health
Response: 200 OK if service is alive

Semantic Health Check (The Plague Cross):

GET /health/ready
Checks:
- Can connect to database? ✓
- Can connect to message queue? ✓
- Can connect to cache? ✓
- Is memory usage < 80%? ✓
- Is CPU usage < 90%? ✓
- Are dependencies healthy? ✓

Response: 200 OK if all checks pass
Response: 503 Service Unavailable if any check fails

Your load balancer checked /health/ready every 5 seconds. If a service instance failed 3 consecutive checks, it was removed from the pool.

The Database Failure Incident:

One night, your Booking Service’s database ran out of disk space. The database became read-only. The Booking Service could read existing bookings but couldn’t create new ones.

Without semantic health checks:

• Booking Service would appear “healthy” (responding to requests)
• Load balancer would keep sending traffic
• All booking attempts would fail with cryptic errors
• Travelers would see: “Internal Server Error”

With semantic health checks:

• Booking Service detected: “Cannot write to database”
• Health check returned: 503 Service Unavailable
• Load balancer removed the instance from the pool
• Traffic routed to other healthy instances
• Travelers saw: “Service temporarily unavailable, please try again”

Your on-call engineer was alerted immediately. They expanded the disk, and the instance automatically rejoined the pool once healthy.

The Tragedy of the False Positive #

However, we must be careful. In the panic of the plague, sometimes healthy families were boarded up by mistake. An over-sensitive Circuit Breaker can trigger a Thundering Herd.

The Royal Travel Service’s Thundering Herd Incident:

During a traffic spike, one Booking Service instance became slow (garbage collection pause). The load balancer detected this and removed it from the pool. The traffic redistributed to the remaining 9 instances.

But now each instance had 11% more traffic (10% increase). This pushed them closer to their capacity limit. One by one, they started slowing down. One by one, they were removed from the pool.

Within 2 minutes, all 10 instances were marked unhealthy. The platform was down, even though the instances were actually fine—just temporarily slow.

To prevent this, we use Shedding Load - deliberately dropping low-priority requests (like “Recommendation Updates”) to save high-priority ones (like “Checkout”).

The Royal Travel Service’s Load Shedding Implementation:

The team implemented priority-based load shedding:

Priority Levels:

1. Critical: Booking journeys, processing payments
2. High: Viewing existing bookings, customer support
3. Medium: Searching for journeys, viewing recommendations
4. Low: Generating reports, updating analytics

Load Shedding Rules:

• If CPU > 70%: Reject Low priority requests (503 Service Unavailable)
• If CPU > 80%: Reject Medium priority requests
• If CPU > 90%: Reject High priority requests
• Critical requests always processed (until CPU hits 100%)

The Traffic Spike:

When the traffic spike hit again:

• Instances started rejecting Low priority requests (reports, analytics)
• This freed up capacity for Critical requests (bookings)
• Instances stayed healthy enough to avoid removal from pool
• Platform stayed up for critical operations

The Result: During the spike:

• Bookings: 100% success rate
• Viewing bookings: 100% success rate
• Searching: 80% success rate (some requests shed)
• Reports: 20% success rate (most requests shed)

This was acceptable. The King could still book his journeys. Corporate clients had to wait for their reports, but that was a reasonable trade-off.

Conclusion: The City That Burns #

The Monolith was a Castle. It was designed to be defended. It was rigid, strong, and brittle. When the wall was breached, the castle fell.

The Microservices Architecture is a City. It is designed to burn. We accept that fires will happen. We accept that the plague will come. We do not build a city that cannot fail; that is a fantasy. We build a city that fails partially.

• We use Circuit Breakers to blow up the bridges to the burning district.
• We use Bulkheads to seal off the sick wards.
• We use Timeouts to stop waiting for lost causes.

The Royal Travel Service’s Resilience Transformation:

Four years after moving to the City, the platform had experienced and survived:

• 12 major service failures
• 47 minor service degradations
• 3 database outages
• 8 external API failures
• 2 network partitions

Availability Metrics:

• Year 1 (no resilience patterns): 95% uptime
• Year 2 (circuit breakers added): 98% uptime
• Year 3 (bulkheads added): 99.5% uptime
• Year 4 (full resilience suite): 99.9% uptime

The Philosophy Shift:

The team’s mindset changed from:

• “How do we prevent failures?”
• To: “How do we survive failures?”

The team accepted that in a distributed system of 50 services, something is always broken. The question is not “if” but “when” and “how do we handle it?”

The King’s Commendation:

The King sent a letter after the platform survived a particularly nasty cascade of failures:

“Four years ago, a single service failure brought down the entire platform for 47 minutes. Today, I watched as three services failed simultaneously, yet I was still able to book my journey to Rome. The booking took 5 seconds instead of 2 seconds, and I saw a default profile picture instead of my royal portrait, but the booking succeeded. This is the resilience I expect from a service that serves the entire kingdom. The Royal Travel Service has learned to build a city that burns partially, not completely. Well done.”

By designing for catastrophe, we ensure that when the smoke clears, the city still stands.

Next in the Chronicle: The City Watch - We explore the eyes and ears of the architecture: Observability, Distributed Tracing, and the Royal Inspector of Weights and Measures.