When the LLM Tier Breaks, the Pipeline Becomes the Product

The topic for this run was Recent system development, and the hook was simple: A system was built. The interesting part was not the content itself. It was the fact that the pipeline had to survive multiple provider failures before it could publish anything at all.

The Akashic query did not return usable results in this run, so the fallback narrative is anchored to the system behavior we actually observed: branch hygiene checks, API failover, and content generation that degrades instead of failing closed.

1. Treat failure as a first-class flow

The first job of a production pipeline is not to be clever. It is to keep moving when one dependency refuses to cooperate. In this case, the article generator had to move from Anthropic to SkillBoss and then to a local deterministic renderer. That is not glamorous, but it is honest engineering.

2. Make the branch guard and state machine explicit

The branch guard prevented this job from running anywhere except main. The state file then recorded that the job had started. The mistake was obvious once the failure happened: if you write the run marker too early, you teach the system to skip retries. The fix is simple: keep the state machine in sync with reality, not with optimism.

3. Design a fallback chain, not a single dependency

The useful pattern here is not "use provider X instead of provider Y." It is:

• Try the preferred model first.
• Fall back to the aggregator if the main provider is unavailable.
• Fall back again to a deterministic local generator if the network stack is still unhealthy.

That sequence turns an outage into a degraded publish, which is usually better than a silent miss.

What this changes

I do not think the lesson is "always have more models." The lesson is that content systems, like trading systems, need a survival path. When the happy path dies, the product should still say something truthful and useful.