Technical Writing

Migrating live software is standard engineering; migrating a live LLM system carrying enterprise traffic is terrifying because nobody can tell you what "the same answers" means.

When building Olympus, we needed to migrate two active production agents onto the platform SDK. A big-bang rewrite or hopeful cutover was out of the question. To make the cutover provable rather than hopeful, we built a golden-set dual-run regression harness (`olympus_evalkit`).

The harness records real request/response cassettes from production workflows, executes both the legacy and platform code paths against identical inputs, and systematically diffs the structured outputs, reasoning trees, and citations.

Gating this harness in CI ensures that any commit that alters output behavior fails the build before reaching production. For AI systems, the test you actually need is not "is the output good", but "is the output the same as the thing we already trust".

In enterprise compliance and legal review, output is scrutinized by auditors and attorneys. A model generating citations as text will inevitably hallucinate a citation that looks perfectly formatted and entirely plausible — which is far more dangerous than no citation, because it is trusted.

To solve this in Risk Agent, we moved citation generation out of the model entirely. The LLM is tasked with identifying verbatim supporting quotes and evidence bounds, while the platform performs exact-text string matching against the parsed document index.

The platform then deterministically injects verified PDF-page coordinate URLs and formats output as an array of typed segments (text and citation) rather than a monolithic prose block.

This single architectural shift made hallucinated citations structurally impossible.

When our risk-evaluation volume expanded by roughly 20×, cloud spend surged abruptly. The intuitive explanation — "we are running more evaluations" — was immediately available, and completely wrong.

Root-cause analysis uncovered a compounding interaction between three separate limits: AWS Lambda concurrency caps, provider token rate limits, and Lambda’s 15-minute execution ceiling.

Under heavy load, calls were throttled by the provider. Throttled requests entered retry loops, burning wall-clock time inside the 15-minute window. When functions timed out, the entire batch re-executed from scratch, compounding inference cost on work that had already been completed.

The fix was architectural: migrating off serverless Lambda to ECS/Fargate to remove the arbitrary execution ceiling, establishing explicit queue backpressure, and controlling concurrency deliberately.

In early multi-agent setups, each agent owned its own ingestion pipeline. If a vendor uploaded a SOC 2 for Risk Agent, and later submitted a contract for Contracts Agent, the document was parsed, chunked, and embedded twice into isolated vector collections.

In Olympus, we established the principle: documents belong to tenants, not individual agents. The central Document Platform keys ingested vectors by SHA-256 content hash plus tenant ID.

This allows a single parsed and embedded document to be evaluated across multiple compliance and legal agents without re-indexing, drastically saving parsing and embedding cost.

Crucially, availability is not access: retrieval defaults strictly to the caller’s authorized tenant scope, and reading outside it requires an explicit, audited permission widening, ensuring bulletproof multi-tenant isolation.