Architecture Case Studies

Three AI products at Certa (Risk, Contracts, and Data agents) had each independently built the same foundational infrastructure: separate document ingestion, separate tenant isolation, separate auth, and separate result delivery. Universal capabilities were trapped inside whichever agent invented them — deduplication in one, URL handling in another, the rule engine and eval framework in a third. Every new agent meant rebuilding the entire plumbing from scratch.

Architected a shared platform delivered as versioned libraries in a uv-workspace monorepo rather than a central runtime service, allowing consuming agents to adopt it incrementally without introducing a single point of failure. Selected a strangler migration pattern rather than a rewrite because two of the three consumer agents were already carrying live enterprise traffic.

Seven versioned Python packages: error taxonomy, tenancy, agents, notifications, documents, runtime, response — plus an eval kit. Centred on a single `run_agent()` entrypoint featuring strict JSON-schema structured output, a bounded tool-use loop, 429-aware retry and backoff, and mandatory usage/cost telemetry. Authored HLD, LLD, and infra designs, led review with staff and principal engineers, and decomposed delivery into five milestones.

M1 Foundations shipped end-to-end. Migrated Risk Agent and Contracts Agent onto the platform SDK, deleting two bespoke LLM layers. Established the core architectural principle: documents are tenant assets, not agent assets — keyed by content hash plus tenant, reusable across agents without re-ingestion, with logged access widening.

A contract-analysis pipeline was living in an n8n no-code canvas. While functional as a proof-of-concept, it had to become a robust, production-ready enterprise service before its first major customer deployment, with delivery required in weeks.

Evaluated our internal Temporal-based workflow engine as the migration target and recommended against it. While orchestration was solid, agentic RAG loops, Qdrant vector retrieval, MongoDB state, and Sentry had no native support and would collapse into opaque code steps with execution timeouts inadequate for 5–10 minute clause-family analysis. Recommended and shipped a standalone Python microservice.

FastAPI application factory with ASGI lifespan, Celery workers across 3 ECS services (web, embedding, analysis) and 2 SQS queues split by workload to scale embedding and inference independently. Asynchronous job model in MongoDB with idempotency guards and max-3 retries with 60s backoff. Structural multi-tenant isolation with custom Qdrant shard keys and per-tenant Mongo connections. Implemented complete CI quality baseline (ruff, pytest, Safety scanning, automated changelog generator).

Passed architecture review with zero blockers, cleared QA tenant-isolation validation, and deployed directly into the first enterprise client's live legal and contracting workflow. Later diagnosed obligations pipeline redundancy, collapsing 4 runs into 1 and cutting inference cost per contract by roughly 60%.

Reading a vendor's SOC 2, ISO, PCI DSS, and pentest reports against a client's control set requires 10–12 hours of skilled analyst review. Automating it is easy to prototype and very hard to make trustworthy — because the fatal failure mode in compliance is not an outright error, but a confident wrong answer paired with a plausible, hallucinated citation.

Made two structural decisions to take work away from the model: First, citation generation was moved out of the LLM into the platform via exact-text extraction and deterministic URL injection. Second, specialized individual agents by document type across 13+ standards to eliminate domain conflation (e.g. payment-card scope bleeding into privacy reasoning).

Attribute-level accuracy root-cause analysis program against human ground truth, grouping 33+ issues into 7 root-cause buckets. Evaluation framework with LLM-as-judge scoring for citation accuracy and sub-24h feedback loops. Pre-indexing validation gate cascade rejecting blank, corrupt, template, and marketing documents before burning inference tokens. V1/V2 dual-version runtime enabling both versions to serve production traffic side by side.

A single heavy run now evaluates 21 risk domains, 840 controls, and ~2,500 attributes (~8,400 model calls, 210M input tokens at 100-way concurrency) in under 10 minutes. Delivered 100% citation coverage for SOC 2 reports and established an attribute-level accuracy baseline measured against human ground truth, with root-cause analysis feeding a sub-24-hour evaluation loop.

The risk-evaluation pipeline was running slowly under heavy concurrent assessment workloads. The initial instinct across the team was that internal application code required profiling and micro-optimizations.

Reframed the core bottleneck: the pipeline was not compute-bound, it was throttled by provider rate limits. The question was not "how do we make our code faster" but "what is the provider's actual token and request ceiling under real production concurrency".

Constructed a comprehensive throughput model evaluating real production workloads against measured P50 and P99 latency and published provider TPM/RPM ceilings. Modelled multi-region routing efficiency for GPT-4.1 (~1.26M effective TPM across 6 regions at 70% efficiency) against Claude Sonnet 4.5 (30M TPM / 1,100 RPM) and Haiku 4.5 (13M TPM / 2,600 RPM).

Demonstrated that a 10-file heavy package would collapse from 12 hours to 68 minutes (10.6× speedup) and light packages by 5.5×, while concurrent tenant capacity expanded from 1 to 5–6. Provided the concrete, quantitative foundation for the platform-wide migration to AWS Bedrock.