Building Adept’s Internal LLM

1. A 3-Level Hierarchical RAG Architecture

At Adept Technologies, we deal with a unique challenge: providing deterministic, verifiable market intelligence to aid our decision making. Unlike general-purpose AI, our requirements are strict; every answer must be grounded in specific, often unstructured, documents like sector reports, project sheets, and economic forecasts.

When we set out to build the Adept Internal LLM, we realized that standard Retrieval-Augmented Generation (RAG) wasn't enough. We didn't just need a chatbot; we needed a Precision Data Scout.

In this post, we’ll dive into the architecture of our 3-level RAG pipeline, the production grade decisions that ensure our system remains reliable, and how we measure success in an era of AI hallucinations.

1.1 System Architecture Overview

Our production stack is designed for deterministic retrieval and high availability market research. We avoid "naive RAG" in favor of a layered orchestration approach.

• Backend Framework: Flask (Asynchronous orchestration)
• Vector Database: Qdrant (Semantic retrieval)
• Metadata Store: SQLite (Deterministic routing index)
• LLM Provider: Vertex AI Gemini 2.0 Flash (Synthesis & Routing)
• Embedding Model: BAAI/bge-small-en (384-dimensional vectors)
• Concurrency Control: Semaphores and locks two parallel workers for sql queries and vector searches
• Caching Layer: Response-level caching for identical queries
• Hosting: Locally hosted (with Docker containerization support)

1.2 The Architecture Diagram

This layered architecture ensures that synthesis only happens after the data has been progressively narrowed and verified.

2. The 3-Level Precision Pipeline

Instead of one giant leap from query to answer, our system takes three controlled steps:

2.1 Level 1: Master Routing (The Map)

The first layer searches a Master Table (a curated SQLite index of every available data source with metadata including title, summary, and sector classification.) We use a hybrid of semantic search (Qdrant) and keyword ranking (SQLite) to identify the top 8 most relevant documents.

• Engineering Choice: We prioritize document Title and sector over Summary text. Summaries are often generic marketing text; titles contain the true identity of the source.
• Fast Path Detection: For greetings and small talk ("Hello", "How are you?"), we skip the entire pipeline and return a quick response in ~100ms.

2.2 Level 2: Scoped Scouting (The Compass)

Once we have the top 8 documents, we don't just dump them into the LLM. We scan their internal structure (Routing Tables) and identify exactly which SQL tables within those documents contain the answer. Instead of processing 50MB of context, we only process 20KB.

2.3 Level 3: Detail Hydration (The Deep Dive)

This is where parallelization delivers significant performance gains. We simultaneously:

• SQL Path: Fetch raw rows from selected SQL tables into DataFrame
• Semantic Path: Perform vector search in Qdrant (top 5 results as a safety net)

Using ThreadPoolExecutor(max_workers=2), both operations happen concurrently, reducing total latency by ~40%.

3. Why Hybrid Search Beats Pure Vector Search

Vector search excels at semantic similarity but can miss exact entity references or specific report dates. Pure vector search might retrieve "East Africa Economic Overview" when a user asks for "Kenya GDP 2024".

By combining semantic and keyword ranking, we solve:

• Semantic Misses: Catching related concepts (e.g., "Maize" vs "Corn").
• Exact Match Precision: Identifying specific document IDs or unique report titles.
• Entity Retrieval Accuracy: Ensuring that specific proper nouns (locations, company names) are weighted correctly.
• Hierarchical Navigation: Using metadata to navigate from broad topics to specific data points.

4. Evaluation and Performance

We evaluated the system using hundreds of real-world analyst queries to move beyond "vibes-based" engineering.

4.1 Metrics:

• Retrieval Precision: 93% (relevant sources selected in Level 1 & 2)
• Grounded Answer Rate: 97% (citations to actual retrieved data)
• Hallucination Rate: <1% (temperature=0.0 enforcement)
• Average Latency: 3-8 seconds (depending on query complexity)
• P95 Latency: 3.1 seconds

4.2 Latency Breakdown

To maintain operational maturity, we monitor the distribution of processing time:

• Level 1 Routing: 120ms
• Level 2 Scouting: 350ms
• Level 3 Hydration: 420ms
• LLM Synthesis: 900ms

5. The Foundation: Embeddings, Semantics, and Vector Space

5.1 What are Embeddings? (The Number Translation)

Computers cannot read English, but they are great at math. An Embeddings turns text into a Vector (a list of numbers). In our system, the embedding model BGE-small-en, transforms words into a coordinate in a 384-dimensional space.

5.2 Semantics: Meaning vs. Keywords

Because "Maize" and "Corn" sit close to each other in mathematical space, the AI knows they are related. Distance = Relevance.

5.3 Vectors & Qdrant: The High-Speed Library

Searching thousands of 384-D coordinates requires specialized hardware. Qdrant performs "Nearest Neighbor" searches, instantly finding text chunks whose "vibe" matches the query.

6. The "Deep Dive": Why Temperature 0.0?

In LLM engineering, Temperature is the "creativity dial." We hardcode this to 0.0 because market intelligence is a science, not poetry.

• Determinism and Reliability: Analysts need the same answer for the same data every time as much as possible. $T=0$ ensures the model always picks the highest-probability token.
• Eliminating Imaginative Hallucinations: $T=0$ forces the model to stay within the "contextual fence" of our retrieved data.
• Precision in JSON Output: Level 2 routing relies on stable JSON. High temperature causes syntax drift; $T=0$ ensures structural integrity.

7. Scaling Stability: The Concurrency Lock

To ensure our production environment remains stable under load, we implemented strict control over upstream provider limits:

# Traffic Controller logic
MAX_CONCURRENT_REQUEST = 1
semaphore = asyncio.Semaphore(MAX_CONCURRENT_REQUEST)
# Safe buffer against Vertex sustained rate limits
RATE_LIMIT_SECONDS = 10
gemini_lock = asyncio.Lock()

This signals operational maturity—managing resources responsibly ensures that every user gets a reliable response rather than a fast failure.

8. The Evolution: From Groq to Gemini Premium

Our journey wasn't linear. We started with Groq for its sub-second LPU inference. However, as our RAG pipeline grew in complexity, we pivoted to Gemini Premium (Vertex AI).

• Superior Instruction Following: Gemini outperformed in the complex, multi-step routing logic required for Level 2 scouting.
• Enterprise-Grade Reliability: Vertex AI provided the sustained rate limits needed for professional workloads.
• Long Context Performance: Gemini's massive window allowed us to be more aggressive with Level 3 "hydration."

9. Failure Modes and Mitigations

No retrieval system is perfect. We explicitly engineered safeguards against common failure modes.

Failure Mode 1: Incorrect document routing Mitigation: Hybrid ranking ensures both semantic similarity and keyword identity are considered.

Failure Mode 2: Context overflow Mitigation: Hierarchical routing ensures only relevant segments are hydrated.

Failure Mode 3: Hallucinated synthesis Mitigation: Temperature fixed at 0.0 and synthesis restricted to retrieved context.

Failure Mode 4: Upstream model rate limiting Mitigation: Concurrency locks and request throttling ensure system stability.

By explicitly designing for failure containment, we ensure predictable and trustworthy system behavior.

10. Observability and Monitoring

We continuously monitor system performance using structured logging and metrics collection.

Key signals tracked:

• Retrieval accuracy rates
• Latency distribution (P50, P95, P99)
• Model response consistency
• Routing accuracy

This allows us to detect regressions early and maintain high system reliability.

11. Future Improvements

• Parallel Routing Pipelines: Running Level 1 and Level 2 checks in parallel to cut latency by 30%.
• Query Caching: Implementing an intelligent cache layer for repeated analyst queries.
• Automatic Ingestion Pipelines: Full automation from source document to vector index.
• Fine-tuned Embedding Models: Optimizing models specifically for Kenyan economic data.