Llm Applications

Retrieval Augmented Gen

Genai

AI

Ai Research

RAG Without the Vector DB: Say Hi To Multilevel LLM Routing!

Less infrastructure, fewer headaches, better results!

Building effective documentation based AI chatbots traditionally requires complex RAG (Retrieval-Augmented Generation) systems with vector databases and embeddings.

But what if there’s a simpler way?

Today, I’m exploring an alternative approach that uses multilevel LLM routing instead of traditional RAG methods.

After building SimplerLLM to help developers “like me” work with LLMs more easily, I recently added an LLM router feature and have been experimenting with it in different use cases.

Created by Author

I’ve been testing it for AI agent decision making, AI teams collaboration, and now as an alternative to RAG.

In this post, I’ll compare multilevel LLM routing with traditional RAG for documentation search, with simple prototypes to demonstrate both concepts.

The Documentation Problem

Documentation is essential but often challenging to navigate.

Many companies now use AI-powered search to help users find answers quickly. There are two main approaches:

1. Traditional RAG: Convert documentation to embeddings, store in a vector database, query using similarity search, then generate answers
2. LLM Routing: Use an LLM to intelligently route questions through a hierarchical structure of documentation

Let’s compare these approaches with simple prototypes.

Approach 1: Traditional RAG with Vector Embeddings

The traditional RAG approach involves:

1. Chunking documentation
2. Converting chunks to embeddings
3. Storing in a vector database
4. Retrieving relevant chunks
5. Generating answers using an LLM with the retrieved context

Here’s a simplified implementation:

import os
from SimplerLLM.language.llm import LLM, LLMProvider

# 1. Define our documentation chunks (in a real system, this would be automated)
documentation = [
    "# Product Overview\nOur software helps businesses automate workflows.",
    "## Features\nThe product includes dashboard, reports, and integrations.",
    "## Dashboard\nThe dashboard shows real-time metrics and KPIs.",
    "## Reports\nReports can be scheduled and exported in various formats.",
    "## Integrations\nWe support integrations with CRM, accounting, and email tools.",
    "# Installation\nThe software can be installed on Windows, Mac, or Linux.",
    "## System Requirements\nRequires 4GB RAM, 2GHz processor, and 10GB disk space.",
    "## Windows Installation\nDownload the .exe file and run the installer.",
    "## Mac Installation\nDownload the .dmg file and drag to Applications.",
    "# Support\nWe offer email and chat support during business hours.",
    "## Common Issues\nTroubleshoot login problems, data sync, and reporting errors.",
    "## Contact\nReach support at support@example.com or call 555-123-4567."
]

# 2. In a real system, we would convert chunks to embeddings and store them
# For simplicity, we'll skip the actual vector DB implementation
# and simulate retrieval with a simple keyword search

def simple_retrieval(query, docs):
    """Simulate vector retrieval with simple keyword matching"""
    relevant_docs = []
    query_terms = query.lower().split()
    
    for doc in docs:
        score = sum(1 for term in query_terms if term in doc.lower())
        if score > 0:
            relevant_docs.append((doc, score))
    
    # Sort by relevance score
    relevant_docs.sort(key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in relevant_docs[:3]]  # Return top 3 matches

# 3. Answer generation
def generate_answer(query, context_docs):
    # Initialize an LLM
    llm = LLM.create(
        provider=LLMProvider.OPENAI,
        model_name="gpt-4o-mini"
    )
    
    # Combine context
    context = "\n\n".join(context_docs)
    
    # Create prompt
    prompt = f"""
    Based on the following documentation:
    
    {context}
    
    Answer this question: {query}
    
    If the documentation doesn't contain relevant information, say so.
    """
    
    # Generate response
    return llm.generate_response(prompt=prompt)

# Usage
def rag_answer(query):
    retrieved_docs = simple_retrieval(query, documentation)
    return generate_answer(query, retrieved_docs)

# Example
# answer = rag_answer("How do I install on Mac?")

This approach works, but requires:

• Embedding generation (computationally expensive)
• Vector database setup and maintenance
• Chunking strategy decisions
• Regular reindexing when documentation changes

Approach 2: Multilevel LLM Routing

Now let’s implement a multilevel LLM router approach that avoids embeddings entirely:

from SimplerLLM.language.llm import LLM, LLMProvider

# Initialize an LLM
llm = LLM.create(
    provider=LLMProvider.OPENAI,
    model_name="gpt-4o-mini"
)

# Define our documentation hierarchy (3 levels)
# Level 1: Main categories
level1_categories = [
    "Product Overview: Information about the product, features, and capabilities",
    "Installation: How to install and set up the software",
    "Support: Getting help and resolving issues"
]

# Level 2: Subcategories (keyed by Level 1 category)
level2_categories = {
    "Product Overview": [
        "Features: Overview of all product features",
        "Dashboard: Information about the dashboard",
        "Reports: Information about report generation",
        "Integrations: Information about connecting with other tools"
    ],
    "Installation": [
        "System Requirements: Hardware and software requirements",
        "Windows Installation: Installing on Windows OS",
        "Mac Installation: Installing on Mac OS"
    ],
    "Support": [
        "Common Issues: Troubleshooting common problems",
        "Contact: How to reach the support team"
    ]
}

# Level 3: Detailed content (keyed by Level 2 subcategory)
level3_content = {
    "Features": "The product includes a dashboard for real-time monitoring, " +
                "reports for data analysis, and integrations with popular tools.",
    
    "Dashboard": "The dashboard shows real-time metrics and KPIs. " +
                 "Users can customize widgets and create multiple dashboards.",
    
    "Reports": "Reports can be scheduled and exported in various formats " +
               "including PDF, Excel, and CSV. Custom reports are available.",
    
    "Integrations": "We support integrations with CRM, accounting, and email tools. " +
                    "API access is available for custom integrations.",
    
    "System Requirements": "The software requires 4GB RAM, 2GHz processor, " +
                           "and 10GB disk space. Supported OS: Windows 10+, macOS 10.15+.",
    
    "Windows Installation": "Download the .exe file from our website and run the installer. " +
                            "Follow the on-screen instructions to complete installation.",
    
    "Mac Installation": "Download the .dmg file from our website and drag the application " +
                        "to your Applications folder. Open to start using.",
    
    "Common Issues": "Common problems include login issues (try clearing cookies), " +
                     "data sync delays (check internet connection), and report errors " +
                     "(verify input data format).",
    
    "Contact": "Reach support at support@example.com or call 555-123-4567. " +
               "Our hours are Monday to Friday, 9am-5pm Eastern Time."
}

# Multilevel LLM Router implementation
def route_query(query):
    # Level 1: Route to main category
    prompt_l1 = f"""
    Based on the user's question: "{query}"
    
    Select the most relevant category from the list below:
    
    {"\n".join(level1_categories)}
    
    Return only the name of the category (before the colon).
    """
    
    category = llm.generate_response(prompt=prompt_l1).strip()
    
    # Level 2: Route to subcategory
    if category in level2_categories:
        subcategories = level2_categories[category]
        
        prompt_l2 = f"""
        Based on the user's question: "{query}"
        
        Select the most relevant subcategory from the list below:
        
        {"\n".join(subcategories)}
        
        Return only the name of the subcategory (before the colon).
        """
        
        subcategory = llm.generate_response(prompt=prompt_l2).strip()
        
        # Level 3: Generate answer based on content
        subcategory_clean = subcategory.split(":")[0].strip()
        
        if subcategory_clean in level3_content:
            content = level3_content[subcategory_clean]
            
            prompt_l3 = f"""
            Based on the following documentation about {subcategory_clean}:
            
            {content}
            
            Answer this question: {query}
            
            If the documentation doesn't contain enough information, say so.
            """
            
            return llm.generate_response(prompt=prompt_l3)
        else:
            return "I couldn't find specific information about that topic."
    else:
        return "I couldn't determine the relevant category for your question."

# Usage
# answer = route_query("How do I install the software on my Mac?")

Comparing the Approaches

Let’s compare these two methods across several key metrics:

1. Implementation Complexity

Traditional RAG:

• Requires setting up embedding models
• Needs vector database configuration
• Chunking strategy development
• Regular reindexing

Multilevel LLM Routing:

• Simpler implementation
• No vector databases required
• No embeddings generation
• Works with hierarchical content structures

2. Response Time

Traditional RAG:

• Time to generate embeddings for query
• Vector search time
• LLM response generation time

Multilevel LLM Routing:

• Multiple LLM calls (one per level)
• No vector search overhead
• Small routing prompts = faster responses at each level

3. Accuracy

Traditional RAG:

• Can find information regardless of structure
• May struggle with semantic understanding
• Quality depends on embedding and chunking strategy

Multilevel LLM Routing:

• Requires well-organized hierarchical documentation
• LLM understands semantic meaning at each routing step
• Highly accurate when hierarchy matches user mental model

4. Technical Requirements

Traditional RAG:

• Embedding models
• Vector database
• Storage for embeddings
• Regular maintenance

Multilevel LLM Routing:

• Only requires LLM access
• Simple content hierarchy
• No specialized infrastructure

When to Use Each Approach

Use Traditional RAG when:

• Documentation is unstructured or flat
• Content changes frequently
• Very large documentation sets
• Need to support highly specific queries

Use Multilevel LLM Routing when:

• Documentation is naturally hierarchical
• Content is relatively stable
• Structure is well-defined
• Implementation simplicity is prioritized
• Limited computational resources

Conclusion

While RAG with vector databases has become the standard approach for documentation search, multilevel LLM routing offers a compelling alternative with significantly lower technical requirements.

By using the semantic understanding capabilities of modern LLMs, we can create simpler, more maintainable solutions that perform well for many use cases.

For organizations with well-structured documentation and limited resources, the multilevel routing approach may provide a faster path to implementation with fewer maintenance headaches.

What do you think about these approaches?

Have you implemented either one?

I’d love to hear about your experiences in the comments.

If you found this helpful, consider subscribing to my newsletter where I share more insights about AI, development, and simplifying complex technologies.

Get an email whenever Hasan Aboul Hasan publishes.

Looking to learn more about working with LLMs? Check out my courses on LearnWithHasan.com, where I teach AI for beginners, including how to build with LLMs using Python.