Retrieval Augmented Generation (RAG) is a powerful technique that allows enterprises to harness the capabilities of large language models (LLMs). While it’s relatively easy to demonstrate RAG’s potential, transitioning from a demo to a production environment can be quite challenging. Let’s delve into why this is the case and how these challenges can be overcome.

The Ease of RAG Demos

RAG’s simplicity and the availability of supportive frameworks make it easy to demonstrate its capabilities.

Simple Architecture

The basic RAG pipeline consists of a few key components: a vector database, source chunking, vector matching, and an LLM interface. This simplicity makes it easy for engineers to understand and implement the architecture.

Availability of Frameworks

Frameworks like Langchain and LLamaIndex simplify the process even further. They come with built-in support for chunking, vector databases, and LLMs, allowing developers to build an RAG pipeline with just a few lines of code. The impressive language abilities of the LLMs mean that powerful demonstrations can be built quickly with RAG.

Types of RAG

One of the first things to consider when developing a RAG product for your organization is to think about the types of questions that emerge in that specific workflow and data you are building RAG for, and what type of RAG is likely to be required.

In RAG systems, we encounter two main types: simple (or naive) and complex. In practice, this is a classification of the types of questions you will have to tackle, and depending on your use case, it is likely to have scenarios where the same workflow or the same user will have both complex and simple RAG questions.

Simple RAG systems handle straightforward queries needing direct answers, such as a customer service bot responding to a basic question like ‘What are your business hours?’. The bot can retrieve a single piece of information in a single step to answer this question.

Complex RAG systems, in contrast, are designed for intricate queries. They employ multi-hop retrieval, extracting and combining information from multiple sources. This method is essential for answering complex questions where the answer requires linking diverse pieces of information which are found in multiple documents.

A multi-hop process enables RAG systems to provide comprehensive answers by synthesizing information from interconnected data points. For example, consider a medical research assistant tool. When asked a question like “What are the latest treatments for Diabetes and their side effects?” the system must first retrieve all the latest treatments from one data source or document, then make subsequent retrievals in another data source or document in the database to gather details about their side effects.

The Challenges of RAG in Production

While RAG demos are straightforward, using RAG in a production environment presents several challenges.

Correct but Not Comprehensive

While RAG can provide satisfactory responses to simpler questions, it often falls short when faced with more complex queries. Users may find the answers to be correct but not comprehensive, covering only basic aspects and not fully addressing their needs.

Source Requirements

For RAG to work effectively, the necessary knowledge sources must be added to the vector index. This requires significant effort to understand user requirements and build the index accordingly. Moreover, the index must be continually updated with current information.

Easy to Add, Difficult to Remove

Once a source is added to the index, it starts influencing the answers. This can lead to unreliable answers when multiple sources contain conflicting information.

Data Privacy

Quick RAG implementations often use an external LLM API such as OpenAI or Google. This means that an organization’s internal data is sent to the outside world, potentially creating data privacy issues.

Latency

As the index size grows, the time taken to select the right material for a query increases. This can be unacceptable for real-time use. Minimizing the number of chunks reduces latency.

Overcoming the Challenges

Overcoming these challenges involves building a more complex pipeline. Here are some techniques that can help:

• Re-ranking: This involves re-ordering the retrieved documents based on their relevance to the query.
• Knowledge Graph: A knowledge graph can be used to store and retrieve structured and semi-structured data.
• Internal Models: These are models that are trained on an organization’s internal data.
• Trust Layers: These are additional layers of verification to ensure the reliability of the information.

‍