What is context optimization in AI and why does it matter?

Context optimization is the practice of keeping the context window, the information an AI model processes during a conversation, as small and relevant as possible. It matters because a cleaner context leads to more accurate, focused, and cost-effective AI responses. By using design patterns like multi-agent routing, RAG, and skill discovery, developers can ensure the model only sees the information it truly needs for a given task.

What is the difference between the Multi-Agent Routing Pattern and the Skill Discovery Pattern?

Both patterns aim to reduce context size by handling tasks in separate, smaller contexts. The Multi-Agent Routing Pattern uses an orchestrator that delegates tasks to specialized sub-agents, each running in their own context window. The Skill Discovery Pattern, on the other hand, breaks processes into mini-skills (tool or skill definitions) that the LLM can call on demand. The key difference is that multi-agent routing relies on separate agents, while skill discovery relies on tool/skill definitions invoked within the same agent.

How does RAG (Retrieval-Augmented Generation) help optimize context in LLM applications?

RAG helps optimize context by retrieving only the relevant chunks of a document instead of loading the entire file into the context window. It uses a Vector Database to store document sections as numerical representations (vectors) and a framework like LangChain or RAGflow to perform similarity searches. This way, only the parts of a document that are related to the user's query are added to the context, keeping it lean and improving the quality of the AI's response.

Context Optimization: Design Patterns for Cleaner Context

Last time, we discussed the Context Window in our "Understanding Context Window" post, where we talked about how important it is to keep a “clean” context. Now, we are going to dive into one of my favourite topics: Context Optimization.

What can be in context?

There are a few options for what can be contained within the context window. Here is the list:

System Prompts: These are the default instructions regarding how the conversation should be handled (applicable to all conversations). For example: "Be kind and answer the user with respect."
User Prompts: This refers to the input that the user adds to the conversation right now.
Prompt History: With every new prompt, the system saves the conversation history. This allows the LLM to understand what the user and the model have previously discussed and to answer questions based on that context.
Tools: These are embedded capabilities that allow the AI to perform specific functions, such as searching the web, creating files, modifying files, etc.
MCP Tools: MCP (Model Context Protocol) is a standard that allows LLM agents to communicate with each other when they need information or functionality from third-party services. For example, a Google Calendar integration might offer tools to add, delete, or modify events.
Docs: We can add documents (files) to the context so the AI understands the specific material we are discussing. For example, we might upload a resume for the AI to review.

Context engineering - design patterns

Let’s explore some design patterns that will improve our context optimization:

The Multi-Agent Routing Pattern

This is one of the most well-known patterns. Here, we give a specific functionality to an agent and assign it a specific task. In this architecture, we have two main levels:
- Orchestrator: This is the manager that controls the flow. It gets the full context and assigns each task to a specific agent.
- Specialists / Sub-agents: These are agents that specialize in something specific. They see only the relevant context for their task and run in a separate context window.

The Semantic Pattern (RAG) RAG (Retrieval-Augmented Generation) is a technique for retrieving relevant chunks of documents. The main goal here is to avoid loading full documents when parts of them are unrelated to the conversation. To do this, we need two things:
- Vector DB: These are databases that know how to save document sections by converting text into vectors (numerical representations).
- RAG / Similarity Check Framework: We need a framework like LangChain or RAGflow. These frameworks know how to convert text into vectors and search the Vector DB for the right part of the document.

The Skill Discovery Pattern A "Skill" is an ability that you can add to your agent. You can split a process into mini-skills, where each skill includes instructions on when to use it. The LLM can then decide when to call a specific skill, it will use it in a separate context and then return to the main process. It is similar to Multi-Agent Routing, but instead of a sub-agent, we use a tool/skill definition.

Conclusion

Now that you know there are ways to manipulate the context, choose one and try it out! Even better, try mixing these patterns to find your favourite workflow.

Loading...

Vibki