With the rise of large language models such as ChatGPT and DeepSeek, people can now interact with AI through fluent conversations. Yet even the most advanced LLMs still face a “memory dilemma”: they can only recall knowledge seen during training, and that knowledge may already be outdated by the time the model is used. In real-world applications, especially when handling domain-specific questions, LLMs may generate responses that sound plausible but are factually wrong.
This “knowledge freezing” problem limits the value of LLMs in enterprise applications. So how can LLMs gain better “memory” and support dynamic knowledge updates and precise retrieval?
The answer lies in two key technologies: RAG (Retrieval-Augmented Generation) and vector indexing.
RAG: Connecting LLMs to a Knowledge Base
RAG, or Retrieval-Augmented Generation, combines external knowledge retrieval with AI generation. By retrieving relevant information before generating a response, RAG helps reduce the limitations of static model knowledge and hallucinations, making answers more accurate and grounded in real-time information.
With RAG, an LLM gains a form of “dynamic memory.” It can access the latest information, provide more accurate responses for specific domains, and reduce the risk of hallucination.
Why RAG Needs Dedicated Vector Indexing
In AI, multimodal data such as text, images, and audio can be converted into high-dimensional vectors, or embeddings. The “distance” between these vectors in high-dimensional space represents semantic similarity.
However, as data volume grows, similarity computation in high-dimensional space faces the so-called “curse of dimensionality.” This causes compute costs to grow rapidly and makes brute-force comparison increasingly impractical.
Vector Indexes: The “Memory Retrieval Engine” for LLMs
A vector index is an indexing structure designed specifically for high-dimensional vector data. Its core function is to quickly find the top kkk vectors in a feature database that are most similar to a query vector. Because this involves high-dimensional vector data, implementation is much more complex than traditional data queries.
In simple terms, vector indexing trades a small amount of precision for a much larger performance gain. Its core evaluation metric is recall, which measures the proportion of relevant results that can be correctly returned under a given query condition.
Today, two mainstream algorithms are widely used in vector indexing: HNSW (Hierarchical Navigable Small World) and IVFPQ (Inverted File with Product Quantization). Each has its own characteristics and is suitable for different scenarios:
The Key Role of Vector Indexes in RAG
In the RAG framework, vector indexing plays the key role of a “memory retrieval engine.” The process consists of five steps:
- Knowledge preprocessing
- Vector storage and indexing
- Semantic retrieval
- Knowledge augmentation
- Generation of fact-based answers
Vector indexes use approximate nearest neighbor (ANN) algorithms to optimize query efficiency and reduce the burden of high-dimensional computation. This allows RAG systems to respond to user queries in milliseconds and provide LLMs with real-time, precise external knowledge support.
How StarRocks Vector Indexes Help Models Enhance Memory
As a high-performance analytical database, StarRocks has supported vector indexes, including HNSW and IVFPQ, since version 3.4. These indexes can be used for efficient approximate nearest neighbor search (ANNS).
StarRocks vector indexes are built on Segment files and record mappings between search items and data row numbers. By quickly looking up index files, StarRocks can directly locate the corresponding data rows and avoid brute-force vector distance computation.
StarRocks vector indexes provide the following core capabilities:
- Multiple index types: Native support for the two mainstream indexing algorithms, HNSW and IVFPQ.
- Flexible parameter configuration: A rich set of parameter options to meet the needs of different scenarios.
- SQL-friendly interface: Vector creation and queries through standard SQL syntax, reducing the learning curve.
- Integration with analytics capabilities: Combined with StarRocks’ OLAP capabilities, vector data and traditional data can be analyzed together.
StarRocks Performance Optimization Strategies
To improve vector retrieval performance, StarRocks implements several optimization strategies:
- Layered caching mechanism: IVFPQ indexes support a block-based flexible caching strategy. Users can adjust the cache ratio of index blocks to balance memory cost and compute latency. HNSW indexes use a full-file caching mechanism to ensure high-performance graph traversal.
- Pre-sorting and incremental index construction: Pre-sorting is performed at the Tablet layer, and incremental index construction is supported. This reduces random read operations and improves read/write performance.
- Adaptive dynamic parameter tuning: SQL hints allow users to dynamically adjust parameters for different queries, such as the
efsearchparameter in HNSW, further balancing performance and retrieval accuracy.
Tencent Big Data’s Vector Retrieval Practice Based on StarRocks
In LLM and large-scale data analytics scenarios, Tencent Big Data faced challenges such as a complex multi-component architecture, long call chains, high latency, and high resource costs. The original system had to maintain multiple databases, and after data went through multiple processing layers, TOP 10,000 query latency exceeded 15 seconds with high resource consumption.
To address these challenges, the team built an integrated vector database solution based on StarRocks. This architecture extends high-concurrency vector query services on top of StarRocks MPP, enabling unified storage and compute for vectors and traditional data. It simplifies the query process, consolidates multi-stage processing into one-stop queries, and can adaptively choose HNSW or IVFPQ indexes.
After introducing StarRocks vector indexes, the team achieved significant results:
- The system was simplified from multiple systems, including MongoDB, Elasticsearch, and Redis, into a unified OLAP engine.
- Query latency dropped from 15 seconds to 2 seconds, improving efficiency by more than 7x.
- Operating costs were reduced to one-third of the original level.
- A unified SQL interface significantly reduced development and maintenance costs.
StarRocks vector indexes show strong advantages in processing large-scale vector data, especially in balancing cost and performance, making them suitable for enterprise applications.
Click to learn more about the Tencent Big Data case.
Application Scenarios: Giving LLMs Enterprise Knowledge Memory
Scenario 1: Enterprise Knowledge Base
The combination of StarRocks vector indexing and RAG provides a new solution for building intelligent enterprise knowledge bases.
Applicable scenarios:
- Internal enterprise knowledge bases, including document search and FAQ
- Q&A for professional fields such as legal, finance, and healthcare
- Code search and software development documentation queries
Implementation approach:
- Document embedding: Use models such as DeepSeek to convert enterprise knowledge bases, FAQs, technical documents, and other data into vectors.
- Storage and indexing: Store vectors in StarRocks and build indexes with HNSW or IVFPQ.
- Retrieval-Augmented Generation: The user enters a question → a query vector is generated → StarRocks performs vector matching → relevant documents are returned → the LLM generates the final answer based on the documents.

Scenario 2: Intelligent Customer Service and Q&A Systems
StarRocks vector indexing also provides strong support for intelligent customer service systems.
Applicable scenarios:
- Intelligent customer service for banking, securities, and e-commerce
- Professional consulting in legal and medical scenarios
- Automated Q&A for technical support
Implementation approach:
- Customer conversation log embedding: Convert historical customer service conversations into vectors.
- Storage and indexing: Use vector indexes to help the customer service system efficiently find similar cases.
- Retrieval-Augmented Generation: Generate answers based on historical customer service conversations, the knowledge base, and the LLM.
RAG-enhanced execution flow for intelligent customer service systems:

This vector index-based RAG solution significantly improves the answer quality of intelligent customer service. It is especially effective when handling enterprise domain-specific knowledge, where it can provide more accurate and relevant responses and reduce cases where the system says “I don’t know” or provides incorrect information.
Conclusion
Vector indexing allows AI to move beyond the fixed knowledge learned during training and dynamically access external knowledge bases, enabling a “knowledge as a service” capability. StarRocks’ vector indexing implementation provides a cost-effective path for enterprise deployment.
In the future, vector indexing performance will continue to be optimized, with support for freshness retrieval, upward retrieval, hybrid retrieval, multi-path recall and custom sorting, text preprocessing, and other capabilities. These improvements will provide enterprises with a more complete solution for enhancing LLM memory and help industries accelerate intelligent transformation.
