A Data Lake is a centralized storage system for many types of data. With its flexibility, scalability, and low cost, it has become increasingly attractive to enterprises. However, while a Data Lake reduces storage costs, it can still face performance bottlenecks in Data analytics, especially in Real-time analytics scenarios.

This article explains how the open-source project StarRocks can improve Iceberg query performance and provide enterprises with faster, more flexible Data analytics capabilities.

1. Performance Challenges in Iceberg Data Lake Queries

Iceberg architecture

Iceberg provides strong functionality and flexibility. In practice, however, engineers often encounter query performance bottlenecks when working with Iceberg tables. For growing enterprises, this does not only create latency in production systems. It can also affect overall business delivery and decision-making. Once performance issues occur, enterprises may need to spend significantly more effort and cost to resolve them.

1.1 Data Read Performance Bottlenecks

First, because Iceberg stores data in a Distributed file system, data read performance is limited by the performance of the underlying file system.

File-level metadata in a Data Lake is powerful, but scanning an entire table for queries can take a long time. Iceberg manifest files are usually highly compressed to save storage space. For large manifest files, decompression takes more time. When reading and parsing these large files with languages such as C++, parsing may still take a long time even under optimal conditions.

1.2 Limited Performance for Complex Multi-dimensional Queries

Second, Iceberg has limitations in complex Multi-dimensional analytics, especially when processing large datasets. Its indexing and Partitioning methods can improve performance, but they may still be insufficient in specific scenarios. Snapshot queries can reflect the latest data state in a timely manner, but in complex query scenarios, they may significantly increase the amount of data scanned and reduce query efficiency.

In summary, Iceberg offers strong functionality and flexibility, but enterprises must also invest in technical tuning strategies to achieve optimal system management and performance. For application scenarios that require efficient complex queries, a simpler and more efficient solution should be considered.

2. The Best Way to Accelerate Iceberg Queries: StarRocks

StarRocks architecture

In addition to analyzing data stored locally, users can connect directly to an Iceberg Data Lake through the External Catalog provided by StarRocks. This enables seamless integration and efficient queries on Iceberg data without Data migration. It simplifies operations and greatly reduces Data access latency.

2.1 Optimize Query Execution with StarRocks to Significantly Improve Query Performance

StarRocks uses multiple optimization techniques, such as Columnar storage, index acceleration, and parallel processing, to significantly improve Query performance.

To address performance issues in Iceberg metadata acquisition and parsing, StarRocks uses its Massively parallel processing (MPP) architecture to read and filter metadata files in parallel. This reduces pressure on the Master node and significantly improves metadata processing efficiency, accelerating query plan generation.

The Cost-based optimizer (CBO) in StarRocks can collect statistics for External tables, including Iceberg tables, and use these statistics to generate efficient execution plans. Since version 3.2, StarRocks has been able to evaluate query costs more accurately and select the optimal execution path.

2.2 DataCache and Materialized Views Deliver a Data Warehouse-Level Query Experience

To further improve query efficiency, users can deploy a caching mechanism in StarRocks to store hot data. When a query request arrives, if the relevant data exists in the cache, StarRocks can read it directly from the cache without accessing Iceberg.

For scenarios with complex compute logic and higher performance and concurrency requirements, users can also use a Pre-aggregation strategy to generate Materialized views. Complex query results can be precomputed and stored in advance to accelerate query response. StarRocks currently supports Synchronous materialized views, Asynchronous materialized views, automatic Materialized views, and other capabilities.

3. Best Practice: Xiaohongshu’s Lakehouse Practice Improves Performance by 6–7x

Xiaohongshu data architecture

In Xiaohongshu’s data platform architecture, Iceberg is the core component of the Data Lake and is responsible for storing massive amounts of data. As the business developed and data volume grew, Iceberg Data Lake query performance became a bottleneck.

StarRocks 3.0’s lake analytics capabilities helped Xiaohongshu smoothly migrate its Query engine from Presto to StarRocks in self-service analytics scenarios through a gray release migration mechanism. For the migrated workloads, average Query performance improved by 6–7x. At the same time, through StarRocks’ automated data synchronization process, Xiaohongshu could directly analyze Iceberg data and ensure that data in Iceberg was synchronized to StarRocks in real time or on a schedule to maintain Data consistency.

Through optimization methods such as elastic scaling, Xiaohongshu simplified its technical stack while maintaining Query performance and reducing IT costs. In particular, it achieved further cost savings during off-peak periods by returning Spot instances.

Click here to learn more technical details and user practices related to Iceberg integration.

Conclusion

Data Lake architectures offer enterprises flexibility and cost advantages, but query performance can become a major bottleneck as data volumes and analytics requirements grow. By connecting directly to Iceberg through External Catalog, optimizing query execution with MPP architecture and the Cost-based optimizer (CBO), and using DataCache and Materialized views, StarRocks can help enterprises accelerate Iceberg queries and deliver a more responsive Data analytics experience.

For enterprises building Lakehouse architecture, this approach provides a practical way to combine open data flexibility with high-performance analytics.