1. Why Lakehouse Is Becoming a Must-Have for Enterprise Data Architecture
More enterprises are building core competitiveness through real-time data processing capabilities. Users expect apps to accurately capture their needs and respond in real time. Enterprise employees expect business systems to deliver second-level query response. Behind these scenarios is the agile orchestration of hundreds of billions of data assets.
According to IDC forecasts, global data volume will exceed 175 ZB in 2025, with unstructured data accounting for more than 80%. By 2027, global data generation is expected to exceed 300 ZB.
Enterprises are entering a golden stage for realizing data value, and Lakehouse is becoming a key engine in this transformation:
Multimodal data integration: As AI applications go deeper into enterprise workflows, the joint analytics of structured data, semi-structured data, and unstructured data is creating new business scenarios. Lakehouse architecture natively supports unified management of multi-source, heterogeneous data, making cross-modal analytics such as video analytics and document parsing possible.
Continuous improvement in cost-performance: By adopting an architecture based on Separation of storage and compute, such as S3 or OSS object storage, enterprises can reduce storage costs to around 10% of those in a traditional Data warehouse. At the same time, elastic compute resources enable flexible cost control for real-time queries and batch processing.
Lakehouse architecture is becoming strategic infrastructure for enterprises to realize data value.
According to available market data, the global Real-time Lakehouse market is expected to grow from USD 285 million in 2025 to USD 1.241 billion in 2031. This further confirms the strategic value of Lakehouse architecture.
With Open table format, such as Iceberg, Hudi, and Paimon, together with Unified batch and streaming engines, enterprises can realize One Data, All Analytics, and achieve PB-scale storage and sub-second query response time at a lower cost.
Over the past year, tools across the Lakehouse ecosystem have developed rapidly, gradually pushing the ecosystem toward maturity:
Convergence around Open table format: Snowflake open-sourced Polaris Catalog, and Databricks acquired Tabular, accelerating Apache Iceberg’s move toward becoming a de facto standard for cross-cloud Data lakes. Amazon S3 and Azure Data Lake Storage now fully support Apache Iceberg, showing that cloud vendors are aligning with the open ecosystem.
Competition at the metadata layer: The open sourcing of Unity Catalog and Polaris Catalog marks the transition of metadata management from proprietary protocols to open collaboration. Enterprises can now manage permissions, Data lineage, and models across engines in a unified way.
As digital transformation enters a deeper stage, Lakehouse architecture gives enterprises a solution that combines elastic scalability, strong performance, and controllable cost.
2. Three Core Technology Evolutions in Lakehouse
Storage Layer: Unified Data Source and Low-Cost Support for Full-Scenario Analytics
The storage layer implements a Single source of truth, supporting unified storage and data usage for scenarios such as AI training and BI analytics. It avoids the loss caused by data movement in traditional architectures and significantly reduces enterprise storage costs.
Through Open table format, the storage layer now supports not only structured data, but also semi-structured data such as JSON, as well as unstructured data such as video and audio for AI scenarios.
In addition, the storage layer introduces Multi-level indexing, including Primary Key index and Vector inverted index, to improve data access efficiency. It also supports multiple processing modes, from batch operations to Streaming, real-time reads and writes, meeting a broad range of enterprise data requirements.
Compute Engine Layer: Diversified Expansion and Strong Cost-Performance
The Compute engine layer is moving toward diversified convergence. Typical engines such as Apache Spark for batch processing, Apache Flink for stream processing, and real-time analytics engines are actively expanding their functional boundaries, breaking traditional segmentation, and evolving toward a Unified engine.
At the same time, the engine layer is placing greater emphasis on cost-performance. By supporting new hardware such as ARM architecture and GPU, and introducing innovative mechanisms such as Global cache, engines can improve compute efficiency and deliver higher resource utilization while retaining their specialized strengths.
For example, as a Query engine, performance and cost-performance remain core optimization priorities. It will continue enhancing support for the Lakehouse ecosystem, allowing tasks that once required multiple engines to be completed by one Unified engine, making data processing more convenient and intelligent:
- Improve cost-performance by offering better statistics collection, indexes, and Materialized view support to enhance performance.
- Follow new evolutions in the storage layer by supporting semi-structured data types, DELETE Vector, and other capabilities optimized for real-time scenarios.
- Provide complete capabilities for reads and writes, DDL, Stored procedure, and Table migration.
- Advance data storage optimization, including Compaction service and Automatic data layout optimization.
Metadata Layer: From Catalog Management to Data Intelligence
The metadata layer has expanded from traditional tables and views to multidimensional Data assets such as AI models, features, and Metrics. By providing a unified Metadata service, it enables seamless integration with different Compute engines. In Data governance, the metadata layer also provides comprehensive capabilities for permission management, Scheduling and orchestration, and Data lineage analysis.
The metadata layer is also becoming deeply integrated with AI and BI technologies, supporting Intelligent discovery and management of Data assets. Current solutions such as Unity Catalog and Apache Gravitino are actively improving their product capabilities, pushing metadata management toward greater intelligence.
3. Four Trend Predictions for 2025
As Lakehouse architecture gained mainstream global recognition in 2024, 2025 will mark a deeper stage of development. Based on our research into market dynamics and user practices, we see four key trends ahead.
Trend 1: Open Table Format, Especially Apache Iceberg, Becomes the De Facto Standard
As more enterprises migrate from traditional architectures, such as Hadoop and ClickHouse, to Lakehouse based on Open table format, the global Open table format market is showing a dual-track competitive landscape.
In overseas markets, Apache Iceberg has established a leading position with its mature ecosystem and support from mainstream cloud vendors, becoming the de facto standard for Open table format. In the China market, Apache Paimon is the primary technology choice for some leading internet companies.
Their relationship is similar to the market pattern between Apache Spark and Apache Flink. The two will complement each other in different scenarios. It is worth noting that Apache Paimon already supports generating Apache Iceberg-compatible metadata, giving users more flexible technology choices.
Trend 2: Interface Standardization Creates Opportunities for Modular Lakehouse Architecture
As Open standards become established and the ecosystem matures, Modular Lakehouse architecture will gain wider recognition among enterprises, especially leading enterprises. Companies can flexibly combine storage, engines, and metadata based on business requirements to achieve optimal cost-performance.
Key drivers behind this transition include:
- Increasing maturity of Open standards, which makes collaboration between components more flexible
- Enterprises’ desire to avoid Vendor lock-in and enable independent Technology selection
- The core demand to reduce cost and improve efficiency, or in other words, to achieve better cost-performance
Leading enterprises with stronger technical capabilities are more likely to build a data architecture that best fits their own requirements by flexibly combining storage, Compute engines, and metadata management capabilities, rather than choosing an integrated solution from a single vendor.
Trend 3: Real-Time Lakehouse Becomes the Core Entry Point for Enterprise Upgrades in China
Real-time data processing requirements are evolving from T+1 and hourly analytics to second-level query response. This makes Real-time Lakehouse the preferred entry point for enterprise data architecture upgrades. Compared with traditional architectures, a Real-time Lakehouse solution offers two key advantages: meeting business requirements for real-time capabilities and significantly reducing Total cost of ownership (TCO).
With its strong Unified batch and streaming capabilities, Apache Paimon performs well in real-time scenarios and is gaining favor among more Chinese enterprises. The combination of Apache Paimon with a real-time analytics engine has also become one option for enterprises building a Real-time Lakehouse.
Based on benchmark cases in industries such as internet and finance over the past one to two years, we observe that Lakehouse adoption in China still needs to go through three stages:
Short term: Enterprises will use Real-time Lakehouse as the entry point and adopt combinations such as Apache Paimon with a real-time analytics engine for rapid deployment in response to business-side requirements. During this stage, Separation of storage and compute will also develop in parallel as an important solution.
Medium term: As the ecosystem matures, Open table format will cover more scenarios. Enterprises will migrate Near-real-time and some offline workloads to Lakehouse, moving toward unified data storage and applications.
Long term: Traditional Data warehouses will gradually be replaced by Lakehouse, forming a truly unified Lakehouse architecture. This will also support deeper integration between AI and Data analytics scenarios, further enabling data intelligence.
Trend 4: AI-Native Lakehouse Becomes New Infrastructure
Lakehouse is becoming an important part of enterprise AI infrastructure. According to a KPMG AI report, 86% of overseas enterprises plan to unify their analytics data to support AI development. In China, this trend is also clear. Leading internet companies such as Tencent, Bilibili, and Xiaohongshu have adopted Lakehouse architectures that involve AI applications to varying degrees.
AI training data requirements will drive more enterprises to migrate from traditional Data warehouses to Lakehouse architecture, especially for capabilities such as unstructured data processing and Vector retrieval. Lakehouse can support real-time Stream processing and Machine learning algorithms, meeting model training requirements.
These four trends influence and reinforce one another, jointly driving the evolution of Lakehouse technology. Regardless of whether an enterprise chooses Lakehouse architecture, it needs to comprehensively assess its current technology landscape.
4. Recommendations for Enterprise Data Platform Technology Selection
4.1 Assess the Current Enterprise Architecture
First, enterprises need to analyze their current data scale and growth trends to determine whether they face excessive storage costs and whether they have urgent requirements for real-time analytics. This helps evaluate the impact of the current architecture on business operations and efficiency.
Second, they need to assess the capabilities and limitations of their current technology stack, including Query performance bottlenecks, operations complexity, upstream and downstream extensibility and compatibility, and support for emerging AI applications.
Migration cost is also a major consideration. In addition to infrastructure investment, it includes training costs for technical and operations teams, business interruption risks, and other factors.
Through a comprehensive assessment, enterprises can better understand whether Lakehouse architecture is truly suitable for them and what migration approach they should take. MirrorShip has prepared a Data capability self-assessment table to help enterprises quickly understand their requirements and current state.
4.2 Analyze Technology Selection and Apply the Right Solution to the Right Requirement
Based on practical experience in the Lakehouse field, MirrorShip has summarized Technology selection directions for different business requirements as a reference for enterprises.
Focus on Real-Time Analytics
Enterprises with this type of requirement can prioritize an architecture that combines a real-time analytics engine with Apache Paimon. This approach can meet second-level query response requirements and deliver more real-time data insights through Unified batch and streaming processing.
For example, Ele.me upgraded from an Offline Data warehouse plus real-time Lambda architecture to a Lakehouse solution based on Apache Flink, Apache Paimon, and a real-time analytics engine. The new architecture supports scenarios such as self-service analytics for real-time transaction subsidies and customer satisfaction service Dashboards. Compared with a Real-time data warehouse, the solution reduced storage costs by about 90% and Apache Flink compute overhead by about 50%, with only a 15% increase in Query latency.
Focus on Cost Reduction and Efficiency Improvement
Enterprises whose primary goal is cost reduction and efficiency improvement can consider an architecture based on Separation of storage and compute combined with an Intelligent caching strategy. This approach offers significant cost advantages while delivering performance close to Shared-nothing mode. Through elastic resource scheduling and comprehensive Data lifecycle management, enterprises can significantly reduce operating costs while meeting real-time scenario requirements and maintaining performance.
Focus on AI Application Expansion
Open table format, such as Apache Iceberg, has certain advantages in this scenario. It can support Multi-engine access, meet data requirements across different AI training scenarios, and reduce storage costs through Unified data management. This also explains why more internet companies are migrating AI-related data to Lakehouse architecture.
Lakehouse Data Platform Best Practice: Xiaohongshu
Architecture upgrades require not only Technology selection, but also a deep understanding of business requirements, followed by continuous optimization and adjustment in practice.
As a leading internet company in China, Xiaohongshu achieved significant technical breakthroughs by adopting a Lakehouse architecture based on a real-time analytics engine and Apache Iceberg:
- Query performance: P90 query response time improved by 3x and remained stable within 10 seconds
- Storage efficiency: 50% less storage space than the original ClickHouse architecture
- Data governance: Introduced an intelligent Sort key selection mechanism to continuously optimize Query efficiency
- Architecture flexibility: Supports multiple JOIN strategies and improves Self-service data exploration capabilities
Through a series of innovative technical solutions, including DataCache Local caching, Z-Order intelligent sorting, and flexible JOIN strategies, Xiaohongshu improved P90 query response time by 3x. At the same time, by optimizing the Data Skipping mechanism, it achieved a breakthrough in storage efficiency, saving 50% storage space compared with the original architecture.
Click here to learn more about the case
5. Conclusion
Driven by Data asset value realization and the rapid growth of AI, Lakehouse has evolved from a technical concept into a core foundation for enterprise digital transformation. Enterprises should focus on three key directions:
- Build a Data governance framework for Open table format, and enable Cross-platform data access through Standard interfaces such as Apache Iceberg, Apache Hudi, Apache Paimon, and Delta Lake.
- Make Real-time Lakehouse capabilities a strategic priority, and use real-time engines to unlock the value of Streaming data.
- Reserve unstructured data processing capabilities in AI infrastructure planning, and build a future-ready architecture that supports Multimodal data applications.
