1. Overview of Traditional Data Warehouse Analytics Architectures

Traditional Data warehouse analytics is typically implemented with a Lambda architecture. As shown in the diagram, the traditional architecture mainly consists of two layers: the upper layer is the real-time processing path, and the lower layer is the offline processing path.

Data first enters through the Data ingestion layer on the left. It is then integrated through different paths into a message queue middleware such as Kafka. The data is split into two identical copies, processed separately by the real-time path and the batch path, and finally aggregated into the data service layer to provide Data analytics services to users.

Lambda architecture diagram for traditional data warehouse analysis

The Lambda architecture emerged because users started to require real-time analytics, while Stream processing technologies were becoming more mature. However, it also has several obvious drawbacks. As shown in the diagram, it requires two separate systems, which increases both deployment costs and labor costs. When business requirements change, both systems must be modified to adapt to the change.

As Stream processing technologies continued to mature, the Kappa architecture was introduced after Lambda architecture, as shown in the diagram below.

Kappa architecture diagram for data warehouse

The Kappa architecture uses a Stream processing path to replace the original Lambda architecture. As Stream processing became more mature, it became possible to use a single system to handle both real-time and offline computing.

However, the Kappa architecture is based on one assumption: repeated computation over historical data is unnecessary unless required. As a result, when users need to recompute historical data or adapt to new business changes, they often have to replay the entire Data ingestion process. Consuming a large volume of historical data in this way inevitably wastes resources and creates bottlenecks.

2. Building a Lakehouse Analytics Solution with Paimon + StarRocks

2.1 Data Lake-Centric Architecture

The first solution is a data lake-centric architecture for building Lakehouse analytics with Paimon and StarRocks.

Paimon + StarRocks unified lakehouse data analysis architecture

StarRocks is an Massively parallel processing (MPP) architecture database and can connect to multiple data lake components in different formats. It can also be used purely as a Query engine to connect to data lake components and enable query capabilities. As shown in the diagram, both StarRocks and Spark can query Paimon components in data layers such as ODS.

In this architecture, Paimon persists data to disk and builds indexes. This compensates for the limitations of message queue middleware in the Kappa architecture described above, especially in data modification, backtracking, and query scenarios. As a result, the architecture provides higher fault tolerance and broader capabilities. For Batch processing, Paimon is also fully compatible with Hive capabilities.

2.2 Query Acceleration

The second solution is a query acceleration architecture for building Lakehouse analytics with Paimon and StarRocks.

Paimon and StarRocks lakehouse query acceleration architecture

The difference from the first solution is that almost the entire system is handled by StarRocks alone. After data is ingested into Paimon and Paimon acts as the ODS layer, StarRocks reads data from Paimon through its External table capability and builds a Materialized view layer as the DWD layer.

StarRocks Materialized view provides certain ETL capabilities. After it serves as the DWD layer, a second layer of nested Materialized view is built as the DWS layer, which ultimately provides data to the data service layer for Data analytics.

This architecture, which combines StarRocks with Paimon, has two main advantages:

  • It simplifies operations and maintenance. Instead of maintaining multiple components, users only need StarRocks and Paimon to build a Data analytics solution.
  • It delivers faster query performance. StarRocks provides a complete data lake engine covering index construction, data storage, and Query optimization, which makes it faster than the other Query engines discussed above.

2.3 Materialized View

Materialized view features and SQL example for Paimon + StarRocks

The SQL on the right side of the diagram shows how to create a StarRocks Asynchronous materialized view. It has the following key characteristics:

  • Defined through SQL, making it easy to get started with and maintain.
  • Uses Pre-aggregation to reduce Query latency and repeated computation overhead.
  • Supports automatic query routing, so users do not need to rewrite SQL and can achieve transparent acceleration.
  • Supports Async refresh, Scheduled refresh, and intelligent partition-based refresh.
  • Supports construction from multiple tables. Base tables can come from Native tables, External tables, and existing Materialized views.

2.4 Hot and Cold Data Separation

This section describes how Paimon + StarRocks supports hot and cold data separation.

Paimon+StarRocks cold and hot data separation architecture

The idea of hot and cold data separation is to store frequently queried hot data in a fast-query OLAP engine such as StarRocks, while storing infrequently queried cold data in more cost-effective remote file storage components such as OSS and HDFS.

In the Paimon + StarRocks hot and cold data separation example shown in the diagram, if such a hot-and-cold separated Materialized view table is built, queries against this table automatically select hot data distributed in StarRocks and cold data distributed in Paimon. The query results are then merged and returned to the user.

3. Usage and Implementation Principles of StarRocks with Paimon

3.1 Using Paimon External Tables

Thanks to StarRocks’ abstraction of external table Catalogs, soon after Paimon was released, StarRocks implemented support for Paimon External tables by implementing the corresponding interfaces.

When connecting to a Paimon External table, users only need to execute the following CREATE EXTERNAL CATALOG statement in StarRocks, set Type to Paimon, and provide the corresponding path. They can then directly query data in Paimon.

SQL examples for using Paimon external tables

3.2 JNI Connector

JNI Connector is an important capability that enables StarRocks to work with Paimon.

StarRocks JNI Connector architecture and features

The background is that StarRocks’ data processing components are written in C++, while most SDKs provided by data lake components are written in Java and do not provide C++ SDKs. If StarRocks wants to access the underlying data of data lake components through BE, it can only access native formats such as ORC or Parquet, and cannot use the advanced capabilities provided by these components.

JNI Connector is an abstract connector that works with Java SDKs for all External tables. It runs on the StarRocks BE component and acts as a middle layer between BE and the Java SDKs of data lake components.

The main function of JNI Connector is to call the Java SDK of a data lake component to read data from the data lake, then write the data into a block of Direct memory / Native memory using a memory layout that StarRocks BE can recognize. The memory is then handed over to the BE C++ program for execution. This allows BE and Java SDKs to work together.

JNI Connector has the following characteristics:

  • Quickly connects to various Java Data sources without requiring users to handle data conversion.
  • Provides simple and easy-to-use Java interfaces.
  • Already supports Hudi MOR Table and Paimon Table.
  • Supports complex types such as Struct, Map, and Array.
  • Requires zero intrusion into BE code and does not require users to consider the specific C++ implementation.

The diagram below introduces some details of JNI Connector.

JNI Connector fixed and variable length field storage structure

The upper section shows the storage format for fixed-length fields, while the lower section shows the storage format for variable-length fields.

  • Fixed-length field storage format
    • The first part defines whether each row in the column is Null.
    • The second part is the data section, where specific fixed-length data is stored.
  • Variable-length field storage format
    • The first part is an array that defines whether each row in the column is Null.
    • The second part describes the starting address from which each row of data in the third part should be read.
    • The third part contains the actual data.

4. Future Roadmap for StarRocks Community Lakehouse Analytics

StarRocks currently supports part of Paimon’s capabilities, while some capabilities have not yet been implemented. The future plan is to improve Paimon table analytics with the following capabilities:

  • Support analytics over complex types.
  • Support column statistics.
  • Support metadata caching.
  • Support time travel.
  • Support streaming Materialized views based on Paimon External tables.

Q&A

Q: How can Materialized views be managed effectively?

A: After a Materialized view is created, it can be automatically refreshed and scheduled without relying on external components to trigger the refresh. The Query rewrite capability allows users to query only the Base table without specifying a particular Materialized view. These two capabilities reduce many management challenges.

For dependencies between Materialized views and Base tables, as well as dependencies between nested Materialized views, EMR-Serverless-StarRocks will later introduce a web-based display feature for task scheduling and table dependency relationships.

Q: Does the Paimon + StarRocks Lakehouse analytics solution have specific plans for data security, such as access control and data auditing?

A: Based on the current understanding in the source article, StarRocks manages data permissions through role-based permissions such as viewing and modification permissions, granting different permissions to different roles. In addition, data on OSS or HDFS has corresponding component-level authentication capabilities.

Q: In a StarRocks-centric Lakehouse architecture, after data is read from Paimon, is it written back to Paimon?

A: After data is read from the ODS layer in Paimon, it flows into StarRocks Materialized views. It then enters another layer of nested StarRocks Materialized views and is not written back to Paimon.