Lenovo Liansheng Zhida is part of Lenovo Group, a global leader in the PC market. The company is committed to building a technology-driven, flexible, agile, and service-oriented smart logistics ecosystem platform.

Serving enterprise customers across industries, Lenovo Liansheng Zhida provides integrated logistics solutions and has become an intelligent supply chain technology company serving customers in China and around the world.

To better meet growing analytics requirements, the Liansheng Zhida big data team gradually introduced multiple OLAP analytics engines. By adopting a new-generation MPP database with fully vectorized computing technology, the team built a new unified data service platform, greatly reducing data pipeline development complexity and significantly improving BI analytics efficiency.

Evolution of OLAP Engines at Lenovo Liansheng Zhida

Stage 1

Before 2018, Liansheng Zhida’s total data volume was still relatively small. At this stage, the team used a traditional relational database, SQL Server. A data warehouse system had not yet been established, and many data requirements were fulfilled through SQL script development.

As business complexity increased and data volume grew rapidly, this model soon encountered bottlenecks. The most obvious issue was slower query response time. For example, a task that previously took 10 or 20 minutes began taking one hour or longer, leading to a serious decline in query efficiency.

Data storage capacity also became a bottleneck and could no longer meet the storage demands of rapidly growing business data.

Stage 2

In 2019, as the data warehouse was built and improved on the Apache Hadoop® and Apache Hive™ ecosystem, all ETL tasks were migrated to the Hadoop cluster. At this stage, the team used dozens of Presto nodes for OLAP analytics.

Presto naturally shared metadata with Apache Hive™ and used the same physical data storage. A large number of flexible queries on data warehouse tables were completed using Presto. At the front-end BI layer, Tableau connected directly to Presto to support data analysis and exploration.

Stage 3

In 2021, the Liansheng big data team redesigned and rebuilt the offline data warehouse as a whole. The new architecture needed to support low-latency BI reports, complex ad hoc queries, and efficient detail-level queries.

At this stage, the team introduced a high-performance OLAP engine based on scenario requirements. It supported ad hoc multi-table joins and complex nested subqueries similar to Presto, while also providing strong single-table detail queries and multidimensional materialized-view rollup acceleration to meet the need for fast BI analytics.

Data Analytics Architecture

Current OLAP Architecture

The overall data analytics system consists of data collection, data storage and compute, data query and analysis, and data applications.

Original architecture diagram:

Current OLAP and data warehouse architecture diagram

Data Collection

  1. Sqoop was used to read data from RDBMS and import it into Apache Hive™.
  2. Flume was used to synchronize log files to Apache Hive™.
  3. Web crawling technology was used to collect online data and store it in RDBMS. Sqoop then read the data from RDBMS and imported it into Apache Hive™.

Data Storage and Compute

Offline Data Processing: Apache Hive™ was used for its highly scalable batch processing capabilities, handling all ETL tasks and data model processing for the offline data warehouse.

Data Query and Analysis

The data sharing layer mainly provided the underlying data storage and query interface for external services.

After offline ETL, data was written into RDBMS or MPP databases and served multiple downstream services. It provided OLAP query and analytics capabilities for scenarios such as Tableau BI, multidimensional standard reports, and ad hoc queries.

On the application side, the architecture supported BI reporting platforms, ad hoc query analysis platforms, and the data visualization platform, Control Tower.

Data Application Layer

The data application layer mainly served management and operations teams through reports that required low-latency responses. These requirements also changed frequently and continued to evolve.

For ad hoc queries by data analysts, the OLAP engine needed to support complex SQL processing and quickly filter data from massive datasets.

Comparison of OLAP Analytics Tool Options

ClickHouse

Advantages

  • Strong single-table query performance, suitable for OLAP multidimensional analytics queries based on large wide tables.
  • Provides a rich MergeTree Family and supports pre-aggregation.
  • Well suited to large-scale log detail data ingestion and analysis.

Limitations

  • Does not support true delete and update operations.
  • Join support is not very user-friendly.
  • Relatively low concurrency.
  • MergeTree compaction is incomplete.

High-Performance MPP Analytics Engine

Advantages

  • Strong performance for both single-table and multi-table queries, supporting both wide-table query scenarios and complex multi-table queries.
  • Supports high-concurrency queries.
  • Supports real-time micro-batch ETL processing.
  • Strong support for both streaming and batch data ingestion.
  • Compatible with the MySQL protocol and standard SQL.

Limitations

  • Large-scale ETL capability was insufficient at the time.
  • Resource isolation was not yet fully mature at the time.

Application Practice in the SEC Data Center

The core data of Channel Warehouse and Distribution Management, or SEC, came from two major areas: consumer business and SMB business, including Think and Yangtian.

Based on this data, the team aggregated relevant business statistics according to different business scenario requirements and provided external query and analytics services.

Original Solution

Before the upgrade, the team used a large number of Apache Hive™ tasks for business logic cleaning and processing. Some processed data remained in Apache Hive™, while other data was written into MySQL or SQL Server for persistence.

The front-end BI layer connected to Hive, MySQL, SQL Server, and other systems through the Presto compute engine to support report analysis and data visualization.

Technical Pain Points

The original architecture had two main issues:

  1. Data logic was not well consolidated or unified.
    Maintenance workload was high, and new requirements could not be addressed quickly.
  2. Presto response time was slow for complex Tableau reports with many SQL queries.
    This could not meet the business requirement for instant data access.

As a result, the team wanted to optimize the original system. The core idea was to use one OLAP engine to unify this layer. This placed high requirements on the OLAP engine:

  • Support high-throughput data ingestion.
  • Support flexible queries across multiple dimension combinations, with response latency under 100 ms.
  • Provide strong support for multi-table joins.
  • Support single-table queries over more than 1 billion rows, with response latency under 100 ms.

After extensive research, the selected high-performance MPP analytics engine aligned well with the data center’s overall requirements. Its efficient query capabilities could provide integrated data services for data center reports.

The new architecture offered the following advantages:

  • The architecture became clearer. RDBMS focused on data cleaning, while business logic computation was moved from Apache Hive™ into the OLAP engine. The OLAP engine became the endpoint for data business logic.
  • The team could maintain unified metric definitions, with one data input and multiple application interface outputs.
  • The MPP distributed architecture better supported distributed aggregation and join queries.
  • Strong compatibility with Tableau helped meet core BI analytics requirements.

Solution Based on the New MPP Analytics Architecture

Upgraded architecture diagram:

StarRocks OLAP and data warehouse architecture diagram

Table Design

1. Data Model Design

The selected analytics engine provided three data models: detail model, aggregate model, and update model.

For SEC business, the team primarily used the detail model. Other models would be considered later if additional scenarios required them.

2. Data Partitioning and Bucketing

The data partitioning and bucketing capabilities helped improve detail-query performance in historical inventory and turnover scenarios.

For example, a common historical inventory query scenario is to query inventory turnover over a past time period. The team partitioned data by outbound time, allowing unnecessary partitions to be filtered out, reducing the total amount of data scanned and enabling faster positioning.

Partitioning, bucketing, and prefix indexes significantly improved point-query concurrency. These capabilities were also important for supporting business growth and potential high-concurrency scenarios in the future.

When querying the historical trajectory of a material barcode, or SN, the system could quickly retrieve all historical inbound and outbound trajectory information for that barcode, helping the team efficiently complete full lifecycle traceability across the supply chain.

Materialized Views

The team used materialized views that could be built in real time and on demand, added or removed flexibly, and used transparently.

They created materialized views based on inventory material SN granularity, product-type feature granularity, warehouse granularity, and distributor granularity. These materialized views greatly accelerated queries.

Data Import

Two data import approaches were used:

  1. Offline data warehouse tables from Apache Hive™ were imported through Broker Load.
  2. Data from SQL Server and MySQL was imported through DataX.

Results

Flexible Modeling Improved Development Efficiency

The team used both wide-table and star-schema models.

Wide tables and materialized views ensured report performance and concurrency, while star schemas allowed analytical processing to be modeled in a way similar to transaction processing and queried directly through joins. This meant that not all scenarios needed to rely on prepared wide tables, improving both data consistency and development efficiency.

Many tables were stored in MySQL. By exposing queries through external tables, the team avoided the data import process and greatly reduced business-side development and migration cycles.

Distributed join capabilities, combined with views, helped build a unified view layer for different BI reports. This improved metric consistency and reduced duplicate development.

BI Experience Improved Significantly

Some early BI visualizations were built on SQL Server and MySQL. As certain dashboards were continuously optimized and enriched with additional requirements, especially flexible multidimensional filters, each load became slow. Some Tableau reports took a long time to load, which business users could not accept.

After introducing the new architecture, the team used DataX to import SQL Server data. The underlying Stream Load interface provided efficient vectorized import.

MySQL data could be imported through external-table INSERT INTO SELECT streaming import, or queried directly through external tables, making the process very convenient.

Tableau charts loaded in seconds, delivering a major improvement in user experience.

Lower Operations and Maintenance Costs

The data center was a core online service, so high availability and flexible scaling were critical requirements.

The architecture supported multiple data replicas and a simple design with only two core roles, FE and BE. When a single node failed, the cluster could still maintain high availability.

At large data scale, the system could also scale elastically online without downtime during expansion, avoiding impact on online business. This was an important capability for the team.

Summary

Lenovo Liansheng Zhida began evaluating the new MPP analytics engine in April 2021. During the POC testing phase, the team used only one quarter of the resources to replace dozens of Presto nodes.

The platform is now running stably in production. After the upgrade, Lenovo Liansheng Zhida unified its data services, greatly simplified offline data processing pipelines, and met query-latency requirements.

Going forward, the platform will be used to improve data service and query capabilities across more business scenarios.