Tencent Games’ Public Data Platform Department supports hundreds of games, processing daily incremental data at the scale of hundreds of terabytes and trillions of records. To meet growing demand for timely analytics, simplify operations, and scale compute independently of storage, the team selected StarRocks and collaborated with its open-source community to develop a cloud-native storage-compute separation architecture.
Solution
Introduce stateless Compute Nodes (CNs) to separate query processing from storage-bearing Backend (BE) nodes. Combine Kubernetes-based compute scaling with Iceberg-based hot and cold data tiering, retaining frequently accessed data on BE nodes and offloading historical data to Tencent Cloud Object Storage (COS) or HDFS.
The architecture provides a foundation for allocating separate compute resources to different analytics workloads, allowing exploratory queries to scale independently of recurring BI reporting.
Results
The team completed development and testing of the first version and reported that the architecture on Tencent Kubernetes Engine (TKE) and COS was online and running stably.
After optimizing Iceberg metadata access, table refreshes, and execution plans, the performance gap between the storage-compute separated and coupled paths narrowed from 50–100× to 5–10×.
The comparison used typical business SQL against a 1 TB table containing 26 billion rows, with 12 CN nodes on the separated path and 12 BE nodes on the coupled path. These workload-specific results demonstrated a narrower performance gap, not performance parity.
Challenge
Game operations increasingly required timely, multidimensional insights and a unified approach to real-time and offline analytics. The existing platform relied on multiple components, increasing architectural complexity and operational overhead.
Compute demand and storage growth did not increase at the same rate. Adding BE nodes to address CPU or memory bottlenecks introduced unnecessary storage capacity and triggered data migration.
Recurring reports and exploratory queries also competed for resources. At the same time, retaining years of historical data required a more cost-effective storage approach.
Approach
The team separated compute capabilities from BE nodes into stateless CNs. BE nodes continued to scan locally stored data and perform initial aggregation, while CNs handled subsequent processing. External data scans could be scheduled directly on CNs.
On TKE, the team implemented a custom Operator using a ComputeNodeGroup resource and a controller to manage CN deployments. CN containers automatically registered with the StarRocks FE after startup. Kubernetes Horizontal Pod Autoscaler adjusted capacity using CPU and memory metrics, with scaling requests limited to one every five minutes to reduce excessive fluctuations.
For data tiering, the team exported BE data as ORC files, uploaded them to COS or HDFS, and registered partition information and statistics through the Iceberg API. Query planning selected local or external scan operators based on partition metadata, combining results through Union when both tiers were required.
To improve performance, the team added Iceberg FileIO caching, reduced table refreshes to once per SQL statement, and pushed first-stage aggregation below Union to reduce data transferred through Exchange operators.
Business impact
Independent compute scaling reduced the need to expand storage-bearing nodes solely to address compute bottlenecks. Kubernetes-based elasticity enabled resource allocation to adapt to workload spikes during game anniversaries, marketing campaigns, and holidays.
Separate compute resources provided a foundation for isolating exploratory analytics from recurring reporting. Historical data could be placed on lower-cost storage while remaining available for analysis through StarRocks.
Together, these capabilities gave Tencent Games greater flexibility in balancing query performance, workload isolation, resource utilization, and infrastructure cost.