Resource isolation has long been a major topic for users running mixed workloads on shared analytics clusters. In practice, requirements for resource isolation usually center on four areas:
- Isolating resources for critical workloads.
When queries from core business applications are running, users want to limit the resources available to other task types, helping protect response time for priority workloads. - Interrupting large queries early.
Large queries should be terminated as soon as possible when they exceed predefined limits, instead of failing only after consuming significant cluster resources. - Limiting CPU, I/O, and memory across tenants and task types.
Users need to isolate CPU, I/O, memory, and other resources between tenants, as well as between different workload types within the same tenant. - Improving resource utilization.
Users also want strong utilization when multiple workload types run on the same cluster. If some resource groups are idle, other active resource groups should be able to elastically use those available resources.
In short, users expect resource isolation capabilities that can reasonably control CPU, I/O, memory, and other resources. The goal is to support multi-tenant resource isolation while fully utilizing available resources, allowing different workloads to run on the same cluster and reducing cluster management costs.
Resource isolation also provides an important foundation for future serverless architectures. Compared with the “SQL-out” model, which scales by increasing the number of nodes, the “SQL-up” model scales by increasing the resources of a single node and can provide faster elasticity, potentially within seconds. However, the SQL-up model also means multiple tenants may share the same cluster, making resource isolation a critical technology.
The key challenge is improving resource utilization while preserving resource isolation. Without resource isolation, a cluster can achieve strong resource utilization but offers no isolation. With complete physical isolation, isolation is strong, but elasticity is limited.
The chosen approach is to implement scheduling policies in user space and provide logical isolation.
#01 Existing Resource Isolation Capabilities
Current resource isolation capabilities fall into two categories: resource isolation and resource limits.
For resource isolation, the system provides:
- hard isolation for memory resources;
- soft isolation for CPU and I/O resources;
- hard isolation of CPU and I/O resources for short-query resource groups.
For resource limits, the system provides:
- concurrency limits;
- circuit-breaking controls for large queries.
1. Defining Resource Groups
Before implementing resource isolation, it is necessary to define how resources should be isolated between tenants and between different workload types within a tenant.
This is where the concept of a resource group is introduced. As shown in the figure below, a resource group can define quotas for different types of resources and bind multiple classifiers. Classifiers are then used to match different tenants and different workloads within a tenant to different resource groups.
2. Query Classification
Query classification is based on characteristics such as user, role, and IP address.
More specifically, a classifier can provide any of the following five conditions:
- user;
- user role;
- SQL type;
- IP address from which the query is initiated;
- database accessed by the query.
At present, SQL type supports only SELECT statements, that is, queries. Support for other task types such as data loading is being developed progressively.
After a query is initiated, classifiers from multiple resource groups may satisfy the matching conditions. In that case, the system needs to select the most appropriate classifier. The selection process mainly consists of three steps:
- Prefer classifiers with a database condition.
- If there are nnn classifiers with a database condition, or no classifier has a database condition, prefer the classifier with the largest number of conditions.
- If multiple classifiers have the same number of conditions, choose the classifier with the most precise conditions.
In the figure above, classifiers A through D are shown in ascending order of priority. Their priority differs depending on whether they include a database condition, the number of conditions they contain, and the precision of those conditions.
After resource groups and classifiers are defined, each resource isolation capability can be applied.
3. Hard Isolation for Memory Resources
Hard isolation for memory means that a resource group can use only the memory resources specified by its memory_limit.
When the memory usage of a resource group exceeds this limit, queries in that resource group return a failure status.
3. Hard Isolation for Memory Resources
Hard isolation for memory means that a resource group can use only the memory resources specified by its memory_limit.
When the memory usage of a resource group exceeds this limit, queries in that resource group return a failure status.
4. Soft Isolation for CPU Resources
Soft isolation for CPU resources allocates CPU time slices across all running resource groups according to the ratio of their cpu_core_limit values.
For example, assume there are three resource groups with a ratio of 1:3:4. They share all 16 CPU cores according to this ratio. Logically, they are allocated 2 cores, 6 cores, and 8 cores respectively.
When resources are idle, active resource groups can elastically use more resources. After the first scheduling cycle ends, if all tasks in the third resource group have completed, resource groups 1 and 2 can use all CPU resources in scheduling cycles 2 and 3. In other words, each cycle can allocate more time slices to the remaining active groups.
5. Hard CPU Isolation for Short-Query Resource Groups
Soft CPU isolation may introduce a small amount of scheduling latency. For extremely latency-sensitive scenarios such as high-priority point queries, short-query resource groups are provided.
When queries from a short-query resource group are running, CPU resources equal to its cpu_core_limit are reserved for that group. Other resource groups can use only the remaining CPU resources.
As shown in the figure below, in the first scheduling cycle, resource groups 1 and 2 have already reached their CPU usage limits. Even if all tasks in the short-query resource group are blocked and no task is ready to run, tasks from resource groups 1 and 2 will not be scheduled for execution. After the first scheduling cycle ends and all tasks in the short-query resource group are completed, resource groups 1 and 2 no longer have a hard CPU restriction in scheduling cycles 2 and 3, allowing them to use all CPU resources.
6. Soft I/O Isolation and Hard Isolation for Short-Query Resource Groups
The I/O resource isolation strategy also has two parts:
- Each resource group is allocated I/O time slices according to the ratio of its
cpu_core_limit. - When queries from a short-query resource group are running, I/O resources equal to its
cpu_core_limitare reserved for that group.
Together, these mechanisms provide isolation for CPU and I/O resources.
However, if too many queries enter the system at the same time, the cluster can still become overloaded. For this reason, hard limits are also provided for concurrency resources.
7. Hard Limits for Concurrency Resources
Hard concurrency limits mean that a query can process only up to concurrency_limit concurrent requests at the same time.
If the number of initiated requests exceeds this threshold, the query immediately returns a failure status. Future versions are expected to support real-time query queuing to avoid directly returning failures.
8. Circuit Breaking for Large Queries
When a query exceeds thresholds for CPU time, I/O scanned rows, or memory resources, the large query is immediately interrupted and returns a failure status.

These mechanisms together define resource groups, classifiers, and resource isolation capabilities.
In summary, to use resource isolation, users first create resource groups and classifiers to specify resource quotas and match tenants and workload types. When a query is initiated, it is matched to the most appropriate resource group through classifiers. Each node then schedules CPU and I/O resources according to the resource quota of that resource group, limits memory and concurrency resources, and checks whether circuit breaking is required.
#02 Practical Effects of Resource Isolation
To evaluate the actual effects of resource isolation, three types of tests were conducted:
- tests where different resource groups run the same workload;
- tests combining small and large queries;
- tests for short-query resource groups.
1. Running the Same Workload in Different Resource Groups
Two resource groups were used, with a cpu_core_limit ratio of 2:1. Each resource group ran the same query workload with concurrency of 4.
The expected QPS ratio between the two resource groups was 2:1, demonstrating that resources were being used proportionally.
The figure below shows the SSB 100 GB test results. As shown, the QPS of each query follows the 2:1 ratio.
In the TPC-DS 100 GB test results for the first 20 queries, each query also basically followed the 2:1 QPS ratio. This shows that, when different resource groups run the same workload, the two resource groups can use resources proportionally and remain isolated from each other.
2. Testing Mixed Small and Large Queries
In real-world usage, workloads that mix small and large queries are more common. Therefore, this type of test was also conducted.
The setup still used two resource groups, with a cpu_core_limit ratio of 2:1. The first resource group ran small queries with concurrency of 8, while the second resource group ran large queries with concurrency of 4.
The expected ratio between the QPS of small queries mixed with large queries and the QPS of small queries running alone was 2:3. This would show that the small-query resource group used two-thirds of the resources and was not affected by large queries in another resource group.
A variety of query types were tested for both small and large queries.
The figure below shows the results of the first small-query test. The yellow line represents the results with resource groups disabled, while the blue line represents the results with resource groups enabled. The section between the two dotted lines shows the QPS of small queries running together with large queries. The ratio between the QPS of small queries in the mixed workload and the QPS of small queries running alone matches the expected 2:3 ratio.
The second small-query test also basically followed the 2:3 ratio. This shows that, in mixed small-query and large-query tests, the resource group running small queries was not affected by the other resource group running large queries, achieving effective resource isolation.
3. Testing Short-Query Resource Groups
For high-priority, very short point queries, the short-query resource group feature was tested.
The cpu_core_limit ratio between the short-query resource group and another resource group was 3:1. The short-query resource group ran point queries with concurrency of 8, while the other resource group ran large queries with concurrency of 4.
The expected ratio between the QPS of point queries mixed with large queries and the QPS of point queries running alone was 3:4. This would show that the short-query resource group used three-fourths of the resources.
The figure below shows the detailed test results. The QPS ratio between point queries running together with large queries and point queries running alone basically matches 3:4. This indicates that three-fourths of resources can be reserved for the short-query resource group.
#03 How Resource Isolation Works
The figure below shows the overall architecture of resource isolation.
When a query is initiated, it is bound to a suitable resource group based on classifiers. The query is then split into multiple execution-unit tasks and distributed to backend nodes. Each backend node schedules CPU and I/O tasks according to the resource quota of the corresponding resource group, limits memory and concurrency resources, and checks whether large-query circuit breaking is required.
1. Hard Limits for Memory Resources
Hard limits for memory reuse the existing memory statistics and checking mechanisms.
TCMalloc hooks are used to track and check memory when memory is allocated and released. When each thread runs, a Memory Tracker is bound to a thread-local variable. This Memory Tracker records and checks memory usage.
The Memory Tracker uses a multi-level tree structure. For resource isolation, each resource group’s Memory Tracker is added to this tree structure.
2. Soft Isolation for CPU and I/O Resources
Soft isolation for CPU and I/O resources is implemented based on the pipeline execution engine. The pipeline execution engine schedules the execution-unit tasks of all queries in user space.
More specifically, there are execution threads equal to the number of cores, as well as a ready queue for query execution-unit tasks. Each execution thread takes a suitable task from the ready queue and executes it.
A running task can yield the execution thread and return to the ready queue when its time expires or due to other scheduling factors. The execution thread then takes the next suitable task from the ready queue.
Because the pipeline execution engine schedules execution-unit tasks for all queries in user space, it can conveniently implement different scheduling policies.

When resource isolation is disabled, all tasks are scheduled through a multilevel feedback queue.
When resource isolation is enabled, a two-level queue is used for scheduling. This approach is similar to the Completely Fair Scheduler used for Linux processes:
- the first level is a priority queue of resource groups;
- the second level is a multilevel feedback queue inside each resource group.
Each time an execution thread selects a suitable task to run, it chooses a task from the resource group with the smallest current Vruntime.
Vruntime is calculated by dividing CPU execution time or I/O execution time by cpu_core_limit.
This method ensures that all resource groups use CPU and I/O resources according to the ratio of their cpu_core_limit.
To reduce scheduling latency as much as possible, the system continuously checks whether the Vruntime of the currently running task is still the smallest. If it is no longer the smallest, the task yields CPU or I/O resources in a timely manner.
To avoid switching tasks too frequently, each task is guaranteed to run for at least 5 milliseconds before yielding.
In addition, if a resource group that has had no tasks for a long time receives new tasks again, its Vruntime may be much smaller than that of other resource groups because those other groups have been running continuously and their Vruntime has kept increasing.
To avoid continuously scheduling tasks from this resource group and causing starvation in other groups, the system adjusts its Vruntime to the current smallest resource group Vruntime, minus a constant value as a reward.
3. Hard CPU and I/O Isolation for Short-Query Resource Groups
For high-priority, very short point queries, short-query resource groups are provided.
When tasks from a short-query resource group are present, other resource groups can use only resources within their hard_limit in each scheduling cycle. If their resource usage exceeds this upper threshold within a cycle, their tasks will no longer be scheduled in that cycle.
However, tasks from multiple resource groups may be running at the same time. When an over-limit condition is detected, their actual resource usage may already have exceeded the limit by a significant amount.
To address this, a penalty strategy is added to ensure overall hard isolation for these resource groups:
- If their resource usage in the current cycle is between one times and two times the
hard_limit, the initial resource usage value for the next cycle is set to the current usage minus thehard_limit. This means they can use only a small amount of resources in the next cycle. - If their current usage is more than two times the
hard_limit, the initial resource usage value for the next cycle is set to thehard_limit. This means their tasks cannot be scheduled in the next cycle.

#04 Future Roadmap
In summary, to use resource isolation, users first define resource groups and classifiers to specify quotas for different resource types and match tenants and workload types.
When a query is initiated, it is matched to an appropriate resource group through classifiers. CPU and I/O resources are then softly isolated according to the quota of the corresponding resource group. When tasks from a short-query resource group exist, hard isolation is applied to CPU and I/O resources. At the same time, memory and concurrency resources are limited, and the system checks whether large-query circuit breaking is required.
Current development work focuses on task queuing and spill mechanisms to avoid query failures when concurrency resources or memory resources exceed limits.
Future plans also include support for more workload types, including analytical database loading and compaction, as well as support for multiple short-query resource groups to meet a wider range of user scenarios.
