Resource isolation has long been a major topic for users running mixed workloads on shared analytics clusters. In practice, requirements for resource isolation usually center on four areas:

  1. Isolating resources for critical workloads.
    When queries from core business applications are running, users want to limit the resources available to other task types, helping protect response time for priority workloads.
  2. Interrupting large queries early.
    Large queries should be terminated as soon as possible when they exceed predefined limits, instead of failing only after consuming significant cluster resources.
  3. Limiting CPU, I/O, and memory across tenants and task types.
    Users need to isolate CPU, I/O, memory, and other resources between tenants, as well as between different workload types within the same tenant.
  4. Improving resource utilization.
    Users also want strong utilization when multiple workload types run on the same cluster. If some resource groups are idle, other active resource groups should be able to elastically use those available resources.
Workloads and resource groups for resource isolation

In short, users expect resource isolation capabilities that can reasonably control CPU, I/O, memory, and other resources. The goal is to support multi-tenant resource isolation while fully utilizing available resources, allowing different workloads to run on the same cluster and reducing cluster management costs.

Resource isolation also provides an important foundation for future serverless architectures. Compared with the “SQL-out” model, which scales by increasing the number of nodes, the “SQL-up” model scales by increasing the resources of a single node and can provide faster elasticity, potentially within seconds. However, the SQL-up model also means multiple tenants may share the same cluster, making resource isolation a critical technology.

The key challenge is improving resource utilization while preserving resource isolation. Without resource isolation, a cluster can achieve strong resource utilization but offers no isolation. With complete physical isolation, isolation is strong, but elasticity is limited.

The chosen approach is to implement scheduling policies in user space and provide logical isolation.

#01 Existing Resource Isolation Capabilities

Current resource isolation capabilities fall into two categories: resource isolation and resource limits.

For resource isolation, the system provides:

  • hard isolation for memory resources;
  • soft isolation for CPU and I/O resources;
  • hard isolation of CPU and I/O resources for short-query resource groups.

For resource limits, the system provides:

  • concurrency limits;
  • circuit-breaking controls for large queries.

1. Defining Resource Groups

Before implementing resource isolation, it is necessary to define how resources should be isolated between tenants and between different workload types within a tenant.
This is where the concept of a resource group is introduced. As shown in the figure below, a resource group can define quotas for different types of resources and bind multiple classifiers. Classifiers are then used to match different tenants and different workloads within a tenant to different resource groups.

SQL syntax example for creating resource groups

2. Query Classification

Query classification is based on characteristics such as user, role, and IP address.

More specifically, a classifier can provide any of the following five conditions:

  • user;
  • user role;
  • SQL type;
  • IP address from which the query is initiated;
  • database accessed by the query.

At present, SQL type supports only SELECT statements, that is, queries. Support for other task types such as data loading is being developed progressively.

After a query is initiated, classifiers from multiple resource groups may satisfy the matching conditions. In that case, the system needs to select the most appropriate classifier. The selection process mainly consists of three steps:

  1. Prefer classifiers with a database condition.
  2. If there are nnn classifiers with a database condition, or no classifier has a database condition, prefer the classifier with the largest number of conditions.
  3. If multiple classifiers have the same number of conditions, choose the classifier with the most precise conditions.

Classifier priority matching rules example code

In the figure above, classifiers A through D are shown in ascending order of priority. Their priority differs depending on whether they include a database condition, the number of conditions they contain, and the precision of those conditions.

After resource groups and classifiers are defined, each resource isolation capability can be applied.

3. Hard Isolation for Memory Resources

Hard isolation for memory means that a resource group can use only the memory resources specified by its memory_limit.

When the memory usage of a resource group exceeds this limit, queries in that resource group return a failure status.

3. Hard Isolation for Memory Resources

Hard isolation for memory means that a resource group can use only the memory resources specified by its memory_limit.

When the memory usage of a resource group exceeds this limit, queries in that resource group return a failure status.

Memory hard isolation schematic for resource groups

4. Soft Isolation for CPU Resources

Soft isolation for CPU resources allocates CPU time slices across all running resource groups according to the ratio of their cpu_core_limit values.

For example, assume there are three resource groups with a ratio of 1:3:4. They share all 16 CPU cores according to this ratio. Logically, they are allocated 2 cores, 6 cores, and 8 cores respectively.

When resources are idle, active resource groups can elastically use more resources. After the first scheduling cycle ends, if all tasks in the third resource group have completed, resource groups 1 and 2 can use all CPU resources in scheduling cycles 2 and 3. In other words, each cycle can allocate more time slices to the remaining active groups.

CPU soft isolation scheduling illustration

5. Hard CPU Isolation for Short-Query Resource Groups

Soft CPU isolation may introduce a small amount of scheduling latency. For extremely latency-sensitive scenarios such as high-priority point queries, short-query resource groups are provided.

When queries from a short-query resource group are running, CPU resources equal to its cpu_core_limit are reserved for that group. Other resource groups can use only the remaining CPU resources.

As shown in the figure below, in the first scheduling cycle, resource groups 1 and 2 have already reached their CPU usage limits. Even if all tasks in the short-query resource group are blocked and no task is ready to run, tasks from resource groups 1 and 2 will not be scheduled for execution. After the first scheduling cycle ends and all tasks in the short-query resource group are completed, resource groups 1 and 2 no longer have a hard CPU restriction in scheduling cycles 2 and 3, allowing them to use all CPU resources.

Hard CPU isolation timeline diagram for short-query resource groups

6. Soft I/O Isolation and Hard Isolation for Short-Query Resource Groups

The I/O resource isolation strategy also has two parts:

  1. Each resource group is allocated I/O time slices according to the ratio of its cpu_core_limit.
  2. When queries from a short-query resource group are running, I/O resources equal to its cpu_core_limit are reserved for that group.

Together, these mechanisms provide isolation for CPU and I/O resources.

However, if too many queries enter the system at the same time, the cluster can still become overloaded. For this reason, hard limits are also provided for concurrency resources.

Soft and hard I/O isolation diagram for short-query resource groups

7. Hard Limits for Concurrency Resources

Hard concurrency limits mean that a query can process only up to concurrency_limit concurrent requests at the same time.

If the number of initiated requests exceeds this threshold, the query immediately returns a failure status. Future versions are expected to support real-time query queuing to avoid directly returning failures.

Query concurrency limit schematic

8. Circuit Breaking for Large Queries

When a query exceeds thresholds for CPU time, I/O scanned rows, or memory resources, the large query is immediately interrupted and returns a failure status.

Big query circuit breaker

These mechanisms together define resource groups, classifiers, and resource isolation capabilities.

In summary, to use resource isolation, users first create resource groups and classifiers to specify resource quotas and match tenants and workload types. When a query is initiated, it is matched to the most appropriate resource group through classifiers. Each node then schedules CPU and I/O resources according to the resource quota of that resource group, limits memory and concurrency resources, and checks whether circuit breaking is required.

Resource isolation workflow

#02 Practical Effects of Resource Isolation

To evaluate the actual effects of resource isolation, three types of tests were conducted:

  1. tests where different resource groups run the same workload;
  2. tests combining small and large queries;
  3. tests for short-query resource groups.

1. Running the Same Workload in Different Resource Groups

Two resource groups were used, with a cpu_core_limit ratio of 2:1. Each resource group ran the same query workload with concurrency of 4.

The expected QPS ratio between the two resource groups was 2:1, demonstrating that resources were being used proportionally.

The figure below shows the SSB 100 GB test results. As shown, the QPS of each query follows the 2:1 ratio.

SSB 100GB bar chart

In the TPC-DS 100 GB test results for the first 20 queries, each query also basically followed the 2:1 QPS ratio. This shows that, when different resource groups run the same workload, the two resource groups can use resources proportionally and remain isolated from each other.

TPC-DS 100GB Q1-Q20 bar chart

2. Testing Mixed Small and Large Queries

In real-world usage, workloads that mix small and large queries are more common. Therefore, this type of test was also conducted.

The setup still used two resource groups, with a cpu_core_limit ratio of 2:1. The first resource group ran small queries with concurrency of 8, while the second resource group ran large queries with concurrency of 4.

The expected ratio between the QPS of small queries mixed with large queries and the QPS of small queries running alone was 2:3. This would show that the small-query resource group used two-thirds of the resources and was not affected by large queries in another resource group.

A variety of query types were tested for both small and large queries.

The figure below shows the results of the first small-query test. The yellow line represents the results with resource groups disabled, while the blue line represents the results with resource groups enabled. The section between the two dotted lines shows the QPS of small queries running together with large queries. The ratio between the QPS of small queries in the mixed workload and the QPS of small queries running alone matches the expected 2:3 ratio.

Small query Q1 QPS line chart

The second small-query test also basically followed the 2:3 ratio. This shows that, in mixed small-query and large-query tests, the resource group running small queries was not affected by the other resource group running large queries, achieving effective resource isolation.

3. Testing Short-Query Resource Groups

For high-priority, very short point queries, the short-query resource group feature was tested.

The cpu_core_limit ratio between the short-query resource group and another resource group was 3:1. The short-query resource group ran point queries with concurrency of 8, while the other resource group ran large queries with concurrency of 4.

The expected ratio between the QPS of point queries mixed with large queries and the QPS of point queries running alone was 3:4. This would show that the short-query resource group used three-fourths of the resources.

The figure below shows the detailed test results. The QPS ratio between point queries running together with large queries and point queries running alone basically matches 3:4. This indicates that three-fourths of resources can be reserved for the short-query resource group.

Point query QPS line chart

#03 How Resource Isolation Works

The figure below shows the overall architecture of resource isolation.
When a query is initiated, it is bound to a suitable resource group based on classifiers. The query is then split into multiple execution-unit tasks and distributed to backend nodes. Each backend node schedules CPU and I/O tasks according to the resource quota of the corresponding resource group, limits memory and concurrency resources, and checks whether large-query circuit breaking is required.

Overall resource isolation architecture

1. Hard Limits for Memory Resources

Hard limits for memory reuse the existing memory statistics and checking mechanisms.

TCMalloc hooks are used to track and check memory when memory is allocated and released. When each thread runs, a Memory Tracker is bound to a thread-local variable. This Memory Tracker records and checks memory usage.

The Memory Tracker uses a multi-level tree structure. For resource isolation, each resource group’s Memory Tracker is added to this tree structure.

Memory tracking with TCMalloc hooks

2. Soft Isolation for CPU and I/O Resources

Soft isolation for CPU and I/O resources is implemented based on the pipeline execution engine. The pipeline execution engine schedules the execution-unit tasks of all queries in user space.

More specifically, there are execution threads equal to the number of cores, as well as a ready queue for query execution-unit tasks. Each execution thread takes a suitable task from the ready queue and executes it.

A running task can yield the execution thread and return to the ready queue when its time expires or due to other scheduling factors. The execution thread then takes the next suitable task from the ready queue.

Because the pipeline execution engine schedules execution-unit tasks for all queries in user space, it can conveniently implement different scheduling policies.

Hard-limit scheduling across six scheduling periods

WorkGroup priority queue

When resource isolation is disabled, all tasks are scheduled through a multilevel feedback queue.

When resource isolation is enabled, a two-level queue is used for scheduling. This approach is similar to the Completely Fair Scheduler used for Linux processes:

  • the first level is a priority queue of resource groups;
  • the second level is a multilevel feedback queue inside each resource group.

Each time an execution thread selects a suitable task to run, it chooses a task from the resource group with the smallest current Vruntime.

Vruntime is calculated by dividing CPU execution time or I/O execution time by cpu_core_limit.

This method ensures that all resource groups use CPU and I/O resources according to the ratio of their cpu_core_limit.

To reduce scheduling latency as much as possible, the system continuously checks whether the Vruntime of the currently running task is still the smallest. If it is no longer the smallest, the task yields CPU or I/O resources in a timely manner.

To avoid switching tasks too frequently, each task is guaranteed to run for at least 5 milliseconds before yielding.

In addition, if a resource group that has had no tasks for a long time receives new tasks again, its Vruntime may be much smaller than that of other resource groups because those other groups have been running continuously and their Vruntime has kept increasing.

To avoid continuously scheduling tasks from this resource group and causing starvation in other groups, the system adjusts its Vruntime to the current smallest resource group Vruntime, minus a constant value as a reward.

3. Hard CPU and I/O Isolation for Short-Query Resource Groups

For high-priority, very short point queries, short-query resource groups are provided.

When tasks from a short-query resource group are present, other resource groups can use only resources within their hard_limit in each scheduling cycle. If their resource usage exceeds this upper threshold within a cycle, their tasks will no longer be scheduled in that cycle.

However, tasks from multiple resource groups may be running at the same time. When an over-limit condition is detected, their actual resource usage may already have exceeded the limit by a significant amount.

To address this, a penalty strategy is added to ensure overall hard isolation for these resource groups:

  • If their resource usage in the current cycle is between one times and two times the hard_limit, the initial resource usage value for the next cycle is set to the current usage minus the hard_limit. This means they can use only a small amount of resources in the next cycle.
  • If their current usage is more than two times the hard_limit, the initial resource usage value for the next cycle is set to the hard_limit. This means their tasks cannot be scheduled in the next cycle.
ard-limit scheduling across six scheduling periods

#04 Future Roadmap

In summary, to use resource isolation, users first define resource groups and classifiers to specify quotas for different resource types and match tenants and workload types.

When a query is initiated, it is matched to an appropriate resource group through classifiers. CPU and I/O resources are then softly isolated according to the quota of the corresponding resource group. When tasks from a short-query resource group exist, hard isolation is applied to CPU and I/O resources. At the same time, memory and concurrency resources are limited, and the system checks whether large-query circuit breaking is required.

Current development work focuses on task queuing and spill mechanisms to avoid query failures when concurrency resources or memory resources exceed limits.

Future plans also include support for more workload types, including analytical database loading and compaction, as well as support for multiple short-query resource groups to meet a wider range of user scenarios.