How hybrid search, lakehouse data management, and SQL analytics connect raw media to model-development workflows.
By Kang Zhou, Head of the Open-Source OLAP Engine Team at Alibaba Cloud and StarRocks Technical Steering Committee Member
Adapted from Kang Zhou’s presentation at the MirrorShip MIP launch on October 20, 2026. This article has been edited for clarity and readability while preserving the substance of the talk.
An autonomous-driving team discovers that a model struggles when another vehicle cuts in on a rainy night. The next task sounds straightforward: find similar clips in the historical data and work out what needs to change.
In practice, that task crosses several kinds of information. Engineers need the video, descriptions of the scene, vehicle details, model versions, and evaluation records. Finding visually similar clips is useful, but it does not tell them whether those clips cover the right conditions or whether a later model has improved.
That is the problem this architecture addresses: making multimodal data searchable, then connecting the results to the analysis that follows.
At our MIP launch, Kang Zhou explained how StarRocks and Paimon work together across that workflow. His examples covered training-data preparation and retrieval workloads at Alibaba Group, applications on Alibaba Cloud, and an autonomous-driving scenario that brings the pieces together.
From stored files to queryable data
A newly uploaded video tells a storage system relatively little: its location, size, and upload time. Engineers ask different questions. Does it contain pedestrians? Is it raining? Did another vehicle cut into the lane?
Answering those questions requires processing the content. A pipeline parses the original media, generates scene descriptions and labels, and extracts vector embeddings. The original images and videos remain in object storage, while the derived information stays linked to the source objects.
The result is a connected set of representations:
- Text: Descriptions that support keyword and full-text search.
- Labels: Attributes that support filtering.
- Embeddings: Features that support semantic similarity search.
- Object references: Links back to the original media for inspection.
These representations do not all need to live in the same storage system. Frequently accessed data with low-latency requirements can reside in StarRocks native tables. Paimon can manage larger, continuously evolving data collections in lake tables. Where a workflow already uses Lance datasets, access through StarRocks can also be evaluated rather than assuming the data must first be migrated.
The important relationship is between the sample, its derived features, and its business context. Once those are connected, a retrieved clip can lead directly to further questions: Which vehicle produced it? Which model was running? How did that version perform on comparable scenes?
Planning retrieval and analysis together
Consider a request for rainy-night driving clips from a particular vehicle model, collected during the past month. It combines structured filters with content-based retrieval, and the execution plan needs to account for both.
Two design principles guide the architecture: reduce the search space early, and delay expensive reads until they are needed.
Reduce the search space early
If vehicle and time filters can eliminate most of the data, applying them early reduces the work left for retrieval. Running vector calculations across the entire collection, only to discard most results afterward, can waste substantial effort.
The best order depends on the query. The engine needs to consider which conditions are selective, where to obtain candidates, and how many candidates to retain for later ranking. Pruning too aggressively can remove useful candidates before the final ranking stage.
Read larger fields later
During retrieval, the system can work with candidate identifiers and relevance scores. Once the candidate set is smaller, it can fetch the business columns and object references needed for the result.
This is late materialization: postpone reading the larger payload until the query knows which rows matter. The query does not need to load every original video simply to decide which clips are relevant.
In the architecture presented in the talk, the StarRocks frontend plans retrieval paths, while compute nodes carry out candidate processing and ranking within a shared execution pipeline. Structured operations such as filters, joins, and aggregations sit alongside full-text and vector retrieval.
That shared view matters because the query rarely ends at the search result. If the next step is to join clips with evaluation records or aggregate them by model version, the system can plan for that work as part of the same query.
Expressing the request in SQL
The following example preserves the intent of the query shown in the talk: find relevant driving clips within a defined tenant scope and return the top results.
-- Conceptual SQL, not a copy-and-run example.
SELECT
clip_id, oss_uri, model_version
FROM paimon.driving_clips
WHERE tenant_id = 1001
AND scene_text MATCH 'rainy night cut-in'
ORDER BY hybrid_score(
embedding, :query_vector)
LIMIT 20;MATCH, hybrid_score, and the parameter notation illustrate the retrieval logic; exact syntax, scoring behavior, and connector support need to be checked for the target deployment. The query vector is an input to retrieval, so generating embeddings is an upstream step rather than work represented by this query.
Conceptually, execution establishes the data scope and snapshot, combines the relevant filtering and retrieval operations, narrows the candidates, and fetches the remaining fields. With visibility into the whole request, the engine can make trade-offs between early filtering, candidate retention, ranking, and deferred reads.
The same results can then be joined with model metadata, evaluation metrics, and operational records. Engineers can move from finding clips to investigating them without manually exporting each intermediate result between systems.
Paimon manages the lake; StarRocks queries it
For a team working with lake data, the practical question is familiar: can we still filter by time, join model information, and compare results across versions?
Those operations remain central to the design. Paimon manages lake-table data and snapshots, while StarRocks provides the query and analytics layer. Index-based access helps locate relevant records without reading the entire dataset.
A lake table can connect stable business fields with evolving model-generated attributes, text, vectors, and references to media objects. A search result can therefore carry enough context to identify its source, the processing applied to it, and the business object it belongs to.
Locate candidates before reading their payload
Paimon’s Global Index follows a simple idea: first identify where relevant data is likely to be, then read the fields required by the query.
For an exact sample identifier, an index can locate the relevant data. For text or semantic retrieval, the system first obtains candidate identifiers and scores. It can then merge candidates and read the necessary business columns.
The benefit is easiest to see when a query needs only a small result set from a large collection. Reading every payload would be disproportionate to the task. Index-assisted candidate selection and selective reads aim to keep that work focused on relevant records.
Combine different ways of finding the same sample
A driving clip may have a collection timestamp, vehicle attributes, a scene description, and several generations of labels and embeddings. Different questions call for different access paths.
- Exact lookup: Use a known sample identifier.
- Full-text search: Find a specific term or scene description.
- Vector search: Retrieve semantically similar content from a description or reference sample, where the processing pipeline supports it.
- Structured filtering: Restrict any of these searches by vehicle, time, or other business attributes.
These methods become more useful in combination. An engineer might first limit the data to one vehicle model and collection period, then search for a specific scene and inspect how the results are distributed across model versions.
Choosing between Paimon and Lance
The choice starts with the existing workflow and its main access patterns. It should account for how data changes, how it is read, and which tools the team already operates.
For an established Flink and Paimon lakehouse, especially one with continuous updates and business analytics requirements, Paimon is a natural starting point. Multimodal retrieval can extend the existing data pipeline rather than create a separate one.
For workflows centered on Python, Arrow, training datasets, row-oriented access, and feature processing, Lance is worth evaluating. That is particularly relevant when Lance datasets are already part of the training-data pipeline.
The final decision still depends on actual read and write patterns, connector support, and operational costs. A shared StarRocks query entry point can accommodate different storage choices where the required integration is available; it does not require every team to adopt the same lake format.
Workloads at Alibaba Group and on Alibaba Cloud
The talk described applications both within Alibaba Group and across industry workloads on Alibaba Cloud. These examples illustrate different retrieval requirements, rather than a single benchmark or a universal deployment pattern.
A Lake: selecting data before training
In Alibaba Group’s A Lake platform, StarRocks supports data retrieval and sample preparation before training. Paimon wide tables connect multimodal samples with their features, while StarRocks provides hybrid retrieval and filtering.
A training collection may contain text, images, audio, and video, with new labels and features added over time. To prepare a targeted dataset, engineers can restrict the source, quality, and scene conditions, then use keyword or semantic retrieval to identify candidates.
Finding candidates is only part of the task. Engineers also need to examine relevance and coverage: many near-duplicates may add little value, while a small number of samples may cover an important model weakness. Retrieval contributes to dataset selection, with further review determining which samples are suitable for use.
AI Data: large candidate sets and agent observability
The Alibaba Group AI Data platform presented two distinct needs. One involved large-scale candidate retrieval for audience selection, with a workload described as tens of billions of vectors and a top-five-million candidate set.
Those figures describe the workload discussed in the presentation, not a throughput or latency result. The important difference from a typical search interface is the size of the intermediate result: millions of candidates may still require filtering, joins, and analysis after retrieval.
The other need was agent observability. An investigation can begin with a request ID or trace ID, narrow the records by time, tool, and status, then use error text or semantic similarity to locate related failures.
Both cases combine access methods. Exact lookup, structured filtering, full-text search, and vector retrieval address different parts of the same investigation.
Industry applications
The talk also described related needs on Alibaba Cloud: lookup and filtering for automotive cockpit data, training-data organization for smart hardware, and workflows connecting shot analysis, feature generation, and media retrieval in entertainment.
The content differs, but the underlying task is consistent. Teams need to connect original objects, model-generated information, and business records so that retrieval results can support further work.
An autonomous-driving workflow, end to end
Return to the rainy-night cut-in. A model has performed poorly during a road test, and the team needs to understand the failure, find useful data, and assess a revised model.
The data platform supports this process alongside collection, annotation, training, evaluation, and deployment systems. StarRocks and Paimon provide the data-management, retrieval, and analysis capabilities within that broader workflow.
Investigate the failure
An engineer might begin with a driver intervention or a failed evaluation record. The first step is to find the associated clip and connect it with the model version and processing history.
“Rainy-night cut-in” is still a broad description. Inspection may reveal more specific conditions: heavy rain, reflective road surfaces, a short-distance lane entry, and partial occlusion.
Video inspection and structured analysis work together here. Engineers need to see the event and establish when it happened, which vehicle was involved, and which model was running. Comparing related records can also show whether the issue is concentrated in one version or appears across several.
Find comparable historical samples
Manual labels alone may miss useful data. One annotator may use “cut-in,” another “lane entry,” while older clips may lack detailed scene labels altogether.
A combined search can first restrict vehicle and collection time, then use scene descriptions and labels for full-text retrieval and embeddings for semantic similarity. Depending on the models and processing pipeline, the input may be a description such as “a nearby vehicle cutting in on a rainy night” or a reference clip.
The returned set needs analysis as well as inspection. Are most clips from the same route? Do they come from only a few vehicles? Do they cover different rainfall and lighting conditions?
Joins and aggregations help answer those questions. The objective is a candidate dataset with relevant coverage, rather than a large collection of nearly identical clips.
Reuse existing data or collect what is missing
If suitable samples already exist, they can move into annotation review, correction, and quality checks. Original clips collected for an earlier task may become useful again as new descriptions, labels, and features make them easier to find.
If the required conditions are genuinely missing or underrepresented, the team can turn that gap into a targeted collection request. It can specify the weather, lighting, road conditions, and other attributes needed.
An empty search result is not sufficient evidence of a data gap. Engineers first need to check the search scope, labels, features, and processing pipeline. Useful samples may already exist but remain difficult to retrieve.
Track the dataset and evaluate the model
Once samples are ready, the team needs a clear record of what went into training: which clips were included, which annotation versions were used, and which samples were newly added.
A business-level data ID can connect original media, processing records, and annotations. Dataset versions then need to be linked explicitly to model versions.
Paimon snapshots and training-dataset versions serve related but different purposes. A snapshot records the underlying table state; the application still needs to define the training set’s composition and its relationship to a model run.
Evaluation should return to the original failure conditions. Did the revised model improve on rainy-night cut-ins? Did performance regress on other established scenarios?
A scenario library and a failure-case library provide reusable samples and conditions for regression testing. They help the team assess each new version against previously observed problems.
Feed deployment results into the next cycle
After evaluation and deployment checks, a revised model can move through a staged rollout. New field data and failure examples then feed the next round of investigation.
Over time, the linked records also make the development process itself analyzable. Teams can examine how long it takes to turn a failure into a usable training dataset, whether delays come from retrieval or annotation, how much existing data is reused, and which scenarios repeatedly require new collection.
Four questions the data should help answer
Multimodal retrieval becomes useful when it connects content to the decisions that follow. In the driving example, a clip leads to failure analysis, sample selection, dataset preparation, and model evaluation.
Paimon manages data assets and snapshots. StarRocks retrieves samples and analyzes them alongside model, evaluation, and workflow records. Other systems handle collection, annotation, training, and deployment.
Together, that workflow should help engineers answer four concrete questions:
- Where did the problem occur? Connect the event with its scene, vehicle, and model context.
- Where is the relevant data? Find comparable samples across the existing collection.
- Is that data sufficient? Examine coverage before deciding what to reuse or collect.
- Was the improvement validated? Link dataset and model versions to targeted evaluation and regression checks.
The value lies in making existing data easier to use and giving model iteration a more traceable evidence base. Search is one step in that process; the surrounding analysis makes the results actionable.
For more from the launch, visit the MIP event page. The MIP introduction provides the broader platform context.