Literature
Display
← All papers

Chameleon: Fast-Slow Neuro-Symbolic Lane Topology Extraction

Zongzheng Zhang · Xinrun Li · Sizhe Zou · Guoxuan Chi · Siqi Li · Xuchong Qiu · Guoliang Wang · Guantian Zheng · Leichen Wang · Hang Zhao · Hao Zhao

· ICRA 2025 ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

At a glance

Chameleon predicts which lanes connect to each other and which traffic lights or signs govern which lanes. A vision-language model generates scene-conditioned programs for common cases; selected ambiguous cases invoke slower visual question answering. With three annotated example frames in the prompt, it nearly matches supervised lane-to-lane topology prediction and slightly improves one matched lane-to-traffic-element baseline on OpenLane-V2. The few-shot claim applies to topology reasoning: upstream lane and traffic-element detectors remain trained models. The measured slow stage takes seconds per frame, so the paper does not establish real-time operation in difficult scenes.

I Introduction

Detecting lanes and traffic elements is insufficient for mapless driving: the system must infer directed connections and the scope of signs or signals. Pairwise visual questions scale quadratically in the number of lanes. The paper’s example of 100 lanes and 20 traffic elements requires 12,000 questions if every possible pair is queried. Dense visual prompting, exemplified by RedCircle [3], makes individual spatial questions accessible to VLMs but is expensive at this scale. Pure language-generated symbolic programs, exemplified by NS3D [4], can miss visual corner cases.

Chameleon uses visual context during program synthesis, executes inexpensive geometric and symbolic operations, and selectively calls a VLM for ambiguous relationships. The main contribution is this division of work, evaluated with existing detectors and a manually annotated four-task VQA benchmark. Carbon-footprint and monetary savings motivate the design, but neither is directly measured.

Original PDF illustration S1.F1
Fig. 1: VLMs cannot directly address complex 3D scene understanding tasks, such as lane topology extraction. (a) One possible approach is to use dense visual prompting, as in [3], which is accurate but inefficient. (b) Another approach is Neuro-symbolic reasoning, as in [4], which does not effectively leverage visual inputs for program synthesis. (c) Our proposed Chameleon method employs a fast-slow design, where one VLM synthesizes programs and another handles corner cases.
Original PDF illustration S1.F2
Fig. 2: Overview of Chameleon. Given multi-view images as input, the vision models first generate traffic lanes and traffic elements, respectively. The proposed fast system leverages a large Vision Language Model which takes predefined visual-textual few-shot samples and text prompts as inputs, and generates executable codes to process the predictions by the vision models. The proposed slow system consists of a VQA API Set and a Vision Language Model with Chain-of-thought reasoning, where vision prompts and text prompts within the VQA API Set are the inputs of VLM. Subsequently, the topology reasoning results are an combination of code execution results and VLM outputs.

II Related Works

LaneSegNet, TopoNet, TopoMLP [19], and related methods learn structured relationships from annotated maps; TopoLogic adds geometric reasoning. Neuro-symbolic systems such as NS3D, Logicode, and Rekep combine learned perception with executable logic. Chameleon instead conditions program generation on visual examples and exposes some visual questions as callable operations. Large VLMs make this feasible, but road-scene benchmarks such as MAPLM [31] show that spatial understanding remains imperfect. No additional VLM fine-tuning is performed here; this does not remove the supervised training of the visual detectors.

Original PDF illustration S2.F3
Fig. 3: Illustration of Chameleon architecture. Given multi-view images and text prompt as input, Chameleon achieves lane topology extraction. Each API or dense visual prompting VQA task is represented as a node. COT VLM adaptively selects the nodes which are executed to infer the topology results based on input.

III Method

The fast and slow paths share the same detected instances. A generated program proposes topology relationships through geometric APIs and expert constraints; dense visual questions resolve selected cases before the results are combined into adjacency matrices.

III-A Overview

For m lanes and n traffic elements, the outputs are an m × m lane-adjacency matrix and an m × n lane-to-element matrix. Their sparsity motivates avoiding a visual question for every pair. Multi-view images feed trained lane and traffic-element detectors; a VLM receives visual/text examples, API descriptions, and expert rules to generate executable code. Exceptions, explicit VQA calls, and corner-case summaries trigger the slow path. Straight, unobstructed roads are presented as easier cases; intersections with many vehicles, lights, and signs are harder. The paper describes adaptive selection qualitatively, without a calibrated uncertainty threshold or a guarantee that every difficult case is routed correctly.

TABLE I: Performance Comparison of Different Methods. Note that our method sees several topology cases during test time while prior methods are supervised by the topology annotations in the training set of OpenLane-v2[2].
Methods DETl​s\text{DET}_{ls}(%) DETt​e\text{DET}_{te}(%) TOPl​s​l​s\text{TOP}_{lsls} (%) TOPl​s​t​e\text{TOP}_{lste}(%)
Lanesegnet[32] 32.30 N/A 25.40 N/A supervised
TopoLogic[21] 33.0 N/A 30.80 N/A supervised
MapVision[33] 44.00 73.00 40.00 52.00 supervised
Lanesegnet[32]+SMERF[34]+TopoMLP[19] 38.35 54.46 31.87 27.32 supervised
Lanesegnet[32]+SMERF[34]+Ours 38.35 54.46 27.69 25.73 zero-shot
Lanesegnet[32]+SMERF[34]+Ours 38.35 54.46 31.65 27.97 3-shot
Powerful baseline 54.42 67.37 46.59 N/A supervised
Powerful baseline + Ours 54.42 67.37 34.16 45.74 zero-shot
Powerful baseline + Ours 54.42 67.37 46.54 53.05 3-shot

III-B Prompts

API prompts specify function inputs and outputs for operations such as finding the ego vehicle’s lane and parallel lanes. Expert rules constrain lane endpoint distance and direction, and restrict lane-to-traffic-element relationships inside intersections according to the authors’ task convention. Positive and negative examples are rendered in camera view and supplied as coordinate text. Visual-question prompts use both perspective and bird’s-eye views. Table II preserves all four complete prompts, their input images, and example answers; these are the concrete task definitions rather than a general guarantee of traffic-rule understanding.

III-C Code Execution

Lane-to-lane reasoning places VLM-generated Python inside a predefined pairwise framework and executes the returned string with Python’s exec. Lane-to-traffic-element reasoning uses function calling because it requires more APIs. The prompted sequence in Figure 3 locates the ego lane, finds parallel lanes, checks intersection membership and left/right position, identifies connected lanes, and assembles adjacency results. A generated program can skip operations based on the image, while hard visual questions go to the slow VLM. This is a prompted planning procedure, not a learned verifier of the resulting code; the ablation explicitly reports code instability.

III-D Dense Visual Prompting VQA Tasks

The four VQA tasks isolate spatial decisions: Left-or-Right classifies two lane segments as left, right, or unrelated; Is-in-Intersection classifies a lane using perspective and BEV images; Adjacency checks whether two segments connect end to end in the same lane; Vector checks direction compatibility. The vector prompt treats angular deviations under 45° as compatible when ambiguous. These binary or three-way decisions become APIs used by the larger topology program.

IV Experiments

Evaluation combines the official OpenLane-V2 validation topology benchmark [2] with 500 manually annotated samples for the four VQA tasks. All quantitative tables and the four selected corner-case comparisons are retained.

TABLE II: Dense Visual Prompting Examples. Note that this paradigm is inspired by the RedCircle method [3]. But trivially doing this kind of dense visual prompting for lane topology is not practical, so we only use this in the slow system.
Task Input Image Prompt Output
Left-or-Right [Uncaptioned image] Description: In the provided bird’s-eye view (BEV), the black lines in the photos are lane boundaries that are only for references. Color blocks highlighted are different segments of lanes. The colors of the blocks come from green and blue. Question: You are an expert in determining positional relationships of lane segments in the image. Is the green segment on the left of the blue segment, or on the right? Please reply in a brief sentence. The green segment is on the left of the blue segment.
Is-in-Intersection [Uncaptioned image] Description: The provided photo is mosaiced with two images, with the bird’s-eye view (BEV) on the left and the front perspective view (PV) on the right. Normally, the lane segment is considered as not in the intersection when it is in front of the area or at rear of the area. Question: You are an expert in extracting the positional information of lane segments. Let’s determine if the the green segment patch is in the intersection area. Please reply in a brief sentence starting with "Yes" or "No". Yes, the green segment patch is in the intersection area.
Adjacency [Uncaptioned image] Description: In the provided bird’s-eye view (BEV), the black lines in the photos are lane boundaries. Color blocks highlighted are different segments of lanes. Only two lane segments in the same lane end to end adjacent are considered as directly connected. Question: You are an expert in determining adjacency of lane segments. Let’s determine if the the green patch is directly connected with the blue patch. Please reply in a brief sentence starting with "Yes" or "No". Yes, the green patch is directly connected with the blue patch.
Vector [Uncaptioned image] Description: In the provided bird’s-eye view (BEV), the green and blue lane segments are highlighted. The arrow represents the directions of lanes. Normally or when the case is confusing, two directions with deviation of less than 45 degrees are considered as compatible. Question: You are an expert in determining direction relationships of lane segments. Let’s determine if the directions of the two arrows match. Please reply in a brief sentence starting with "Yes" or "No". Yes, the directions of the two arrows match.
TABLE III: VQA Performance Comparison of Different Methods on downsampled OpenLane-V2 [2] validation set
Method Left-or-Right (%) Is-in-Intersection (%) Adjacency (%) Vector (%) Average (%)
MLP (Supervised) 90.20 72.00 97.06 98.04 89.33
GPT-4o 82.43 77.59 88.65 81.30 82.49
GPT-4-vision-preview 67.49 53.43 70.00 78.91 67.46
LLaVA 1.5-13b 53.60 58.72 57.39 50.57 55.07
Original PDF illustration S4.F4
Fig. 4: Qualitative results of TopoMLP and Chameleon(ours) on Openlane-V2 [2] validation dataset. (a) The vehicle has just passed the intersection. (b) There is a left-turn traffic light ahead. (c) The ground lane is marked with a straight-ahead sign. (d) The vehicle is on a one-way right-turn lane. The selected scenes are all corner cases and have undergo further reasoning through dense visual prompting.

IV-A Experiment Settings

Detection is evaluated with mAP for lanes and traffic elements; the official TOP metrics evaluate lane-to-lane and lane-to-element relationships. VQA uses accuracy on annotations described as balancing positive and negative examples. The paper does not report per-task sample counts, class counts for the three-way task, confidence intervals, or uncertainty in the topology scores.

IV-B Implementation Details

The first detector combines LaneSegNet [32] with SMERF’s standard-definition map features [34] and DETR traffic-element detection; TopoMLP supplies the supervised relationship baseline. A stronger baseline adds temporal information from StreamMapNet, a larger VoVNet backbone, and YOLOv8 detection. Only lane-to-lane supervised topology is supplied for that stronger baseline, so its lane-to-element results lack a matched supervised comparator.

Three-shot prompting uses three frames and their annotations. Models are the historical GPT-4-vision-preview and GPT-4o API versions accessed on November 6, 2023 and May 13, 2024, respectively, and LLaVA-v1.5-13b-full_ft-1e. These names and dates identify the evaluated systems, not current API capabilities. The supervised VQA classifier uses a ResNet18 backbone, a 3:1 training/test split, Adam, cross-entropy, and 20 epochs. The main topology system retains extensive supervised detector training despite its few-shot relationship stage.

IV-C Quantitative and Qualitative Results

On the first detector, three-shot Chameleon scores 31.65 lane-to-lane TOP versus the matched supervised baseline’s 31.87, and 27.97 lane-to-element TOP versus 27.32: essentially similar lane connectivity and a 0.65-point gain for element relationships. Zero-shot scores drop to 27.69 and 25.73. With the stronger detector, three-shot scores are 46.54 and 53.05, compared with 46.59 for the matched supervised lane-to-lane baseline. MapVision reports 40.00/52.00, but uses different detectors with different detection accuracy, so the comparison does not isolate the reasoning module.

GPT-4o obtains 82.49% average VQA accuracy versus 89.33% for the supervised classifier, 67.46% for GPT-4-vision-preview, and 55.07% for LLaVA. GPT-4o exceeds the classifier only on intersection membership (77.59% versus 72.00%), while trailing on the other three tasks. Thus the evidence supports useful but imperfect visual reasoning, rather than uniform parity with supervision.

Latency is measured using LLaVA on an RTX 4080, not the more accurate GPT-4o API. Each visual question takes 1,447 ms on average; six calls imply about 8.7 seconds per slow-path frame. Dense pairwise VQA over a 20 × 20 matrix takes over 200 seconds, while TopoMLP takes roughly 140–700 ms. Table IV summarizes Chameleon as 0.1–8 seconds, a rounded range that differs slightly from the textual 8.7-second calculation. The speed improvement over exhaustive questioning is substantial, but the slow path is not real time, and its latency is not a measurement of the model used for the strongest accuracy result.

Figure 4 retains four selected corner cases, each with BEV and front images for TopoMLP and Chameleon. They concern a signal after passing an intersection, a left-turn light governing only the leftmost lane, a straight-ahead road marking, and a right-turn lane unaffected by straight-through lights. Blue marks detected lanes, green true-positive relationships, and pink false positives. These examples illustrate corrected relationships; they do not quantify routing recall or the distribution of failures.

TABLE IV: Data and Inference Efficiency comparision
Method Data Inference Latency
Supervised Model Full annotation 0.1 ∼\sim 0.7s
Dense Visual Prompting Zero-shot Samples >> 200s
Chameleon(Ours) 3-shot Samples 0.1 ∼\sim 8s

IV-D Ablation Study

For lane-to-lane reasoning, API-only program synthesis scores 24.35 TOP. Expert rules raise it to 27.69, and positive/negative few-shot examples raise it to 29.23. Results average three generated programs because of code instability, without reporting spread. The final ablation value differs from the three-shot main-table value of 31.65; the paper does not fully reconcile these settings. The table isolates prompt components, not a matched ablation of slow-path selection, model choice, or the complete fast/slow routing mechanism.

TABLE V: Ablation Study of Prompt Components
Method TOPl​s​l​s\text{TOP}_{lsls} (%)
Pair-wise Prompt to symbolic 24.35
+ Expert Rule 27.69
+ Few-shot examples 29.23

V Conclusion

Chameleon shows that scene-conditioned code and selective visual questions can achieve competitive topology extraction while greatly reducing exhaustive VQA calls. Its strongest evidence is the matched-detector comparison and the prompt-component ablation. Remaining limits include trained-detector and standard-definition-map dependencies, small few-shot/VQA evaluation settings, unstable generated code, imperfect spatial reasoning, unmatched accuracy/latency models, and second-scale corner-case processing. No closed-loop driving experiment, formal traffic-rule correctness guarantee, or measured energy/cost budget establishes deployment readiness.

Concept demonstration

Open image ↗