At a glance
Chameleon predicts which lanes connect to each other and which traffic lights or signs govern which lanes. A vision-language model generates scene-conditioned programs for common cases; selected ambiguous cases invoke slower visual question answering. With three annotated example frames in the prompt, it nearly matches supervised lane-to-lane topology prediction and slightly improves one matched lane-to-traffic-element baseline on OpenLane-V2. The few-shot claim applies to topology reasoning: upstream lane and traffic-element detectors remain trained models. The measured slow stage takes seconds per frame, so the paper does not establish real-time operation in difficult scenes.
I Introduction
Detecting lanes and traffic elements is insufficient for mapless driving: the system must infer directed connections and the scope of signs or signals. Pairwise visual questions scale quadratically in the number of lanes. The paper’s example of 100 lanes and 20 traffic elements requires 12,000 questions if every possible pair is queried. Dense visual prompting, exemplified by RedCircle [3], makes individual spatial questions accessible to VLMs but is expensive at this scale. Pure language-generated symbolic programs, exemplified by NS3D [4], can miss visual corner cases.
Chameleon uses visual context during program synthesis, executes inexpensive geometric and symbolic operations, and selectively calls a VLM for ambiguous relationships. The main contribution is this division of work, evaluated with existing detectors and a manually annotated four-task VQA benchmark. Carbon-footprint and monetary savings motivate the design, but neither is directly measured.
II Related Works
LaneSegNet, TopoNet, TopoMLP [19], and related methods learn structured relationships from annotated maps; TopoLogic adds geometric reasoning. Neuro-symbolic systems such as NS3D, Logicode, and Rekep combine learned perception with executable logic. Chameleon instead conditions program generation on visual examples and exposes some visual questions as callable operations. Large VLMs make this feasible, but road-scene benchmarks such as MAPLM [31] show that spatial understanding remains imperfect. No additional VLM fine-tuning is performed here; this does not remove the supervised training of the visual detectors.
III Method
The fast and slow paths share the same detected instances. A generated program proposes topology relationships through geometric APIs and expert constraints; dense visual questions resolve selected cases before the results are combined into adjacency matrices.
III-A Overview
For m lanes and n traffic elements, the outputs are an m × m lane-adjacency matrix and an m × n lane-to-element matrix. Their sparsity motivates avoiding a visual question for every pair. Multi-view images feed trained lane and traffic-element detectors; a VLM receives visual/text examples, API descriptions, and expert rules to generate executable code. Exceptions, explicit VQA calls, and corner-case summaries trigger the slow path. Straight, unobstructed roads are presented as easier cases; intersections with many vehicles, lights, and signs are harder. The paper describes adaptive selection qualitatively, without a calibrated uncertainty threshold or a guarantee that every difficult case is routed correctly.
| Methods | (%) | (%) | (%) | (%) | |
| Lanesegnet[32] | 32.30 | N/A | 25.40 | N/A | supervised |
| TopoLogic[21] | 33.0 | N/A | 30.80 | N/A | supervised |
| MapVision[33] | 44.00 | 73.00 | 40.00 | 52.00 | supervised |
| Lanesegnet[32]+SMERF[34]+TopoMLP[19] | 38.35 | 54.46 | 31.87 | 27.32 | supervised |
| Lanesegnet[32]+SMERF[34]+Ours | 38.35 | 54.46 | 27.69 | 25.73 | zero-shot |
| Lanesegnet[32]+SMERF[34]+Ours | 38.35 | 54.46 | 31.65 | 27.97 | 3-shot |
| Powerful baseline | 54.42 | 67.37 | 46.59 | N/A | supervised |
| Powerful baseline + Ours | 54.42 | 67.37 | 34.16 | 45.74 | zero-shot |
| Powerful baseline + Ours | 54.42 | 67.37 | 46.54 | 53.05 | 3-shot |
III-B Prompts
API prompts specify function inputs and outputs for operations such as finding the ego vehicle’s lane and parallel lanes. Expert rules constrain lane endpoint distance and direction, and restrict lane-to-traffic-element relationships inside intersections according to the authors’ task convention. Positive and negative examples are rendered in camera view and supplied as coordinate text. Visual-question prompts use both perspective and bird’s-eye views. Table II preserves all four complete prompts, their input images, and example answers; these are the concrete task definitions rather than a general guarantee of traffic-rule understanding.
III-C Code Execution
Lane-to-lane reasoning places VLM-generated Python inside a predefined pairwise framework and executes the returned string with Python’s exec. Lane-to-traffic-element reasoning uses function calling because it requires more APIs. The prompted sequence in Figure 3 locates the ego lane, finds parallel lanes, checks intersection membership and left/right position, identifies connected lanes, and assembles adjacency results. A generated program can skip operations based on the image, while hard visual questions go to the slow VLM. This is a prompted planning procedure, not a learned verifier of the resulting code; the ablation explicitly reports code instability.
III-D Dense Visual Prompting VQA Tasks
The four VQA tasks isolate spatial decisions: Left-or-Right classifies two lane segments as left, right, or unrelated; Is-in-Intersection classifies a lane using perspective and BEV images; Adjacency checks whether two segments connect end to end in the same lane; Vector checks direction compatibility. The vector prompt treats angular deviations under 45° as compatible when ambiguous. These binary or three-way decisions become APIs used by the larger topology program.
IV Experiments
Evaluation combines the official OpenLane-V2 validation topology benchmark [2] with 500 manually annotated samples for the four VQA tasks. All quantitative tables and the four selected corner-case comparisons are retained.
| Task | Input Image | Prompt | Output |
| Left-or-Right |
|
Description: In the provided bird’s-eye view (BEV), the black lines in the photos are lane boundaries that are only for references. Color blocks highlighted are different segments of lanes. The colors of the blocks come from green and blue. Question: You are an expert in determining positional relationships of lane segments in the image. Is the green segment on the left of the blue segment, or on the right? Please reply in a brief sentence. | The green segment is on the left of the blue segment. |
| Is-in-Intersection |
|
Description: The provided photo is mosaiced with two images, with the bird’s-eye view (BEV) on the left and the front perspective view (PV) on the right. Normally, the lane segment is considered as not in the intersection when it is in front of the area or at rear of the area. Question: You are an expert in extracting the positional information of lane segments. Let’s determine if the the green segment patch is in the intersection area. Please reply in a brief sentence starting with "Yes" or "No". | Yes, the green segment patch is in the intersection area. |
| Adjacency |
|
Description: In the provided bird’s-eye view (BEV), the black lines in the photos are lane boundaries. Color blocks highlighted are different segments of lanes. Only two lane segments in the same lane end to end adjacent are considered as directly connected. Question: You are an expert in determining adjacency of lane segments. Let’s determine if the the green patch is directly connected with the blue patch. Please reply in a brief sentence starting with "Yes" or "No". | Yes, the green patch is directly connected with the blue patch. |
| Vector |
|
Description: In the provided bird’s-eye view (BEV), the green and blue lane segments are highlighted. The arrow represents the directions of lanes. Normally or when the case is confusing, two directions with deviation of less than 45 degrees are considered as compatible. Question: You are an expert in determining direction relationships of lane segments. Let’s determine if the directions of the two arrows match. Please reply in a brief sentence starting with "Yes" or "No". | Yes, the directions of the two arrows match. |
| Method | Left-or-Right (%) | Is-in-Intersection (%) | Adjacency (%) | Vector (%) | Average (%) |
| MLP (Supervised) | 90.20 | 72.00 | 97.06 | 98.04 | 89.33 |
| GPT-4o | 82.43 | 77.59 | 88.65 | 81.30 | 82.49 |
| GPT-4-vision-preview | 67.49 | 53.43 | 70.00 | 78.91 | 67.46 |
| LLaVA 1.5-13b | 53.60 | 58.72 | 57.39 | 50.57 | 55.07 |
IV-A Experiment Settings
Detection is evaluated with mAP for lanes and traffic elements; the official TOP metrics evaluate lane-to-lane and lane-to-element relationships. VQA uses accuracy on annotations described as balancing positive and negative examples. The paper does not report per-task sample counts, class counts for the three-way task, confidence intervals, or uncertainty in the topology scores.
IV-B Implementation Details
The first detector combines LaneSegNet [32] with SMERF’s standard-definition map features [34] and DETR traffic-element detection; TopoMLP supplies the supervised relationship baseline. A stronger baseline adds temporal information from StreamMapNet, a larger VoVNet backbone, and YOLOv8 detection. Only lane-to-lane supervised topology is supplied for that stronger baseline, so its lane-to-element results lack a matched supervised comparator.
Three-shot prompting uses three frames and their annotations. Models are the historical GPT-4-vision-preview and GPT-4o API versions accessed on November 6, 2023 and May 13, 2024, respectively, and LLaVA-v1.5-13b-full_ft-1e. These names and dates identify the evaluated systems, not current API capabilities. The supervised VQA classifier uses a ResNet18 backbone, a 3:1 training/test split, Adam, cross-entropy, and 20 epochs. The main topology system retains extensive supervised detector training despite its few-shot relationship stage.
IV-C Quantitative and Qualitative Results
On the first detector, three-shot Chameleon scores 31.65 lane-to-lane TOP versus the matched supervised baseline’s 31.87, and 27.97 lane-to-element TOP versus 27.32: essentially similar lane connectivity and a 0.65-point gain for element relationships. Zero-shot scores drop to 27.69 and 25.73. With the stronger detector, three-shot scores are 46.54 and 53.05, compared with 46.59 for the matched supervised lane-to-lane baseline. MapVision reports 40.00/52.00, but uses different detectors with different detection accuracy, so the comparison does not isolate the reasoning module.
GPT-4o obtains 82.49% average VQA accuracy versus 89.33% for the supervised classifier, 67.46% for GPT-4-vision-preview, and 55.07% for LLaVA. GPT-4o exceeds the classifier only on intersection membership (77.59% versus 72.00%), while trailing on the other three tasks. Thus the evidence supports useful but imperfect visual reasoning, rather than uniform parity with supervision.
Latency is measured using LLaVA on an RTX 4080, not the more accurate GPT-4o API. Each visual question takes 1,447 ms on average; six calls imply about 8.7 seconds per slow-path frame. Dense pairwise VQA over a 20 × 20 matrix takes over 200 seconds, while TopoMLP takes roughly 140–700 ms. Table IV summarizes Chameleon as 0.1–8 seconds, a rounded range that differs slightly from the textual 8.7-second calculation. The speed improvement over exhaustive questioning is substantial, but the slow path is not real time, and its latency is not a measurement of the model used for the strongest accuracy result.
Figure 4 retains four selected corner cases, each with BEV and front images for TopoMLP and Chameleon. They concern a signal after passing an intersection, a left-turn light governing only the leftmost lane, a straight-ahead road marking, and a right-turn lane unaffected by straight-through lights. Blue marks detected lanes, green true-positive relationships, and pink false positives. These examples illustrate corrected relationships; they do not quantify routing recall or the distribution of failures.
| Method | Data | Inference Latency |
| Supervised Model | Full annotation | 0.1 0.7s |
| Dense Visual Prompting | Zero-shot Samples | 200s |
| Chameleon(Ours) | 3-shot Samples | 0.1 8s |
IV-D Ablation Study
For lane-to-lane reasoning, API-only program synthesis scores 24.35 TOP. Expert rules raise it to 27.69, and positive/negative few-shot examples raise it to 29.23. Results average three generated programs because of code instability, without reporting spread. The final ablation value differs from the three-shot main-table value of 31.65; the paper does not fully reconcile these settings. The table isolates prompt components, not a matched ablation of slow-path selection, model choice, or the complete fast/slow routing mechanism.
| Method | (%) |
| Pair-wise Prompt to symbolic | 24.35 |
| + Expert Rule | 27.69 |
| + Few-shot examples | 29.23 |
V Conclusion
Chameleon shows that scene-conditioned code and selective visual questions can achieve competitive topology extraction while greatly reducing exhaustive VQA calls. Its strongest evidence is the matched-detector comparison and the prompt-component ablation. Remaining limits include trained-detector and standard-definition-map dependencies, small few-shot/VQA evaluation settings, unstable generated code, imperfect spatial reasoning, unmatched accuracy/latency models, and second-scale corner-case processing. No closed-loop driving experiment, formal traffic-rule correctness guarantee, or measured energy/cost budget establishes deployment readiness.