At a glance
VLM-Vac trains a small YOLOv8n detector to label floor items for collection or avoidance. Low-confidence images are sent to GPT-4o for semantic descriptions and to OWL-ViT for bounding boxes. Language embeddings group these experiences so replay can retain rare items and older floor domains during later training.
Across nine virtual days assembled from real robot images, replay reaches mean F1 = 0.913 over the last six days, versus 0.930 for cumulative retraining, while using 53% less measured GPU training energy. The study evaluates perception and adaptation with a workstation; it does not establish onboard vacuum deployment, end-to-end cleaning success, or total-system energy savings.
I Introdcution
A vacuum must distinguish dirt from objects it should leave alone, including small valuable items or material that would be spread by cleaning. Large vision-language models provide useful semantic supervision, but repeatedly invoking them can be costly. The proposed system transfers their labels into a compact, task-specific detector and revisits selected past experiences as the environment changes.
The contributions are the teacher-to-detector labeling pipeline, language-guided replay, and a dataset collected from a TurtleBot 4. The paper presents dataset/code release as planned; the evaluated evidence comes from the reported image-based experiments.
II Related Work
Earlier dirt detectors typically use dedicated visual datasets. Elastic Weight Consolidation and other regularizers address forgetting, while replay preserves earlier examples directly. Domain changes such as new floor patterns motivate replay in this work.
The language grouping follows research on text-guided image clustering and caption-based sampling. Open-vocabulary teachers can recognize broader categories, while knowledge distillation transfers selected capabilities into smaller detectors. VLM-Vac combines these ideas for a two-action perception task rather than training a general-purpose embodied foundation model.
III Method Description
The pipeline has two components: acquire action labels and boxes from large teacher models when the student is uncertain, then train the student using a balanced subset of accumulated experiences. Figure 2 retains the complete prompt and data flow.
III-A Knowledge Distillation for Action-based Object Classification
The robot image and a short prompt go to GPT-4o, which returns item category, quantity, collect/avoid action, and a description of the floor. OWL-ViT grounds the named item with a box. Images, descriptions, action labels, and boxes enter an experience pool; YOLOv8n is subsequently trained on a selected buffer.
This is distillation through teacher-generated training labels rather than direct matching of hidden features. The operational confidence threshold is 0.9. A high threshold can trigger help for uncertain predictions, but does not identify all unfamiliar inputs: a confidently wrong student can bypass the teacher, and teacher or grounding errors can enter the training data.
Equations 1
| (1) |
III-B Continual Learning with Language-Based Experience Replay
Training only on the newest room can erase earlier knowledge; retraining on the entire pool increases computation. The proposed compromise embeds descriptions containing object, quantity, action, and floor information, clusters them with k-means, and randomly samples within clusters to build the replay buffer.
The intended balance includes both object identity and background, helping retain small or infrequent items. The retained formulas specify the embedding and clustering objective. The paper does not give a complete buffer-capacity or per-cluster sampling-budget specification, limiting exact reproduction from the text alone.
Equations 2–3
| (2) |
| (3) |
IV Experimental Results
Experiments test clustering quality, sequential detector learning, GPU training energy, and the fraction of images sent to the VLM. The sequential study uses nine virtual days rather than nine independently observed days of autonomous cleaning.
IV-A Setup and Dataset
A TurtleBot 4 Pro with a downward-tilted OAK-D-PRO RGB-D camera provides 2,500 images covering twelve item categories and three floor patterns. Its resemblance to a vacuum motivates the application, but the evaluated output is an action-labeled bounding box.
The detector and learning pipeline run on a workstation with two RTX 6000 GPUs and a 64-core AMD Threadripper PRO because the robot lacks a dedicated GPU. The authors cite YOLO’s potential edge-device performance; they do not measure it on the vacuum platform in this study.
IV-B Language-based Clustering
On 700 randomly selected images, GPT-4o descriptions are embedded with text-embedding-ada-002 and split into twenty k-means clusters. The visual comparator uses ResNet50 features. Manual cluster labels require both object category and background to agree.
Mean class purity is 93.11% for language embeddings and 74.12% for visual embeddings. Figure 3 shows visual groups dominated by floor texture, while language separates small crumbs and rings despite descriptive variation. This establishes a clustering advantage in the sampled data; the later continual-learning experiment does not include a visual-clustering replay baseline.
IV-C Knowledge Distillation and Continual Learning
Images are distributed over nine virtual days cycling through gray carpet, wood, and zigzag carpet. Each room contains seven categories, with common items repeated and rare categories restricted to one or two rooms. Each day’s F1 is measured before that day’s training. The model then trains for 100 epochs; Figure 4 averages ten random-seed runs with standard-deviation bars.
Naive finetuning warm-starts from the previous model but uses only the newest data. Cumulative learning instead trains from scratch each day on the entire accumulated pool. Replay retains selected old examples. Thus the energy comparison includes both different data volumes and different initialization/training procedures.
Across days 4–9, naive finetuning averages F1 = 0.239 at 26.1 kJ GPU training energy, replay 0.913 at 39.3 kJ, and cumulative learning 0.930 at 83.6 kJ. Replay sacrifices 0.017 F1 relative to cumulative retraining while roughly halving this measured energy. The numbers concern GPU driver power during training on this workstation, not the complete robot, teacher-service, or lifecycle energy budget.
Daily VLM-query percentages fall from 100% on day one to 19.29%, 4.31%, and 12.12% on days seven through nine; individual days are not monotonic. The decreasing three-day average is consistent with adaptation to the repeated rooms. Figure 5 retains all twelve-category examples; its displayed boxes were enlarged for visibility and should not be interpreted as exact localization accuracy. F1 emphasizes action classification rather than a comprehensive bounding-box quality assessment.
| Mean | Mean GPU energy consumption (kJ) | |
|---|---|---|
| Naive fine-tuning (warm start) | 0.239 | 26.1 |
| Language-based experience replay | 0.913 | 39.3 |
| Cumulative learning | 0.930 | 83.6 |
V Conclusion and Future Work
VLM-Vac shows a useful tradeoff between retaining old domains and retraining on all accumulated data. Its demonstrated scope is a small, repeated set of household objects and backgrounds with remote computation.
Longer deployments, broader objects/floors, and continued learning without loss of plasticity remain open. The authors propose out-of-distribution detection to address confidently wrong predictions and discuss continual-learning plasticity methods. Neither is part of the evaluated pipeline.