Literature
Display
← All papers

VLM-Vac: Enhancing Smart Vacuums through VLM Knowledge Distillation and Language-Guided Experience Replay

Reihaneh Mirjalili · Michael Krawez · Florian Walter · Wolfram Burgard

· ICRA 2025 ↗

Robotics5 min text · CondensedOriginal paper ↗
Contents

At a glance

VLM-Vac trains a small YOLOv8n detector to label floor items for collection or avoidance. Low-confidence images are sent to GPT-4o for semantic descriptions and to OWL-ViT for bounding boxes. Language embeddings group these experiences so replay can retain rare items and older floor domains during later training.

Across nine virtual days assembled from real robot images, replay reaches mean F1 = 0.913 over the last six days, versus 0.930 for cumulative retraining, while using 53% less measured GPU training energy. The study evaluates perception and adaptation with a workstation; it does not establish onboard vacuum deployment, end-to-end cleaning success, or total-system energy savings.

I Introdcution

A vacuum must distinguish dirt from objects it should leave alone, including small valuable items or material that would be spread by cleaning. Large vision-language models provide useful semantic supervision, but repeatedly invoking them can be costly. The proposed system transfers their labels into a compact, task-specific detector and revisits selected past experiences as the environment changes.

The contributions are the teacher-to-detector labeling pipeline, language-guided replay, and a dataset collected from a TurtleBot 4. The paper presents dataset/code release as planned; the evaluated evidence comes from the reported image-based experiments.

Refer to caption
Fig. 1: Overview of our smart robotic vacuum cleaner system. The TurtleBot 4 platform, shown on the left, is used in our experiments and resembles a robotic vacuum cleaner. The images on the right, captured by the robot’s camera, illustrate our system’s real-time detection of “suck” or “avoid” actions.

II Related Work

Earlier dirt detectors typically use dedicated visual datasets. Elastic Weight Consolidation and other regularizers address forgetting, while replay preserves earlier examples directly. Domain changes such as new floor patterns motivate replay in this work.

The language grouping follows research on text-guided image clustering and caption-based sampling. Open-vocabulary teachers can recognize broader categories, while knowledge distillation transfers selected capabilities into smaller detectors. VLM-Vac combines these ideas for a two-action perception task rather than training a general-purpose embodied foundation model.

III Method Description

The pipeline has two components: acquire action labels and boxes from large teacher models when the student is uncertain, then train the student using a balanced subset of accumulated experiences. Figure 2 retains the complete prompt and data flow.

Refer to caption
Fig. 2: VLM-Vac in a nutshell: We distill relevant knowledge from a Vision-Language Model (VLM) into a compact action-based object detector. The smaller model queries the VLM whenever it encounters an unfamiliar object or background. The new image, its text description from the VLM, and corresponding bounding boxes from the open vocabulary object detector are stored in the experience pool and later used for training the smaller model. Over time, the smaller model learns from these interactions, adapting to its specific environment and thus reducing the need for VLM queries.

III-A Knowledge Distillation for Action-based Object Classification

The robot image and a short prompt go to GPT-4o, which returns item category, quantity, collect/avoid action, and a description of the floor. OWL-ViT grounds the named item with a box. Images, descriptions, action labels, and boxes enter an experience pool; YOLOv8n is subsequently trained on a selected buffer.

This is distillation through teacher-generated training labels rather than direct matching of hidden features. The operational confidence threshold is 0.9. A high threshold can trigger help for uncertain predictions, but does not identify all unfamiliar inputs: a confidently wrong student can bypass the teacher, and teacher or grounding errors can enter the training data.

Equations 1
t⁡(c,q,a,f)=VLM​(p,I)\mathrm{t}(c,q,a,f)=\text{VLM}(\mathrm{p},I) (1)
ℬ={(I1,a1,B1),…,(IN,aN,BN)}\mathcal{B}=\{(I_{1},a_{1},\mathrm{B}_{1}),\dots,(I_{N},a_{N},\mathrm{B}_{N})\}

III-B Continual Learning with Language-Based Experience Replay

Training only on the newest room can erase earlier knowledge; retraining on the entire pool increases computation. The proposed compromise embeds descriptions containing object, quantity, action, and floor information, clusters them with k-means, and randomly samples within clusters to build the replay buffer.

The intended balance includes both object identity and background, helping retain small or infrequent items. The retained formulas specify the embedding and clustering objective. The paper does not give a complete buffer-capacity or per-cluster sampling-budget specification, limiting exact reproduction from the text alone.

Equations 2–3
en​e​w=Embedding​(cn​e​w,qn​e​w,an​e​w,fn​e​w)e^{new}=\text{Embedding}(c^{new},q^{new},a^{new},f^{new}) (2)
minμ∑k=1K∑e∈𝒞k∥e−μk∥2\min_{\mu}\sum_{k=1}^{K}\sum_{e\in\mathcal{C}_{k}}\|e-\mu_{k}\|^{2} (3)

IV Experimental Results

Experiments test clustering quality, sequential detector learning, GPU training energy, and the fraction of images sent to the VLM. The sequential study uses nine virtual days rather than nine independently observed days of autonomous cleaning.

IV-A Setup and Dataset

A TurtleBot 4 Pro with a downward-tilted OAK-D-PRO RGB-D camera provides 2,500 images covering twelve item categories and three floor patterns. Its resemblance to a vacuum motivates the application, but the evaluated output is an action-labeled bounding box.

The detector and learning pipeline run on a workstation with two RTX 6000 GPUs and a 64-core AMD Threadripper PRO because the robot lacks a dedicated GPU. The authors cite YOLO’s potential edge-device performance; they do not measure it on the vacuum platform in this study.

IV-B Language-based Clustering

On 700 randomly selected images, GPT-4o descriptions are embedded with text-embedding-ada-002 and split into twenty k-means clusters. The visual comparator uses ResNet50 features. Manual cluster labels require both object category and background to agree.

Mean class purity is 93.11% for language embeddings and 74.12% for visual embeddings. Figure 3 shows visual groups dominated by floor texture, while language separates small crumbs and rings despite descriptive variation. This establishes a clustering advantage in the sampled data; the later continual-learning experiment does not include a visual-clustering replay baseline.

Refer to caption
Fig. 3: Sample clusters from kk-means clustering. The top row displays a sample cluster from the vision-based clustering approach, while the bottom row shows two sample clusters from the language-based clustering approach.

IV-C Knowledge Distillation and Continual Learning

Images are distributed over nine virtual days cycling through gray carpet, wood, and zigzag carpet. Each room contains seven categories, with common items repeated and rare categories restricted to one or two rooms. Each day’s F1 is measured before that day’s training. The model then trains for 100 epochs; Figure 4 averages ten random-seed runs with standard-deviation bars.

Naive finetuning warm-starts from the previous model but uses only the newest data. Cumulative learning instead trains from scratch each day on the entire accumulated pool. Replay retains selected old examples. Thus the energy comparison includes both different data volumes and different initialization/training procedures.

Across days 4–9, naive finetuning averages F1 = 0.239 at 26.1 kJ GPU training energy, replay 0.913 at 39.3 kJ, and cumulative learning 0.930 at 83.6 kJ. Replay sacrifices 0.017 F1 relative to cumulative retraining while roughly halving this measured energy. The numbers concern GPU driver power during training on this workstation, not the complete robot, teacher-service, or lifecycle energy budget.

Daily VLM-query percentages fall from 100% on day one to 19.29%, 4.31%, and 12.12% on days seven through nine; individual days are not monotonic. The decreasing three-day average is consistent with adaptation to the repeated rooms. Figure 5 retains all twelve-category examples; its displayed boxes were enlarged for visibility and should not be interpreted as exact localization accuracy. F1 emphasizes action classification rather than a comprehensive bounding-box quality assessment.

Original figure
Fig. 4: The F1F_{1} score for naive fine-tuning, language-based experience replay and cumulative learning across 99 consecutive days averaged over 10 runs with different random seeds. Background shadings represent 3-day intervals. The bars indicate standard deviations.
Refer to caption
Fig. 5: Results of action-based object classification with fine-tuned YOLOv8n on days 77, 88, and 99. The model effectively classifies small objects even in complex backgrounds. To improve visualization, we enlarged the bounding boxes.
Original figure
Fig. 6: GPU energy consumption during training for the language-based experience replay approach, cumulative learning, and naive fine-tuning.
TABLE I: Mean F1F_{1} score and energy consumption over the last 66 days.
Mean F1F_{1} Mean GPU energy consumption (kJ)
Naive fine-tuning (warm start) 0.239 26.1
Language-based experience replay 0.913 39.3
Cumulative learning 0.930 83.6
Original figure
Fig. 7: Percentage of images queried to the VLM over nine consecutive days, with the mean query percentage indicated for each three-day interval.

V Conclusion and Future Work

VLM-Vac shows a useful tradeoff between retaining old domains and retraining on all accumulated data. Its demonstrated scope is a small, repeated set of household objects and backgrounds with remote computation.

Longer deployments, broader objects/floors, and continued learning without loss of plasticity remain open. The authors propose out-of-distribution detection to address confidently wrong predictions and discuss continual-learning plasticity methods. Neither is part of the evaluated pipeline.

Concept demonstration

Open image ↗