Sliding-Window Streaming VLMs on Ego-MC-Bench

Setup and Protocol

Models, sliding window, memory, serving and the latency study are shared by both benchmarks: see the overview page.

Data

• Ego-MC-Bench. 40 annotated cooking videos; 39 usable (GX010039 has no instruction in its annotation, so no step can be posed).

Ego-MC evaluation protocol (as in the paper)

• Source. Ego-MC-Bench paper (Bhattacharyya et al., arXiv 2606.09547, Sec. 5 and App. D–F) and its project page, which lists the official QICD evaluator as its eval code.
• Turns. Every 5 s the model is asked whether the current recipe step is complete (YES/NO + why); if not, the detailed mistake prompt (YES. <feedback> / NO) with the feedbacks of the last minute as already-detected mistakes. Both prompts verbatim from App. D, no system prompt.
• Settings. Per-recipe step: each step starts at its ground-truth time with only its instruction and ends when the model confirms completion (else at the next step); window, memory and feedback history reset per step. Full recipe: one continuous stream, instructions released at their ground-truth times regardless of the model.
• Scoring. The official QICD eval.py (commit a9f73f7), unmodified: 30 s detection window (±15 s), BERTScore rescaled with baseline. Ego-MC annotations are mapped to its input format (Feedback → mistake, Success → success; user turns dropped).
• Our variable. Instead of the whole past video, each call sees the 8 most recent frames plus the model's own memory, refreshed by a separate memory-update call at the end of every turn. Muse sampled with its official settings, Qwen greedy, as on Wearable AI.
• Parsing. A leading YES/NO is used as is; otherwise the first standalone upper-case YES/NO (Qwen often answers the completion prompt "Based on the frames … NO"); otherwise the call counts as failed.

Prompts

You are an expert cooking assistant who is observing a person cook.

##INSTRUCTIONS:

The person is currently at the following recipe step: [recipe_step]. Has the person already completed the recipe step? If the person has completed the recipe step answer ‘YES’ else answer ‘NO’. If you answer ‘YES’ describe why you think the person already completed the recipe step.

You are an expert cooking assistant who is observing a person cook. You should look out for mistakes made by the person.

##INSTRUCTIONS:
The person is trying to complete the following recipe step: [recipe_step]. Your task is to check if the person is about to make or has already made a mistake. Mistakes occur when the person performs actions that deviates from the instruction and DIRECTLY INTERFERES WITH SUCCESSFUL INSTRUCTION COMPLETION. Do not penalize actions that do not directly interfere with instruction completion (e.g. washing broccoli before cutting, if the instruction just says ‘cut broccoli’).

Here are some common types of mistakes that you should look out for:
1. Technique Error: A mistake in how a step is physically performed. Examples include chopping with the wrong motion, stirring when folding is required, or spilling during transfer, producing uneven cuts or texture issues even when tools and amounts are right. Ignore minor technique errors such as not holding or gripping objects properly, holding with a risk of dropping etc. that do not interfere directly with recipe completion.
2. Preparation Error: A setup mistake before executing the step . Using the wrong or dirty utensil, not washing/peeling/draining ingredients, insufficient draining of fluid, cutting/chopping without peeling which makes correct execution difficult or unsafe.
3. Measurement Error: An error in quantity — wrong counts, volumes, weights, or units. Mixing up teaspoons and tablespoons, misreading a scale, or miscounting items leads to off ratios and predictable taste or texture problems.
4. Temperature Error: A mistake in heat level or thermal state — the applied temperature, starting temperature, or thermal transition is wrong. Not preheating, using the wrong microwave power, overheating oil, or adding cold liquid when warm is required often causes burning, undercooking, or split emulsions.
5. Timing Error: A mistake in duration – over- or under-doing a step or skipping required rests, proofs, or cooling periods. Overcooking, underblending, or cutting resting time short typically yields incorrect doneness or unstable textures.

Do not repeat previously detected mistakes. Here are the feedbacks corresponding to previously detected mistakes:[prev_detected_mistakes]. Assume the recipe step is still in progress. Your task is to identify any mistake that’s already visible in the partially completed step. Do no penalize partially competed recipe steps. If you observe a mistake answer ‘YES’, else ‘No’. Your response MUST BEGIN WITH ‘YES’ or ‘NO’. In case you answer ‘YES’, please follow with a concise feedback to the user describing the mistake (i.e. YES. <feedback>.). Directly address the person.

You are watching a live cooking stream but only ever see the most recent frames (each labeled with its time in seconds) plus a short written memory that you yourself wrote on the previous turn. Nothing else about earlier footage survives, so the memory must carry everything you will need later: what the person has done for the current recipe step, quantities and counts, the current state, and mistakes already pointed out.
Current recipe step: [recipe_step].
Feedback already given to the person in the last minute: [prev_detected_mistakes].
Hard limits: observed_facts at most 8 entries, completed_steps at most 6, pending_commitments at most 3, uncertainties at most 3; every entry at most 15 words and prefixed with its time (e.g. "12s: ..."); current_state at most 40 words. Merge or drop older entries instead of exceeding these limits. Describe only what the frames support.
Reply with ONE JSON object and nothing else, in exactly this shape:
{"observed_facts": [...], "completed_steps": [...], "current_state": "...", "pending_commitments": [...], "uncertainties": [...]}

Most recent frames (8), oldest first:
[frame at 2.2s] <image> … [frame at 9.2s] <image>
Your memory of earlier footage (written by you on the previous turn):
{"observed_facts": [...], "completed_steps": [...], "current_state": "...", "pending_commitments": [...], "uncertainties": [...]}

 to=self<|message|>Your memory of earlier footage...
We need check if person is about to make or has already made a mistake. Step: get a baguette and slice four half-inch thick pieces from it.

Frames: 5s hands together, 6s same, 7s hands move to baguette, 8s hands on baguette, 9s hands holding bag, 9.2s same.

Person has not yet sliced. They are handling baguette still in paper bag. Mistake? Might be not removing baguette from bag before slicing. The instruction: get a baguette and slice... Might be okay to slice in bag? No. They are manipulating bag. Could be a preparation error: slicing with bag still on? At 7-9s they grasp bag, not baguette. Might be about to slice while still in paper. That would be technique error? Probably a mistake: baguette is still in paper bag, can't slice properly. That directly interferes.

So answer YES. Feedback: baguette still in paper bag, need to remove.

<|eom|><|start|>assistant to=user<|message|>YES. You’re still holding the baguette inside its paper bag and haven’t taken the loaf out onto the cutting board. Slicing four half-inch pieces isn’t possible while it’s wrapped, so remove the baguette from the paper first and place it on the board before cutting.<|eot|>

No. The person is correctly preparing to slice the baguette, and no mistakes have been made yet.<|im_end|>

Quantitative Results

Highlighted: our method (8-frame sliding window + self-written memory). Other rows are reference results from the benchmark papers. ↑ higher is better · ↓ lower is better · (count) descriptive, neither.

Ego-MC-Bench · 39 videos · paper protocol · official QICD evaluator

Paper rows are the zero-shot regular-interval results (Table 2) and the best fine-tuned model (Table 3). Neither of our models appears in the paper, so the comparison mixes model and wrapper differences. IC-Acc credits a mistake-free step only if the single output in it is "Success" within the window, so Muse's frequent feedback also costs it IC-Acc. The official code counts the "Success" that closes a step with mistakes as a missed mistake, which lowers recall for every model alike. BERT and ROUGE-L are computed on true positives only and are not comparable across models with different recall (paper App. F). An earlier Ego-MC run with our own protocol and a ported evaluator is superseded by these numbers.

Visualization

Three videos picked at random with a fixed seed, not chosen for outcome. Each clip shows both models: segments around gold events, decision points and model utterances, with the media time stamped top-left. Clips come from the per-recipe-step paper-protocol run. 480p, no audio.

Observations

• The two models sit at opposite operating points. On Ego-MC, Muse flags mistakes often: recall 0.30–0.35 and precision 0.23–0.28, the highest F1 (0.28–0.29) of any model including the paper's, but it confirms few steps (about 160 "Success" messages vs Qwen's 380), so IC-Acc is 16–18. Qwen confirms steps as well as the paper's Qwen3.5-27B (IC-Acc 44.4 vs 45.5) but flags only about 6 % of mistakes.
• The window holds up over long recipes. The paper's models lose much of their score going from per-recipe to full recipe (e.g. Qwen3.5-27B IC-Acc 45.5 → 30.3); ours barely change (Qwen 44.4 → 40.6, Muse F1 0.29 → 0.28). A plausible reason is that a fixed 8-frame window never fills up with stale video, but this run does not isolate it.

Reproduction

Code: jxb1st/glimmer_qwen_streaming_baseline-code, config configs/full_klone.yaml. On a SLURM cluster with the vLLM Muse Glimmer image:

# Ego-MC with the paper's protocol (official QICD evaluator, reference_code/qualcomm_interactive_cooking_eval @ a9f73f7)
scripts/serve_and_run.sh muse30b 1 10 configs/egomc_paper_per_recipe.yaml   # same for qwen27b and egomc_paper_full_recipe.yaml
python -m streaming_smoke score --config configs/egomc_paper_per_recipe.yaml