← Back to Interactive Model Experiments
SW

Sliding-Window Streaming VLMs: Full Evaluation on Ego-MC-Bench and Wearable AI

completed videostreaminginteractionrecurrent memorylatency

Qwen3.8-27B and Muse Glimmer 30B run as causal online assistants (an 8-frame sliding window plus a memory they write themselves, replayed in media time) on the Wearable AI validation split (all 2,100 sessions, with the official starter-kit protocol and, for EgoProactive, also the Pro2Bench paper protocol) and on Ego-MC-Bench (39 videos) under the Ego-MC-Bench paper's own evaluation protocol and official evaluator. Only the model's view (window plus memory) differs from each benchmark's protocol. An isolated latency study compares both models with and without reasoning.

Overview

Results at a Glance

Highlighted: our method (8-frame sliding window + self-written memory). Other rows are reference results from the benchmark papers. ↑ higher is better · ↓ lower is better · (count) descriptive, neither.

Benchmark · metricOurs · Qwen3.8-27BOurs · Muse Glimmer 30BReference
Ego-MC per-recipe · IC-Acc↑44.418.4paper zero-shot best 45.5 (Qwen3.5-27B)
Ego-MC per-recipe · mistake F1↑0.090.29paper zero-shot best 0.18; fine-tuned 0.20
Ego-MC full recipe · IC-Acc↑40.616.1paper zero-shot best 30.3
Ego-MC full recipe · mistake F1↑0.090.28paper zero-shot best 0.08
Wearable EgoLongQA · accuracy↑0.6590.520leaderboard zero-shot Qwen2.5-VL-7B 0.59–0.69
Wearable EgoConv · LLM judge↑––pending; leaderboard best 0.59
Wearable EgoProactive · macro F1↑ (starter kit)0.4870.364–
EgoProactive · G-mean F1↑ (Pro2Bench protocol)0.2600.533paper zero-shot best 0.57
Ego-MC-Bench →39 cooking videos, the paper's protocol and official QICD evaluator: setup, prompts, full results, clips. Wearable AI →All 2,100 validation sessions with the official starter kit, plus EgoProactive under the Pro2Bench protocol: setup, prompts, full results, clips.

Shared Setup

Models

  • Models. Qwen3.8-27B (non-thinking mode, greedy decoding) and Muse Glimmer 30B (reasoning_strength = low, the model's official sampling: temperature 1.0, top-p 0.95, top-k 64, seeded per call; only the to=user channel is parsed). On the Pro2Bench protocol both are greedy, as in the paper. BF16, pinned revisions.

Streaming wrapper

  • Window. The video is replayed in media time at one frame per second. Each call sees only the 8 most recent frames (long edge ≤ 672 px): never a future frame, never an unobserved interval.
  • Memory. A small JSON object (facts, completed steps, state, commitments, uncertainties) that the model rewrites on every call, bounded to 512 tokens. It is the only state that survives beyond the window.

Serving and compute

  • Backend. vLLM (the official Muse Glimmer image for both models), one server per GPU, 92 % of GPU memory for weights and KV cache. Calls inside one item stay sequential (each depends on the previous memory); throughput comes from running 40–64 items at once per GPU.
  • Hardware. UW Hyak Klone, mostly preemptible H200 (141 GB) plus A100 80 GB. Items are pulled from a shared queue, so preempted jobs lose no work.
  • Differences from the earlier smoke test. All items instead of 5 per benchmark, only these two benchmarks, vLLM instead of transformers, Muse sampled as its documentation requires, and both benchmarks rerun with their official protocols (Ego-MC paper protocol and QICD evaluator; Wearable AI starter kit and Pro2Bench).

Benchmark-specific data, protocols and prompts are on the Ego-MC-Bench and Wearable AI pages.

Pipeline

Eight stages, from pinned sources to scored predictions. Everything before replay is deterministic and frozen before any model runs.

1Inventoryenv · models · locks 2Normalizeitems · quarantine 3Selectall eligible items 4Acquirefull media sets 5Validatedecode all files 6Replaycausal · media time 7Exportbenchmark format 8Scoreofficial / adapted Steps 1–5 run once as CPU jobs and freeze manifests and locks; 6 runs on vLLM servers (one per GPU) fed by many concurrent item streams; 7–8 run afterwards.

One replay step at media time t: the model gets only what is public by t plus its own memory. At a decision point it answers the benchmark's official prompt; a separate call then rewrites the memory.

Video stream (media time) 0 stend 8 frames ≤ t, 1 fps, nothing from the future Public text ≤ t official prompt · question · dialogue history Previous memory (JSON, ≤ 512 tokens) facts · steps · state · commitments · uncertainties Call instruction decision (benchmark prompt) or memory update VLM, one request Qwen3.8-27B (thinking off) Muse Glimmer 30B (reasoning low) ≤ 4,096 output tokens3.5 s (Qwen) / 6.8 s (Muse) per call Output per call kind decision: benchmark's format text: A–D, reply or $silent$ memory: new JSON (bounded) invalid memory → keep old, log failure memory feeds the next call · reset per item and model (and per step in Ego-MC per-recipe) Benchmark export + score Private refs gold, timings scorer only,never the prompt
  • Causal by construction. Frames are fetched by presentation time (never after t, never from unobserved intervals); questions, steps and probes are released only at their annotated times.
  • Memory is the only long-term state. It is rewritten by the model every call and bounded deterministically; it resets per item and per model, and at every Ego-MC instruction.
  • Gold stays private. Answer keys, reference replies and their timestamps are read by the exporter and scorer only.
  • Resumable. State is checkpointed after every call; preempted GPU jobs are requeued and any worker resumes an abandoned item from its last completed call.

Latency

Isolated study: one vLLM server on one H200, one stream at a time (no batching or queueing), the same 20 items for every configuration (5 per Wearable AI subtask and Ego-MC, first 30 calls each, Wearable AI prompts). The server's first, cold call is excluded. "First answer token" is when the user-visible answer starts: after Muse's to=self reasoning, after Qwen's </think>.

↑ higher is better · ↓ lower is better · (count) descriptive, neither.

Per callQwen3.8-27B · thinking offQwen3.8-27B · thinking onMuse Glimmer 30B · reasoning low
End-to-end latency↓ p50 / p903.52 / 4.37 s15.20 / 25.02 s6.72 / 8.80 s
First answer token↓ p50 / p900.44 / 0.50 s10.94 / 20.68 s3.71 / 5.68 s
First token of any kind↓ p500.435 s–0.489 s
Reasoning tokens before the answer (median) (count)0699202
Output tokens (median) (count)201977391
Decode speed↑ (median)64.7 tok/s–62.5 tok/s
Valid outputs↑599 / 599577 / 585 (1 hit the 4,096-token cap)594 / 598
0 s2 s4 s6 s8 s Ego-MCEgoConvEgoProactiveEgoLongQA 2.766.353.816.89 3.496.813.887.24 Median latency per call (s) · ↓ lower is betterQwen3.8-27BMuse
  • Reasoning, not the model, sets the latency. Prefill and decode speed are almost the same for both models. Muse always reasons in its to=self channel (its template has no off switch), so against non-reasoning Qwen it is 1.9× slower end to end and its answer starts about 3.3 s later. With Qwen's thinking turned on, Qwen reasons 3.5× longer than Muse (699 vs 202 tokens) and its answer starts about 3× later (10.9 s vs 3.7 s).
  • The chart shows end-to-end latency for the default configurations (Qwen thinking off, Muse reasoning low) per subtask.
  • Under load. During the full run, with 40–64 concurrent streams per GPU, median call latency rose to 10.2 s (Qwen) and 16.9 s (Muse) on Ego-MC and to 24.6 s and 38.6 s on Wearable AI. These numbers measure throughput settings, not the model.
  • Real time. Neither configuration keeps up with a 2 s refresh cadence on one H200 even in isolation. On Ego-MC, one 5 s turn takes up to three sequential calls. Latency here is measured under media-time replay, where the stream waits for the model. Only latency was measured for Qwen with thinking on; its benchmark accuracy is not known yet.

Observations and Conclusion

  • The two models sit at opposite operating points. On Ego-MC, Muse flags mistakes often: recall 0.30–0.35 and precision 0.23–0.28, the highest F1 (0.28–0.29) of any model including the paper's, but it confirms few steps (about 160 "Success" messages vs Qwen's 380), so IC-Acc is 16–18. Qwen confirms steps as well as the paper's Qwen3.5-27B (IC-Acc 44.4 vs 45.5) but flags only about 6 % of mistakes. On EgoProactive the operating point depends on the prompt: with the starter-kit prompt Muse interrupts in 98 % of chunks (silent recall 0.02) and Qwen is balanced (0.40 / 0.59); with the Pro2Bench prompt Muse is the balanced one (0.76 / 0.35) and Qwen almost never speaks (interrupt recall 0.06).
  • Reasoning costs time. Muse's reasoning delays its answer by about 3 s per call; Qwen's thinking mode delays it by about 10 s. Whether that time buys accuracy is untested for Qwen.
  • Conclusion. With a sliding window and self-written memory, both models run end to end on full benchmarks under each benchmark's own protocol, match or exceed the zero-shot results of the Ego-MC-Bench paper, and Muse comes close to the best zero-shot Pro2Bench result, but neither is a usable proactive assistant yet: each is strong on one half of the task (Muse catching mistakes, Qwen confirming progress) and per-call latency is well above the refresh cadence. The results describe these models inside this wrapper and configuration, not the models in general.

Next steps

  • Repeat Muse with several seeds and with greedy decoding, to measure how much sampling moves the scores (39 Ego-MC videos is a small set).
  • Run Qwen with thinking on through the full benchmarks, to pair its latency with accuracy.
  • Run the paper's whole-video setting on the same two models, to separate the effect of the window and memory from the effect of the model.
  • Reword Ego-MC step-completion questions to see whether Muse's low IC-Acc is the prompt or the model.
  • Score EgoConv with the LLM judge (jobs queued) and compute Pro2Bench PQS with an LLM judge (inputs already exported).
  • Cut per-sample cost to fit the 300 s rule: sparser memory refresh, and per-call text after the frames so consecutive calls share a cached prefix.

Reproduction

Code: jxb1st/glimmer_qwen_streaming_baseline-code, config configs/full_klone.yaml. On a SLURM cluster with the vLLM Muse Glimmer image:

scripts/download_all.sh all                                    # pinned models + full benchmark data
python -m streaming_smoke {inventory,sample,acquire,validate,estimate} --config configs/full_klone.yaml
scripts/serve_and_run.sh muse30b 1 64                          # one worker per GPU; start as many as available
scripts/serve_and_run.sh qwen27b 1 64
python -m streaming_smoke score  --config configs/full_klone.yaml
python -m streaming_smoke report --config configs/full_klone.yaml
scripts/latency_study.sh muse30b 5 30                          # isolated single-stream latency
scripts/latency_study.sh qwen27b 5 30
scripts/latency_study.sh qwen27b 5 30 configs/full_klone_qwen_thinking.yaml

Benchmark-specific commands are on each benchmark page.