# Towards Generative Predictive Display for Vision-Based Teleoperation

**A Zero-Shot Benchmark of Off-the-Shelf Video Models**

Aws Khalil and Jaerock Kwon  
Bio-Inspired Machine Intelligence (BIMI) Lab  
University of Michigan-Dearborn

This repository is the public reproducibility package for a paper on generative predictive display for vision-based teleoperation. It benchmarks off-the-shelf video models as zero-shot short-horizon predictors: each method receives 9 observed front-RGB frames and predicts the next 8 frames at 15 FPS.

The release contains the code, configs, manifests, small metric artifacts, and selected figures needed to inspect and reproduce the paper-facing results at several levels of effort.

**Project Page:** [bimilab.github.io/paper-GenPD](https://bimilab.github.io/paper-GenPD/)

## Overview

The paper uses two evaluation stages:

- **Original N=5 benchmark:** five matched Town01 clips at 256x160 and 512x320, evaluating LTX-Video 2B, LTX-Video 13B, Stable Video Diffusion 1.1, Wan I2V 1.3B, and Wan VACE 1.3B.
- **Expanded N=30 validation:** thirty deterministic 512x320 clips from three Town01 source sequences, comparing Persistence, corrected 9-frame History Farneback, and LTX-2B.

## Key Findings

- LTX-2B is the lowest-error generator in the initial N=5 generative-model benchmark.
- In the N=30 follow-up, Persistence and corrected 9-frame History Farneback outperform LTX-2B on average.
- None of the evaluated implementations satisfies both predictive-fidelity and rollout-latency requirements for this short-horizon predictive-display setting.

## Repository Contents

- `src/`: benchmark CLI, data builders, experiment runner, model wrappers, metrics, and utilities.
- `scripts/`: summary, metric, artifact, qualitative-figure, and release-regeneration helpers.
- `configs/models/`: final-paper model configs only.
- `manifests/`: source clip selections, frozen clip manifests, matrix sources, and frozen generated matrices.
- `artifacts/`: small frozen CSV/JSON metric artifacts for lightweight reproduction.
- `images/`: selected paper/project-facing figures.
- `tests/`: CPU-only metric, baseline, manifest, and release-structure tests.
- `docs/`: setup, data, model, evaluation, artifact, and reproduction notes.

## Quickstart: Level 1 Reproduction

Level 1 regenerates lightweight paper-facing quantitative artifacts from included CSV/JSON files. It does not require GPU access, model weights, source data, clip-bank MP4s, or prediction MP4s.

```bash
python scripts/make_release_artifacts.py \
  --artifacts-dir artifacts \
  --out-dir outputs/release_level1
```

Expected outputs include:

- `outputs/release_level1/performance_tradeoff.png`
- `outputs/release_level1/temporal_mad_n30.png`
- `outputs/release_level1/expanded_quality_table.tex`
- `outputs/release_level1/paired_comparison_table.tex`

Run CPU tests with:

```bash
python -m unittest discover -s tests -v
```

## Documentation

- [Setup](docs/SETUP.md)
- [Data](docs/DATA.md)
- [Models](docs/MODELS.md)
- [Evaluation](docs/EVALUATION.md)
- [Reproduction](docs/REPRODUCTION.md)
- [Artifacts](docs/ARTIFACTS.md)

## Reproduction Levels

- **Level 1:** regenerate small tables/plots from included frozen metric artifacts.
- **Level 2:** recompute MAD/SSIM/baseline metrics from locally available prediction and GT videos.
- **Level 3:** rerun model inference using external checkpoints and GPU setup.
- **Level 4:** rebuild the clip bank from external CARLA/MILE/Roach source frames.

See [Reproduction](docs/REPRODUCTION.md) for exact commands and prerequisites.

## External Assets

This repository does not include model weights, external model repositories, CARLA/MILE/Roach source frames, generated prediction MP4s, clip-bank MP4s, or full result trees. Users are responsible for obtaining external data and models and complying with upstream licenses and terms.

## Citation

Manuscript citation:

```bibtex
@article{khalil2026towards,
  title={Towards Generative Predictive Display for Vision-Based Teleoperation: A Zero-Shot Benchmark of Off-the-Shelf Video Models},
  author={Khalil, Aws and Kwon, Jaerock},
  journal={arXiv preprint arXiv:2605.09670},
  year={2026}
}
```

Update this entry after publication.

## License

The original code in this repository is released under the MIT License. See [LICENSE](LICENSE).

External models, datasets, and upstream software remain subject to their respective licenses and terms.