Scene Summarization via Motion Normalization

Scott Wehrwein, Kavita Bala, and Noah Snavely

IEEE Transactions on Computer Graphics and Visualization (2021)

TL;DR

We speed up long videos by different amounts, then composite the interesting parts together to form a single, seamless video showing a scene's goings-on at multiple timescales at once.

Quick Links

Abstract

When observing the visual world, temporal phenomena are ubiquitous: people walk, cars drive, rivers flow, clouds drift, and shadows elongate. Some of these, like water splashing and cloud motion, occur over time intervals that are either too short or too long for humans to easily observe. High-speed and timelapse videos provide a popular and compelling way to visualize these phenomena, but many real-world scenes exhibit motions occurring at a variety of rates. Once a framerate is chosen, phenomena at other rates are at best invisible, and at worst create distracting artifacts. In this paper, we propose to automatically normalize the pixel-space speed of different motions in an input video to produce a seamless output with spatiotemporally varying framerate. To achieve this, we propose to analyze scenes at different timescales to isolate and analyze motions that occur at vastly different rates. Our method optionally allows a user to specify additional constraints according to artistic preferences. The motion normalized output provides a novel way to compactly visualize the changes occurring in a scene over a broad range of timescales.

Results - With Annotations

Dam
Beach
Mill
Slope
Atrium
Cater
Geyser
Plaza

Results - Fully Automatic

Dam
Beach
Mill
Slope
Atrium
Cater
Geyser
Plaza

Bibtex

@article{wehrwein_scene_2021,
  author={S. {Wehrwein} and K. {Bala} and N. {Snavely}},
  journal={IEEE Transactions on Visualization and Computer Graphics},
  title={Scene Summarization via Motion Normalization},
  year={2021},
  volume={27},
  number={4},
  pages={2495-2501},
  doi={10.1109/TVCG.2020.2993195}
}