Disentangling Where and When

TL;DR

Existing video explanations return one mask that mixes up what the model looks at and when the motion matters. We explain any frozen video classifier with two masks, an appearance mask M_a (where) and a motion mask M_m (when), optimized jointly through a perturbation that removes appearance and motion independently.

3 / 3

datasets with the lowest motion deletion AUC (TSM)

3.5–6.4×

fewer appearance pixels than STEP

0.82

motion F⁻ on EGTEA, vs. 0.12 for STEP + M_m

changes to the classifier: post hoc, any frozen model

Abstract

Post-hoc explanation methods for video action recognition typically conflate spatial appearance and temporal dynamics into a single saliency or importance map. We propose a perturbation-based framework that disentangles these contributions into a spatial appearance mask and a temporal motion mask. The two masks are jointly optimized against a frozen action recognition model through a factored perturbation, which blurs pixels outside the appearance mask and removes motion by replacing frames outside the motion mask with their nearest kept frame. A separation penalty discourages the appearance mask from spending budget on frames that the motion mask has already discarded, and a one-sided area constraint penalizes only explanations larger than their budget. To evaluate these explanations, we introduce a protocol that sweeps spatial pixels and temporal frames independently. Across EPIC-Kitchens-55, EGTEA Gaze+ and Something-Something V2, our method attains the lowest motion deletion AUC on every dataset and the lowest appearance deletion AUC on two of the three with a TSM backbone (all three with TimeSformer), while its appearance masks retain 3.5–6.4× fewer pixels than STEP.

Motivation

One map can't tell you where from when

An action such as put down plate can be recognized from the plate itself or from the downward movement. The two fail in different situations, and a single saliency map cannot tell them apart.

Relies on appearance

Misfires when the plate is occluded by a hand or replaced by a look-alike, such as a flat lid.

Relies on motion

Misfires when something other than a plate is lowered the same way, or the plate is set down unusually.

Method

A perturbation that removes appearance and motion separately

Both masks are optimized for 500 Adam steps against the frozen classifier, so that the perturbed clip keeps the prediction while using as little evidence as possible.

Where: blur outside M_a

Pixels outside the appearance mask are replaced by a heavy Gaussian blur, removing appearance detail while keeping low-frequency context.

When: freeze outside M_m

Frames outside the motion mask are replaced by an exact copy of the nearest kept frame. The choice is hard in the forward pass and differentiable through a straight-through estimator.

Optimize jointly

Separate noun and verb terms (or a flow prior for single-output models), one-sided area budgets, a separation penalty and a smoothness term.

Evaluation sweeps each axis on its own: appearance curves reveal or remove pixels in order of M_a and track the noun (or class); motion curves un-freeze or freeze frames in order of M_m and track the verb (or class).

Qualitative example

Two frames explain pour water

An EGTEA Gaze+ clip predicted as pour water, with rows for the motion mask, the nearest-frame replacement, our appearance mask, STEP and Extremal Perturbation. — An EGTEA Gaze+ clip that a TSM model classifies as *pour water*. The motion mask keeps two of eight frames (t=3, 6), where the hand works the faucet above the pot, and drops the frames showing the whole sink (t=0) and the camera turning away (t=7). On the kept frames, the appearance mask covers the faucet, the pot and the hand; the small region on frozen frame t=5 is never seen by the model, which receives a copy of t=6. STEP and Extremal Perturbation, with a single spatio-temporal mask, spread their evidence over nearly every frame. Each row shows the top 15% of that method's mask values.

Results

Lower deletion AUC with much smaller masks

Deletion AUC (lower is better: removing the explained evidence should destroy the prediction) and appearance mask size, TSM backbone.

Method	EGTEA Gaze+			EPIC-Kitchens-55			Something-Something V2
Method	App. Del. ↓	Mot. Del. ↓	M_a % ↓	App. Del. ↓	Mot. Del. ↓	M_a % ↓	App. Del. ↓	Mot. Del. ↓	M_a % ↓
Extremal Perturbation	.43	–	50.6	.18	–	50.4	.14	–	52.0
EP + M_m	.41	.77	50.5	.15	.50	50.0	.08	.39	52.0
STEP	.51	–	56.4	.23	–	66.1	.17	–	78.9
STEP + M_m	.46	.79	60.6	.20	.62	70.2	.15	.44	79.3
Unified Mask	.43	–	20.2	.22	–	21.1	.18	–	51.1
Ours	.27	.25	12.4	.10	.21	10.3	.11	.11	22.8

Best value per column highlighted. Insertion AUC, fidelity, TimeSformer results, paired Wilcoxon tests and seed variance are in the paper. For binary or near-binary masks, insertion AUC tends to favor larger masks, so our method does not lead on insertion; we treat deletion as the primary measure of localization.

Code

Explain your own model

The method only needs gradients of the classifier's output with respect to its input.

git clone https://github.com/vbhavank/wherewhen && cd wherewhen
pip install -r requirements.txt

# EPIC-Kitchens TSM (weights downloaded automatically), 8-second window of a video
python explain_video.py --model epic --video P01_01.MP4 --start_sec 120 --duration_sec 8 \
    --verb_names EPIC_verb_classes.csv --noun_names EPIC_noun_classes.csv

Any PyTorch classifier works once it is wrapped to take a clip of shape (B, 3, T, H, W) with values in [0, 1]; see examples/custom_model.py. The frame-freezing operator is one PyTorch-only file you can drop into your own project:

from wherewhen.frame_replacement import freeze_frames

# video: (B, C, T, H, W), keep: (B, T) in [0, 1]
frozen, source = freeze_frames(video, keep, return_source=True)
# source[b, t] = the frame each output frame copies; differentiable w.r.t. keep

Documentation: method · usage · hyperparameters

Limitations

What an explanation does and doesn't tell you

A hypothesis, not a certificate. The masks show what the model appears sensitive to; confirm a suspected shortcut by editing the input.
Flow prior for single-output models. It adds its own bias toward frames with visible motion.
Separate perturbations, shared gradients. The verb and noun objectives both reach both masks, so how cleanly the explanations separate depends on the classifier.
Scope. Evidence that is neither appearance nor motion within the clip (audio, off-screen context, long-range structure) is out of scope.

Citation

BibTeX

@inproceedings{vasu2026disentangling,
  title         = {Disentangling Where and When: Factored Spatio-Temporal Explanations for Video Action Recognition},
  author        = {Vasu, Bhavan and Raffa, Giuseppe and Tadepalli, Prasad},
  booktitle     = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year          = {2026},
  eprint        = {2611.11214},
  archivePrefix = {arXiv}
}

This work was funded by Oregon State University and its School of Electrical Engineering and Computer Science. We are grateful to Fuxin Li for valuable advice, and to Devika Venkatesh and Dmitriy Shmidt for their help with computational resources.