ECCV 2024 / VISUAL NEURAL CODING
Aligning neuronal coding of dynamic visual scenes with foundation vision models
Vi-ST predicts how retinal ganglion cells respond to natural video, and tests whether those predictions transfer to a different scene.
TRAIN MOVIE 1 β TEST MOVIE 2
Dynamic natural scene stimuli and RGC responses
RESPONSE OF
Good CC
Middle CC
Worst CC
Curves show a paper example. Predictions are recovered from the supplementary vector figure.
01 / THE QUESTION
Can a model generalize to a new scene?
Retinal ganglion cells encode visual information as changing firing rates. A model can fit responses to a familiar movie, but transferring to a different movie tests whether it has learned features that remain useful across scenes.
We used recordings from 90 salamander retinal cells presented with two grayscale natural movies. Movie 1 has slower changes and features swimming salamanders; Movie 2 contains more rapidly changing outdoor scenes with tigers and deer.
02 / THE COMPARISON
Predicting responses across movies
The models are trained on one movie and evaluated on the other. Switch direction to compare the mean cell-wise Pearson correlation reported in Table 2.
CROSS-MOVIE PREDICTION
Does the model transfer?
Mean Pearson correlation (CC) over retinal cells. Higher is better. Values are reported in the paper; the two directions are separate trained models.
View Table 2For context: within-movie results
The same-movie evaluation uses an oddβeven split: alternating frames of one movie form the training and test sets. These results use a different evaluation setting and serve as context for the cross-movie transfer task.
| Model | Movie 1 β 1 | Movie 2 β 2 |
|---|---|---|
| CRNN (80 cells) | 0.857 | 0.718 |
| I3D + MSTCN | 0.846 | 0.668 |
| DINOv2 + MSTCN | 0.849 | 0.672 |
| Vi-ST | 0.789 | 0.570 |
Mean correlation from Table 2. CRNN used 80 cells; the other models used 90. Same-movie and cross-movie scores have different test protocols.
03 / THE METHOD
From video frames to neural activity
Vi-ST combines a pretrained visual representation, information from earlier frames and measured receptive fields to predict the population response.
Represent each frame
A frozen DINOv2 ViT-L/14 with registers transforms each frame into a 16 Γ 16 visual feature map.
Combine earlier frames
A causal 3D temporal convolutional network extracts spatiotemporal features, followed by four multi-scale blocks that use different temporal kernel sizes.
Condition on receptive fields
Measured 90 Γ 90 receptive fields are downsampled to the feature grid and used to condition the network through 3D AdaLN Zero.
Training and time scales
The four multi-scale blocks use temporal kernels of (1, 25), (1, 13, 21), (1, 7, 9) and (1, 3, 5) frames. Training uses 128-frame windows; the loss combines root mean squared error, a penalty for negative predictions and SoftDTW terms over 6- and 12-frame windows.
Reading the metrics
Pearson correlation (CC) describes agreement over the response sequence. The paper also proposes SD-KL, which compares distributions of response durations; the two metrics describe different aspects of a prediction.
04 / ABLATION STUDY
What does each component contribute?
Select an ablation experiment to explore how a change to the model affects its response predictions.
Compare the variants
PEARSON CC Β· HIGHER IS BETTERMODEL ARCHITECTURE Β· SELECTED EXPERIMENT
SELECTED EXPERIMENT
Vi-ST baseline
The complete Vi-ST model serves as the reference for the ablation experiments.
What stands out
Removing spikes alignment gives the largest reduction in these experiments: CC falls from approximately 0.3340 to 0.1699. Removing the DINOv2 attention and FFN blocks gives approximately 0.2812 while keeping patchify trainable; removing the video extractor gives approximately 0.2832. Using RMSE as the only loss gives approximately 0.3125.
CC values and changes are approximate.
About the interactive exampleData sources and interpretation
The videos use the original stimulus frames and the measured curves use the recorded firing rates. Predictions were recovered from the paper's supplementary vector figures; line simplification makes them approximate. The example CC is the number printed in the paper.
The example covers frames 0β799 of Movie 2 or 0β899 of Movie 1. The population-level results above come separately from Table 2.
What does the overlay show?
The red overlay marks the dominant part of the selected cell's estimated receptive field: the region of the stimulus that influences its response. The estimate comes from white-noise recordings. For display, its magnitude is smoothed and normalized, and its vertical orientation is corrected to match the video. Its brightness is relative within each cell; it does not indicate response sign or prediction certainty.
Limits of the comparison
The experiments use two movies and 90 selected retinal ganglion cells. The within-movie results use an oddβeven frame split and include a CRNN comparison with 80 cells. Correlation measures agreement in temporal trend; it is not an accuracy percentage.
Retinal recordings and visual stimuli: Onken et al., Dryad dataset β.
CITATION
Cite this work
ECCV 2024 Β· Springer proceedings, 2025
@inproceedings{Wu2025ViST,
title = {Aligning Neuronal Coding of Dynamic Visual Scenes with Foundation Vision Models},
author = {Wu, Rining and Zhou, Feixiang and Yin, Ziwei and Liu, Jian K.},
booktitle = {Computer Vision -- ECCV 2024},
series = {Lecture Notes in Computer Science},
volume = {15146},
pages = {238--254},
year = {2025},
publisher = {Springer Nature Switzerland},
doi = {10.1007/978-3-031-73223-2_14}
}