WEBVTT

00:00:01.290 --> 00:00:02.326
Hi, I'm Lyra!

00:00:02.326 --> 00:00:04.397
SiamJEPA predicts a teacher's image patches.

00:00:04.397 --> 00:00:08.541
Scramble those patches, and see what the model is forced to learn.

00:00:10.183 --> 00:00:12.469
A masked model can cheat with position.

00:00:12.469 --> 00:00:15.409
Shuffle the teacher's targets, and that shortcut is gone.

00:00:19.467 --> 00:00:23.341
Each image is masked twice, into two separate sets of visible patches.

00:00:23.341 --> 00:00:24.956
Those are the two views.

00:00:25.376 --> 00:00:27.542
Both views go through one shared encoder.

00:00:27.542 --> 00:00:29.708
That is the Siamese pair of students.

00:00:30.128 --> 00:00:33.773
A slow moving average of the student, the teacher, reads the whole

00:00:33.773 --> 00:00:35.596
image and makes the prediction targets.

00:00:36.016 --> 00:00:37.434
Here is the trick.

00:00:37.434 --> 00:00:40.977
After positions are added, the teacher's patches are randomly shuffled.

00:00:41.397 --> 00:00:43.155
The students predict that scrambled target.

00:00:43.155 --> 00:00:47.256
It only works if each token says what it is, not where it sits.

00:00:48.608 --> 00:00:51.760
With an ordered teacher, slot i always holds patch i.

00:00:51.760 --> 00:00:53.336
A fixed position lookup wins.

00:00:53.336 --> 00:00:54.597
That is the shortcut.

00:00:55.017 --> 00:00:58.547
Shuffle it, and slot i holds a random patch, new every pass.

00:00:58.547 --> 00:01:01.488
The only way to predict it is a semantic token.

00:01:03.275 --> 00:01:04.399
Here is the formula.

00:01:05.275 --> 00:01:09.254
Sim-2 is the prediction loss: match the teacher's latent patch, in

00:01:09.254 --> 00:01:10.701
feature space, not pixels.

00:01:11.121 --> 00:01:16.377
Sim-1 is a small alignment term between the students, weighted zero point zero one.

00:01:16.377 --> 00:01:18.630
It is what makes attention object-centric.

00:01:19.925 --> 00:01:23.408
Probe whether a token still knows its grid position.

00:01:23.408 --> 00:01:26.117
Without shuffling, ninety-four percent; with it, nineteen.

00:01:26.117 --> 00:01:27.665
Class accuracy goes up.

00:01:29.017 --> 00:01:32.643
Shuffle the teacher's targets and position stops being a shortcut, so each

00:01:32.643 --> 00:01:35.061
patch has to encode what the object is.

00:01:37.442 --> 00:01:41.778
To recap: predict latent patches, shuffle the teacher to remove the position

00:01:41.778 --> 00:01:44.668
shortcut, and remember the numbers are still provisional.

00:01:45.088 --> 00:01:47.119
Every source is in the full article.

00:01:47.119 --> 00:01:47.699
I'm Lyra.

00:01:47.699 --> 00:01:47.989
Bye!
