Emerging Properties in Self-Supervised Vision Transformers (DINO)

 Why:

It has information about semantic properties of image better than normal ViT , It achieves good accuracy on K-NN classifiers which means representations of different class are well separated for final classification

How:

Similar to typical contrastive learning different augmented views passed through the 2 (student and teacher) network and then student network tries (learn) to match the probability distribution of teacher’s network

What:

It is called as DINO (self-distillation with no labels) instead of learning the difference between the representation here it’s about mapping the representation

TL: DR

1.Augmentation of images (multi-crop, Gaussian-blurr,etc)

2. All views will be passed through student network and only global view will be passed through teacher’s network

3. For given image , V different views can be generated (It will have at least 2 global views)

4.Student and Teacher network shares same architecture (ViT or ConvNet)

4. Output of network outputs k-dimension distribution which is passed through soft-max called as (s1,s2 (Student) & t1,t2 (teacher)) (Ex: if we only have 2 views)

5. Loss = CE(s1,t2 )+ CE(s2,t1), where CE -> Cross entropy (Ex: if we only have 2 views)

General loss expression

6. For student -> soft-max (Calculated using temperature parameter also called as sharpening ) i.e (s/tmp), where s-> output of the student’s network

7 . For Teacher → soft-max(sharpening & Centering)

Centering -> exponential moving avg , c=m*c+(1-m)*mean(t1,t2)

8. Hence for teacher, t=soft-max((t-c)/tmp), where t-> output of the teacher’s network

9. No gradient updates for teacher network it updates through student network via exponential moving average ,theta(t) = lambda * theta(t) +(1-lamba)*theta(s)

Architecture

Points to Ponder:

1. What is the intuition behind distillation network in general?

2. What is the intuition for update strategy of teacher network?

3. Why it doesn’t work well without centering block in teacher’s network as mentioned in the paper?

link to the paper : https://arxiv.org/pdf/2104.14294.pdf

Comments

Post a Comment

Popular posts from this blog

NeurIPS 2021 - Curated papers - Part 2

Image
Link to my deep learning blogs :  https://rakshithv-deeplearning.blogspot.com/ Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training This work aims at Vision-Language Pre-training (VLP) or multi-modal learning through pre-training. 1.Given an image and text , image will be passed through Vision Transformer (ViT) and output of the ViT is taken as visual tokens. For text, tokens are generated from BERT tokenizer. 2. [Cls]_[Visual tokens]_[SEP]_[Text tokens] are concatenated and masks are generated similar to BERT. 3. Concatenated text and image token is given to the Multi-modal transformer (MT) 4. Pre-training has 3 objective functions: a. MLM (Masked Language Modelling) : Almost similar to BERT , predicting the masked token in the text. b. ITM (Image-Text Matching) : Replacing image with other image with 0.5 probability and binary classification (Same image or different image) is used to learn the inter-modal alignment. c. MFR (Masked Feature Re...
Read more

DALL-E (Zero-Shot Text-to-Image Generation) -PART(2/2)

Image
  Link to my deep learning blogs : https://rakshithv-deeplearning.blogspot.com/ DALL-E consist of two components. First component is d-VAE(discrete-Variational Auto Encoder) and second is Auto-regressive transformer. First component is responsible for generating a tokens of size 1024 for an image of size 224x224. More details on this was covered in part-1 https://rakshithv-deeplearning.blogspot.com/2022/04/dall-e-zero-shot-text-to-image.html . Transformer is decoder only network with 64 layers, each layer has 62 heads and model’s latent size is 64. Most of the ideas are borrowed from sparse attention paper which shows a way to reduce computation of a default self attention which is quadratic in time complexity (Link to the sparse transformer paper : https://arxiv.org/pdf/1904.10509.pdf ). There are three kind of attentions which are used in the transformer. Row attention, Column attention and causal convolution attention. From layer 1 to layer 63, we only have row or col...
Read more

NeurIPS 2021 — Curated papers — Part 1

Image
  UniDoc: Unified Pretraining Framework for Document Understanding Authors has proposed a self-supervised framework for document understanding from multi-modal point of view. Language Pre-training using transformers have become extremely popular. In this work, authors have showed how to do SSL using transformers by taking inputs from different modalities such as image and text. UniDoc has mainly 4 steps : Feature Extraction :  Given a document image I and location of document elements, using OCR sentences and it’s corresponding bounding boxes are extracted. Feature Embedding :  For bounding box, features are extracted through CNN backbone+RoIAlign and they are quantized using Gumble-softmax (similar to Wav2Vec2) and embedding for sentences are extracted from pre-trained hierarchical transformers. Gated cross attention :  It’s one of the main ingredient of the work , where cross-modal interaction takes places between text and visual embedding through typical cross-att...
Read more