README.md
2.9 KB · 76 lines · markdown Raw
1 ---
2 language: en
3 license: mit
4 tags:
5 - vision
6 - video-classification
7 model-index:
8 - name: nielsr/xclip-base-patch16-zero-shot
9 results:
10 - task:
11 type: video-classification
12 dataset:
13 name: HMDB-51
14 type: hmdb-51
15 metrics:
16 - type: top-1 accuracy
17 value: 44.6
18 - task:
19 type: video-classification
20 dataset:
21 name: UCF101
22 type: ucf101
23 metrics:
24 - type: top-1 accuracy
25 value: 72.0
26 - task:
27 type: video-classification
28 dataset:
29 name: Kinetics-600
30 type: kinetics600
31 metrics:
32 - type: top-1 accuracy
33 value: 65.2
34 ---
35
36 # X-CLIP (base-sized model)
37
38 X-CLIP model (base-sized, patch resolution of 16) trained on [Kinetics-400](https://www.deepmind.com/open-source/kinetics). It was introduced in the paper [Expanding Language-Image Pretrained Models for General Video Recognition](https://arxiv.org/abs/2208.02816) by Ni et al. and first released in [this repository](https://github.com/microsoft/VideoX/tree/master/X-CLIP).
39
40 This model was trained using 32 frames per video, at a resolution of 224x224.
41
42 Disclaimer: The team releasing X-CLIP did not write a model card for this model so this model card has been written by the Hugging Face team.
43
44 ## Model description
45
46 X-CLIP is a minimal extension of [CLIP](https://huggingface.co/docs/transformers/model_doc/clip) for general video-language understanding. The model is trained in a contrastive way on (video, text) pairs.
47
48 ![X-CLIP architecture](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/xclip_architecture.png)
49
50 This allows the model to be used for tasks like zero-shot, few-shot or fully supervised video classification and video-text retrieval.
51
52 ## Intended uses & limitations
53
54 You can use the raw model for determining how well text goes with a given video. See the [model hub](https://huggingface.co/models?search=microsoft/xclip) to look for
55 fine-tuned versions on a task that interests you.
56
57 ### How to use
58
59 For code examples, we refer to the [documentation](https://huggingface.co/transformers/main/model_doc/xclip.html#).
60
61 ## Training data
62
63 This model was trained on [Kinetics 400](https://www.deepmind.com/open-source/kinetics).
64
65 ### Preprocessing
66
67 The exact details of preprocessing during training can be found [here](https://github.com/microsoft/VideoX/blob/40f6d177e0a057a50ac69ac1de6b5938fd268601/X-CLIP/datasets/build.py#L247).
68
69 The exact details of preprocessing during validation can be found [here](https://github.com/microsoft/VideoX/blob/40f6d177e0a057a50ac69ac1de6b5938fd268601/X-CLIP/datasets/build.py#L285).
70
71 During validation, one resizes the shorter edge of each frame, after which center cropping is performed to a fixed-size resolution (like 224x224). Next, frames are normalized across the RGB channels with the ImageNet mean and standard deviation.
72
73 ## Evaluation results
74
75 This model achieves a zero-shot top-1 accuracy of 44.6% on HMDB-51, 72.0% on UCF-101 and 65.2% on Kinetics-600.
76