PE-Core-S16-384在视频理解中的突破Kinetics-400数据集76.9%准确率达成【免费下载链接】PE-Core-S16-384项目地址: https://ai.gitcode.com/hf_mirrors/facebook/PE-Core-S16-384Perception EncoderPE是Meta推出的最先进图像和视频理解编码器通过简单的视觉-语言学习训练而成。其核心模型PE-Core-S16-384在视频理解领域实现重大突破尤其在Kinetics-400数据集上达成76.9%的准确率展现出卓越的动作识别能力。什么是Perception EncoderPerception Encoder是一系列大规模视觉编码器模型在各种视觉任务中均表现出最先进的性能。通过强大的对比预训练方案和合成对齐视频微调PE不仅在分类和检索任务上超越现有所有模型还能内部生成强大的通用特征为下游任务提供支持。核心技术优势PE模型的关键创新在于对比预训练采用稳健的对比学习方法进行预训练合成视频微调在合成视频数据引擎生成的数据上进行精细调优特征提取能力能够生成可扩展的通用特征用于下游任务PE-Core-S16-384的性能表现PE-Core系列模型在多个视觉任务中表现出色特别是在视频理解领域。在Kinetics-400数据集上PE-Core-S16-384实现了76.9%的准确率这一成绩在零样本视频分类任务中处于领先地位。多任务性能概览ModelIN-1kIN-v2IN-AObjectNetCOCO-T2IKinetics-400VTT-T2IB/16224px78.471.762.471.950.965.647.6L/14336px83.577.989.084.757.173.450.3G/14448px85.480.292.688.258.176.951.2从表格数据可以看出PE-Core模型在困难基准测试如ObjectNet和ImageNet-A上表现尤为出色证明了其强大的泛化能力和鲁棒性。如何开始使用PE-Core-S16-384环境配置步骤要使用PE-Core-S16-384模型首先需要配置开发环境git clone https://gitcode.com/hf_mirrors/facebook/PE-Core-S16-384 cd perception_models conda create --name perception_models python3.12 conda activate perception_models # 安装PyTorch pip install torch2.5.1 torchvision0.20.1 torchaudio2.5.1 xformers --index-url https://download.pytorch.org/whl/cu124 # 安装视频解码依赖 conda install ffmpeg -c conda-forge pip install torchcodec0.1 --index-urlhttps://download.pytorch.org/whl/cu124 pip install -e .特征提取示例以下是使用训练好的模型提取图像和文本特征的简单示例import torch from PIL import Image import core.vision_encoder.pe as pe import core.vision_encoder.transforms as transforms # 查看可用配置 print(CLIP configs:, pe.CLIP.available_configs()) # 输出: [PE-Core-G14-448, PE-Core-L14-336, PE-Core-B16-224] # 加载模型 model pe.CLIP.from_config(PE-Core-L14-336, pretrainedTrue) # 从HF下载 model model.cuda() # 预处理和分词器 preprocess transforms.get_image_transform(model.image_size) tokenizer transforms.get_text_tokenizer(model.context_length) # 准备输入 image preprocess(Image.open(docs/assets/cat.png)).unsqueeze(0).cuda() text tokenizer([a diagram, a dog, a cat]).cuda() # 特征提取 with torch.no_grad(), torch.autocast(cuda): image_features, text_features, logit_scale model(image, text) text_probs (logit_scale * image_features text_features.T).softmax(dim-1) print(Label probs:, text_probs) # 输出: [[0.0, 0.0, 1.0]]模型配置详情PE-Core目前提供三种尺寸PE-Core G是主要检查点L和B模型是从G蒸馏而来ScaleTowerParamsWidthDepthMLPHeadsCLIP DimResolution / Context LenB/16Vision0.09B768123072121024224pxText0.31B102424409616102432 tokensL/14Vision0.32B1024244096161024336pxText0.31B102424409616102432 tokensG/14Vision1.88B1536508960161280448pxText0.47B128024512020128072 tokens所有PE-Core模型在视觉塔顶部使用带有8个头部的注意力池化块。L和B模型还额外有一个用于全局聚合的类令牌。引用与致谢如果您发现此代码对您的研究有用请考虑引用以下论文article{bolya2025PerceptionEncoder, title{Perception Encoder: The best visual embeddings are not at the output of the network}, author{Daniel Bolya and Po-Yao Huang and Peize Sun and Jang Hyun Cho and Andrea Madotto and Chen Wei and Tengyu Ma and Jiale Zhi and Jathushan Rajasegaran and Hanoona Rasheed and Junke Wang and Marco Monteiro and Hu Xu and Shiyu Dong and Nikhila Ravi and Daniel Li and Piotr Dollár and Christoph Feichtenhofer}, journal{arXiv}, year{2025} } article{cho2025PerceptionLM, title{PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding}, author{Jang Hyun Cho and Andrea Madotto and Effrosyni Mavroudi and Triantafyllos Afouras and Tushar Nagarajan and Muhammad Maaz and Yale Song and Tengyu Ma and Shuming Hu and Hanoona Rasheed and Peize Sun and Po-Yao Huang and Daniel Bolya and Suyog Jain and Miguel Martin and Huiyu Wang and Nikhila Ravi and Shashank Jain and Temmy Stark and Shane Moon and Babak Damavandi and Vivian Lee and Andrew Westbury and Salman Khan and Philipp Krähenbühl and Piotr Dollár and Lorenzo Torresani and Kristen Grauman and Christoph Feichtenhofer}, journal{arXiv}, year{2025} }PE-Core-S16-384模型的推出为视频理解领域带来了新的可能性76.9%的Kinetics-400准确率不仅是一个数字里程碑更是计算机视觉技术在动作识别和视频分析方向上的重要进展。无论是学术研究还是工业应用都能从这一强大的视觉编码器中受益。【免费下载链接】PE-Core-S16-384项目地址: https://ai.gitcode.com/hf_mirrors/facebook/PE-Core-S16-384创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考