简单记录一下BLIP2部署的流程
主要遇到的问题还是有墙导致模型权重无法下载
环境安装
本文采用Lavis进行BLIP2的部署
1.pip 安装lavis，这里记得换一下清华源，下载会快一点
pip install salesforce-lavis
通过下方代码判断lavis库是否安装成功
from lavis.models import model_zoo
print(model_zoo)
2.替换安装的库中权重文件路径
以Image Captioning任务为例
-
修改
lavis/configs/models/blip_caption_base_coco.yaml
该文件在python lavis库中，如果是通过git安装的，则直接更改对应git 项目文件model:arch: blip_captionload_finetuned: Truepretrained: "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_capfilt_large.pth"finetuned: "https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP/blip_coco_caption_base.pth"# vit encodervit_type: "base"vit_grad_ckpt: Falsevit_ckpt_layer: 0image_size: 384将
pretrained和finetuned中的权重文件下载到本地，然后将其替换成权重的绝对路径 -
运行报错
OSError: Can't load tokenizer for 'bert-base-uncased'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'bert-base-uncased' is the correct path to a dir直接命令行通过huggingface镜像运行脚本，从而下载对应bert模型权重
HF_ENDPOINT=https://hf-mirror.com python 测试脚本.py总体思想就是缺少哪个模型权重文件就下载哪个，然后替换成对应绝对路径
另附Image Captioning任务 测试脚本
import torch from lavis.models import load_model_and_preprocess from PIL import Imagedevice = torch.device("cuda" if torch.cuda.is_available() else "cpu") # loads BLIP caption base model, with finetuned checkpoints on MSCOCO captioning dataset. # this also loads the associated image processors model, vis_processors, _ = load_model_and_preprocess(name="blip_caption", model_type="base_coco", is_eval=True, device=device) # preprocess the image # vis_processors stores image transforms for "train" and "eval" (validation / testing / inference) raw_image = Image.open("./merlion.png").convert("RGB")image = vis_processors["eval"](raw_image).unsqueeze(0).to(device) # generate caption res=model.generate({"image": image}) print(res) # ['a large fountain spewing water into the air']
