【Bug已解决】Cant useload_lora_weights()onErnieImageModularPipeline解决方案一、现象长什么样ErnieImageModularPipeline是百度文心系列图像的模块化 pipeline。和 Ideogram4 类似它把组件放进components字典但即使接好了LoraLoaderMixin、桥接了self.transformer加载 LoRA 仍会失败from diffusers import ErnieImageModularPipeline pipe ErnieImageModularPipeline.from_pretrained(baidu/ernie-image) pipe.load_lora_weights(ernie-image-style-lora.safetensors)报错ValueError Could not find any LoRA target modules in transformer; no layers matching [to_q, to_k, to_v, to_out.0] were found.或者KeyError Cannot find corresponding diffusers key for lora_transformer_blocks_0_attn_qkv.lora_up.weight现象总结ErnieImage 的 transformer 模块命名和标准 UNet/DiT 不同——它用attn.qkv融合投影、cross_attn、以及文心特有的register_token_proj等而 diffusers 默认 LoRA 目标模块发现逻辑只认to_q/to_k/to_v/to_out于是找不到任何可注入层或 key 映射因命名差异 KeyError。二、背景标准 diffusers 模型SD/FLUX的注意力模块命名高度统一to_q/to_k/to_v/to_out。LoraLoaderMixin在注入时靠get_lora_pairs遍历 transformer 里名字含这些子串的nn.Linear把它们登记为 LoRA 目标。但 ErnieImage 基于文心自研结构自注意力用融合qkv投影attn.qkv一个nn.Linear输出[3*d, d]没有单独的to_q/k/v交叉注意力叫cross_attn有的实现把文本条件喂进register_token_proj而非标准 cross_attn文心还引入「register tokens」类似 FLUX 的 register由专门的register_token_proj投影。于是get_lora_pairs在 ErnieImage transformer 上扫不到to_q返回空列表 →ValueError: no LoRA target modules。即使强制注入了LoRA key 里的attn_qkv也无法翻译成 diffusers 期望的to_q路径 → KeyError。三、根因根因两点默认 LoRA 目标发现逻辑不认 ErnieImage 的命名get_lora_pairs只匹配to_q/k/v对attn.qkv/cross_attn/register_token_proj视而不见导致目标层为空。key 翻译未覆盖文心特有模块LoRA key 含attn_qkv/register_token_proj转换器没有对应 diffusers 路径映射失败。本质ErnieImage 的 transformer 命名偏离 diffusers 约定而 LoRA 的目标发现 key 翻译两层都写死了标准命名导致「找不到目标」或「key 翻译失败」。四、最小可运行复现用标准库复现「目标发现逻辑只认 to_q/k/v对 qkv 融合返回空」import torch.nn as nn class ErnieAttn(nn.Module): def __init__(self, d): super().__init__() self.qkv nn.Linear(d, 3 * d, biasFalse) # 融合没有 to_q/k/v self.proj nn.Linear(d, d, biasFalse) def get_lora_pairs_default(module): # 默认逻辑只找 to_q/to_k/to_v found [] for name, child in module.named_modules(): if any(seg in name for seg in (to_q, to_k, to_v)): found.append(name) return found m ErnieAttn(8) pairs get_lora_pairs_default(m) print(pairs , pairs) # [] 空因为根本没有 to_q/k/v assert pairs [], ErnieImage 融合 qkv 下默认发现逻辑返回空复现「key 翻译失败」LoRA keylora_transformer_blocks_0_attn_qkv.lora_up.weight在转换器里按to_q模式匹配不到qkvKeyError。五、解决方案第一层最小直接修复最小修复给 ErnieImage 提供专属的目标模块发现 key 翻译把qkv融合当成可注入目标并映射到 diffusers 的attn.to_qkv风格路径import re # ErnieImage 专属目标模块名 ERNIE_TARGET_HINTS (qkv, cross_attn, register_token_proj, proj) def get_lora_pairs_ernie(module): found [] for name, child in module.named_modules(): if any(seg in name for seg in ERNIE_TARGET_HINTS): found.append(name) return found def ernie_lora_key(kohya_key: str): base kohya_key.replace(lora_transformer_, ).replace(lora_up.weight, up) base base.replace(lora_down.weight, down).replace(.alpha, ) # qkv 融合 - attn.to_qkvdiffusers 约定 if attn_qkv in base: return base.replace(attn_qkv, attn.to_qkv) if register_token_proj in base: return base.replace(register_token_proj, register_token_proj) return base这样get_lora_pairs_ernie能找到attn.qkv/cross_attn/register_token_projkey 也能正确翻译LoRA 注入成功。六、解决方案第二层结构性改进把「ErnieImage transformer 的命名 → diffusers 目标模块」收敛成一个 dataclass 单一真源from dataclasses import dataclass, field from typing import Dict, List, Tuple dataclass(frozenTrue) class ErnieImageModularLoraPolicy: ErnieImageModularPipeline LoRA 接入的单一真源。 # 模块名片段 - 是否为 LoRA 目标 target_hints: Tuple[str, ...] (qkv, cross_attn, register_token_proj, proj) # kohya/源 key 片段 - diffusers 路径片段 key_map: Dict[str, str] field(default_factorylambda: { attn_qkv: attn.to_qkv, cross_attn: cross_attn, register_token_proj: register_token_proj, attn_proj: attn.proj, }) # 融合 qkv 的权重形状4D? 否但需按 [3d,d] 处理 fused_qkv: bool True # 是否支持 register token 的 LoRA register_lora: bool True def discover_targets(self, module) - List[str]: return [n for n, _ in module.named_modules() if any(h in n for h in self.target_hints)] def translate_key(self, src_key: str) - str: base src_key.replace(lora_transformer_, transformer.) for src, dst in self.key_map.items(): if src in base: base base.replace(src, dst) break if src_key.endswith(.lora_down.weight): base base.replace(up, lora.down.weight) elif src_key.endswith(.alpha): base base.replace(up, alpha) else: base base.replace(up, lora.up.weight) return base def validate_targets(self, module) - List[str]: found self.discover_targets(module) if not found: return [transformer 中未发现任何 ErnieImage 目标模块] return []加载主流程用policy.discover_targets找目标、policy.translate_key翻译 keyvalidate_targets在注入前校验非空。七、解决方案第三层断言 / CI 守护用 pytest 把「目标发现不空 融合 qkv 可翻译 register 支持 注入生效」固化成回归import torch import pytest from diffusers import ErnieImageModularPipeline from mylib.ernie_lora import ErnieImageModularLoraPolicy POLICY ErnieImageModularLoraPolicy() def test_targets_not_empty(): pipe ErnieImageModularPipeline.from_pretrained(baidu/ernie-image) targets POLICY.discover_targets(pipe.transformer) assert targets ! [], ErnieImage transformer 应至少发现 qkv/cross_attn 等目标 def test_qkv_key_translates(): out POLICY.translate_key(lora_transformer_blocks_0_attn_qkv.lora_up.weight) assert attn.to_qkv in out and out.endswith(lora.up.weight) def test_register_key_translates(): out POLICY.translate_key(lora_transformer_register_token_proj.lora_down.weight) assert register_token_proj in out def test_validate_targets_passes(): pipe ErnieImageModularPipeline.from_pretrained(baidu/ernie-image) assert POLICY.validate_targets(pipe.transformer) [] def test_lora_changes_output(): pipe ErnieImageModularPipeline.from_pretrained(baidu/ernie-image, torch_dtypebf16) base pipe(a cat).images[0] pipe.load_lora_weights(ernie-image-style-lora.safetensors) styled pipe(a cat).images[0] assert not _image_equal(base, styled)CI 把test_targets_not_empty与test_qkv_key_translates作为 ErnieImage LoRA 支持的必过项要求「任何改动 transformer 命名后必须重跑目标发现校验」。八、排查清单ErnieImageModularPipeline 加载 LoRA 失败按顺序查报错no LoRA target modules说明默认发现逻辑没找到to_q/k/v因为 ErnieImage 用融合qkv需换发现逻辑。transformer 是否用attn.qkv融合投影是就用target_hints包含qkv而非to_q。是否有cross_attn/register_token_proj这些是文心特有模块默认逻辑不认需加入目标列表。LoRA key 是否含attn_qkv/register_token_proj转换器需有对应key_map否则 KeyError。加载后图是否变化没变说明目标发现仍空或 key 没翻译对权重没注入。dtype 是否一致LoRA 与 transformer dtype 不一致会注入失败。九、小结「Cant use load_lora_weights() on ErnieImageModularPipeline」本质是ErnieImage 的 transformer 命名融合qkv、文心特有cross_attn/register_token_proj偏离 diffusers 约定而 LoRA 的目标发现与 key 翻译两层都写死了标准命名导致找不到目标或 key 翻译失败。第一层提供 ErnieImage 专属的目标发现 qkv→to_qkv的 key 翻译第二层把命名映射收敛到ErnieImageModularLoraPolicy单一真源用validate_targets保证注入前目标非空第三层用 pytest 守住「目标非空、融合 qkv 可翻译、register 支持、加载生效」。通用教训**每接入一个命名非标准的模型必须为它的 LoRA 目标发现与 key 翻译各写一份「命名→约定」映射否则复用机制LoRA会因默认逻辑不认而全部失效。