https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_24.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_26.png这个生成器会输出一张假的斑马图像。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_26.png循环一致性要求这张重新生成的假斑马图像应该与最初输入的真实斑马图像在内容上保持一致。因此我们计算两者之间的像素级差异如 L1 损失。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_28.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_28.png用公式表示从斑马Z到马H再回到斑马的循环一致性损失为L_cyc_Z || G_H2Z(G_Z2H(zebra)) - zebra ||_1https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_30.png同理反向过程从马到斑马再回到马也会计算一个对称的循环一致性损失。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_30.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_32.pngL_cyc_H || G_Z2H(G_H2Z(horse)) - horse ||_1https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_32.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_34.png身份损失可选为了进一步增强颜色和纹理的稳定性CycleGAN 可以引入可选的身份损失。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_36.png其思想是如果一个斑马图像已经属于目标域斑马域那么通过从马到斑马的生成器G_H2Z后它应该基本保持不变。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_36.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_38.png因此我们将真实的斑马图像也输入G_H2Z得到输出并计算输出与输入之间的像素差异作为身份损失。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_38.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_40.png用公式表示斑马的身份损失为L_id_Z || G_H2Z(zebra) - zebra ||_1同样对马图像和生成器G_Z2H也进行相同的操作。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_42.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_40.pngL_id_H || G_Z2H(horse) - horse ||_1https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_44.png损失函数总结https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_46.png现在让我们将所有损失项汇总起来。CycleGAN 的巧妙之处在于通过简单叠加这些直观的损失项就能实现强大的无监督图像转换。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_44.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_48.png以下是生成器的总损失构成共有六项https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_46.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_50.png对抗损失两项每个生成器对应一项用于欺骗其对应的判别器。G_Z2H的对抗损失L_adv_Z2HG_H2Z的对抗损失L_adv_H2Z循环一致性损失两项确保转换的可逆性。前向循环损失L_cyc_Z后向循环损失L_cyc_Hhttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_52.png身份损失两项可选稳定生成器的颜色和纹理。G_H2Z的身份损失L_id_ZG_Z2H的身份损失L_id_H生成器的总损失是这些项的加权和L_G_total L_adv_Z2H L_adv_H2Z λ_cyc * (L_cyc_Z L_cyc_H) λ_id * (L_id_Z L_id_H)其中λ_cyc和λ_id是超参数用于平衡各项损失的重要性。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_54.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_50.png这个总损失函数同时用于优化两个生成器。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_56.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_52.png对于判别器其任务则相对简单每个判别器只负责一项最小二乘对抗损失马的判别器D_H的损失L_D_H斑马的判别器D_Z的损失L_D_Zhttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_58.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_54.png总结https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_60.png本节课中我们一起学习了 CycleGAN 的整体架构和损失函数。总结来说CycleGAN 由两个GAN组成一个循环它们相互依赖通过计算多种损失项来学习无配对的域间映射。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_58.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_62.png生成器的损失包含六项核心的最小二乘对抗损失、保证内容不变的循环一致性损失以及可选的身份损失。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_64.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_60.png判别器的目标则相对单一仅使用最小二乘对抗损失来提升自身的判别能力。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_66.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/04f1c5f0d146fcb22fbce9aea7af5827_64.png理解这些损失项如何共同作用是掌握 CycleGAN 工作原理的关键。83CycleGAN应用及变种 ➡️https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_0.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_2.png在本节课中我们将学习CycleGAN的实际应用场景以及其重要的变体模型。我们将了解CycleGAN如何被用于艺术创作、医疗数据增强等领域并探索UNIT和MUNIT这两种基于共享潜在空间假设的先进无监督图像翻译模型。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_4.png应用领域上一节我们介绍了CycleGAN的基本原理本节中我们来看看它在现实世界中的具体应用。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_6.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_8.pngCycleGAN可以用于许多不同的事情。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_10.png以下是其主要应用方向社交媒体滤镜例如Snapchat上的各种滤镜使人变老或改变性别感知。物体转换例如将马变成斑马并且能可靠地保持转换后物体如斑马条纹的特征。场景风格迁移改变场景的季节或将照片转换为特定的绘画风格如莫奈风格反之亦然。数据增强在医疗等领域CycleGAN是一种常用的数据增强技术。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_12.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_14.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_16.png医疗数据增强示例https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_18.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_20.png现在让我们深入了解CycleGAN在医疗数据增强中的具体作用。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_22.png找到同时包含肿瘤和未包含肿瘤的配对医学图像非常困难。这种预知能力目前可能无法实现因此CycleGAN提供了一种数据增强的方法。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_24.png以下是CycleGAN在医疗图像处理中的两种应用创造肿瘤可以在健康图像上生成肿瘤以模拟患病情况。消除肿瘤可以从患病图像中移除肿瘤以模拟健康情况。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_26.png这些生成的配对示例可用于下游任务例如肿瘤分类、分割以及监测肿瘤生长这在医疗应用中非常重要。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_28.png例如使用CycleGAN可以移除CT扫描图像中的肿瘤大块或者向图像中添加逼真的肿瘤。在数据增强中像SegGAN这样的模型能够生成极其逼真的分割示例其效果是标准数据增强方法无法达到的。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_30.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_32.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_34.png无监督图像翻译变体UNIThttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_36.pngCycleGAN并不是唯一能够进行无配对图像翻译的模型。接下来我们介绍一种重要的变体——UNIT。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_38.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_40.png有一种变体称为UNIT它代表无监督图像到图像转换。无配对图像转换本身就是无监督的因为你没有成对的图像标签。例如实现白天与夜间驾驶场景的转换。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_42.png这个模型的关键洞察被称为共享潜在空间。其核心思想是两个不同域的图像可以映射到同一个潜在空间中的向量。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_44.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_46.png我们可以用以下公式化描述给定潜在空间中的一个噪声向量z生成器G1可以在域X1中生成图像X1 G1(z)同时存在一个编码器E1可以将X1域的图像映射回潜在向量z E1(X1)同样的潜在向量z也可以通过另一个生成器G2在域X2中生成图像X2 G2(z)并且域X2的图像也能通过编码器E2映射回潜在空间。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_48.png你可以想象这两个域X1是白天的驾驶场景X2是夜间的驾驶场景。X1和X2共享相同的内容即潜空间向量z但表现出不同的风格白天或夜间。模型可以学习在这两个风格之间来回映射。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_50.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_52.png多模态生成MUNIThttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_54.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_56.pngUNIT模型的能力不仅限于单一风格的转换。它的一个扩展版本称为多模态UNITMUNIT能够实现更丰富的生成。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_58.png多模态的意思是从一个域的输入如草图可以生成第二个域的多种不同风格输出。MUNIT不仅能找到一个映射还能发现并生成目标域的多种潜在模式。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_60.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_62.png这非常酷因为你从未明确告诉模型关于所有不同的风格。例如给定一个鞋子的草图它能够自动找出所有不同的鞋类风格如靴子、运动鞋、高跟鞋并生成对应的多种图像。这一切都是在无监督的情况下完成的模型自行发现了鞋子之间的风格差异。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_64.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_66.png技术背景与总结https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_68.pnghttps://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_70.png最后我们来总结一下本节课的核心内容。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_72.png需要快速说明的是UNIT和MUNIT这两种技术都依赖于将图像编码到潜在空间从变分自编码器VAE中获得灵感同时使用GAN的组件来确保生成图像的真实性。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_74.png总结来说CycleGAN的扩展模型包括UNIT和MUNIT它们使用了共享潜在空间的假设。特别是MUNIT实现了多模态生成这非常强大。GAN尤其是其变体在艺术、医疗、视频和游戏等领域有大量应用。https://github.com/OpenDocCN/dsai-notes-pt1-zh/raw/master/docs/dlai-dl-25/img/10bc1ab3f7fe1740078b6fcd98a231cd_76.png本节课中我们一起学习了CycleGAN的广泛应用、其在医疗数据增强中的价值以及UNIT和MUNIT这两种基于共享潜在空间的无监督图像翻译模型。它们为图像生成和转换任务提供了更强大、更灵活的解决方案。