{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Vision Transformer图像分类\n", "\n", "[](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.8/tutorials/application/zh_cn/cv/mindspore_vit.ipynb) [](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.8/tutorials/application/zh_cn/cv/mindspore_vit.py) [](https://gitee.com/mindspore/docs/blob/r1.8/tutorials/application/source_zh_cn/cv/vit.ipynb)\n", "\n", "作者:[崔馨予](https://gitee.com/cui-xinyuwhc)。\n", "\n", "本案例支持在GPU环境下运行。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Vision Transformer(ViT)简介\n", "\n", "近些年,随着基于自注意(Self-Attention)结构的模型的发展,特别是Transformer模型的提出,极大地促进了自然语言处理模型的发展。由于Transformers的计算效率和可扩展性,它已经能够训练具有超过100B参数的空前规模的模型。\n", "\n", "ViT则是自然语言处理和计算机视觉两个领域的融合结晶。在不依赖卷积操作的情况下,依然可以在图像分类任务上达到很好的效果。\n", "\n", "### 模型结构\n", "\n", "ViT模型的主体结构是基于Transformer模型的Encoder部分(部分结构顺序有调整,如:Normalization的位置与标准Transformer不同),其结构图[1]如下:\n", "\n", "\n", "\n", "### 模型特点\n", "\n", "ViT模型主要应用于图像分类领域。因此,其模型结构相较于传统的Transformer有以下几个特点:\n", "\n", "1. 数据集的原图像被划分为多个patch后,将二维patch(不考虑channel)转换为一维向量,再加上类别向量与位置向量作为模型输入。\n", "2. 模型主体的Block结构是基于Transformer的Encoder结构,但是调整了Normalization的位置,其中,最主要的结构依然是Multi-head Attention结构。\n", "3. 模型在Blocks堆叠后接全连接层,接受类别向量的输出作为输入并用于分类。通常情况下,我们将最后的全连接层称为Head,Transformer Encoder部分为backbone。\n", "\n", "下面将通过代码实例来详细解释基于ViT实现ImageNet分类任务。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 环境准备与数据读取\n", "\n", "开始实验之前,请确保本地已经安装了Python环境并安装了[MindSpore Vision套件](https://mindspore.cn/vision/docs/zh-CN/r0.1/index.html)。\n", "\n", "首先导入相关模块,配置相关超参数并读取数据集,该部分代码在MindSpore Vision套件中都有API可直接调用,详情可以参考以下链接:https://www.mindspore.cn/vision/docs/zh-CN/r0.1/index.html 。\n", "\n", "可通过<http://image-net.org>下载完整的ImageNet数据集。\n", "\n", "本案例应用的数据集是从ImageNet中筛选出来的子集,运行第一段代码时会自动下载并解压。\n", "\n", "请确保你的数据集路径如以下结构。\n", "\n", "```text\n", ".dataset/\n", " ├── ILSVRC2012_devkit_t12.tar.gz\n", " ├── train/\n", " ├── infer/\n", " └── val/\n", "```" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "510368768B [03:36, 2353191.94B/s] \n" ] } ], "source": [ "from mindvision.dataset import DownLoad\n", "\n", "dataset_url = \"https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/vit_dataset.zip\"\n", "\n", "dl = DownLoad()\n", "dl.download_and_extract_archive(dataset_url, \"./\")" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import mindspore as ms\n", "from mindvision.classification.dataset import ImageNet\n", "\n", "ms.set_context(mode=ms.GRAPH_MODE, device_target='GPU')\n", "\n", "data_url = './dataset/'\n", "resize = 224\n", "batch_size = 16\n", "\n", "dataset_train = ImageNet(data_url,\n", " split=\"train\",\n", " shuffle=True,\n", " resize=resize,\n", " batch_size=batch_size,\n", " repeat_num=1,\n", " num_parallel_workers=1).run()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 模型解析\n", "\n", "下面将通过代码来细致剖析ViT模型的内部结构。\n", "\n", "### Transformer基本原理\n", "\n", "Transformer模型源于2017年的一篇文章[2]。在这篇文章中提出的基于Attention机制的编码器-解码器型结构在自然语言处理领域获得了巨大的成功。模型结构如下图所示:\n", "\n", "\n", "\n", "其主要结构为多个Encoder和Decoder模块所组成,其中Encoder和Decoder的详细结构如下图[2]所示:\n", "\n", "\n", "\n", "Encoder与Decoder由许多结构组成,如:多头注意力(Multi-Head Attention)层,Feed Forward层,Normaliztion层,甚至残差连接(Residual Connection,图中的“Add”)。不过,其中最重要的结构是多头注意力(Multi-Head Attention)结构,该结构基于自注意力(Self-Attention)机制,是多个Self-Attention的并行组成。\n", "\n", "所以,理解了Self-Attention就抓住了Transformer的核心。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Attention模块\n", "\n", "以下是Self-Attention的解释,其核心内容是为输入向量的每个单词学习一个权重。通过给定一个任务相关的查询向量Query向量,计算Query和各个Key的相似性或者相关性得到注意力分布,即得到每个Key对应Value的权重系数,然后对Value进行加权求和得到最终的Attention数值。\n", "\n", "在Self-Attention中:\n", "\n", "1. 最初的输入向量首先会经过Embedding层映射成Q(Query),K(Key),V(Value)三个向量,由于是并行操作,所以代码中是映射成为dim x 3的向量然后进行分割,换言之,如果你的输入向量为一个向量序列($x_1$,$x_2$,$x_3$),其中的$x_1$,$x_2$,$x_3$都是一维向量,那么每一个一维向量都会经过Embedding层映射出Q,K,V三个向量,只是Embedding矩阵不同,矩阵参数也是通过学习得到的。**这里大家可以认为,Q,K,V三个矩阵是发现向量之间关联信息的一种手段,需要经过学习得到,至于为什么是Q,K,V三个,主要是因为需要两个向量点乘以获得权重,又需要另一个向量来承载权重向加的结果,所以,最少需要3个矩阵。**\n", "\n", "$$\n", "\\begin{cases}\n", "q_i = W_q \\cdot x_i & \\\\\n", "k_i = W_k \\cdot x_i,\\hspace{1em} &i = 1,2,3 \\ldots \\\\\n", "v_i = W_v \\cdot x_i &\n", "\\end{cases}\n", "\\tag{1}\n", "$$\n", "\n", "\n", "\n", "2. 自注意力机制的自注意主要体现在它的Q,K,V都来源于其自身,也就是该过程是在提取输入的不同顺序的向量的联系与特征,最终通过不同顺序向量之间的联系紧密性(Q与K乘积经过Softmax的结果)来表现出来。**Q,K,V得到后就需要获取向量间权重,需要对Q和K进行点乘并除以维度的平方根,对所有向量的结果进行Softmax处理,通过公式(2)的操作,我们获得了向量之间的关系权重。**\n", "\n", "$$\n", "\\begin{cases}\n", "a_{1,1} = q_1 \\cdot k_1 / \\sqrt d \\\\\n", "a_{1,2} = q_1 \\cdot k_2 / \\sqrt d \\\\\n", "a_{1,3} = q_1 \\cdot k_3 / \\sqrt d\n", "\\end{cases}\n", "\\tag{2}\n", "$$\n", "\n", "\n", "\n", "$$ Softmax: \\hat a_{1,i} = exp(a_{1,i}) / \\sum_j exp(a_{1,j}),\\hspace{1em} j = 1,2,3 \\ldots \\tag{3}$$\n", "\n", "\n", "\n", "3. 其最终输出则是通过V这个映射后的向量与Q,K经过Softmax结果进行weight sum获得,这个过程可以理解为在全局上进行自注意表示。**每一组Q,K,V最后都有一个V输出,这是Self-Attention得到的最终结果,是当前向量在结合了它与其他向量关联权重后得到的结果。**\n", "\n", "$$\n", "b_1 = \\sum_i \\hat a_{1,i}v_i,\\hspace{1em} i = 1,2,3...\n", "\\tag{4}\n", "$$\n", "\n", "通过下图可以整体把握Self-Attention的全部过程。\n", "\n", "\n", "\n", "多头注意力机制就是将原本self-Attention处理的向量分割为多个Head进行处理,这一点也可以从代码中体现,这也是attention结构可以进行并行加速的一个方面。\n", "\n", "总结来说,多头注意力机制在保持参数总量不变的情况下,将同样的query, key和value映射到原来的高维空间(Q,K,V)的不同子空间(Q_0,K_0,V_0)中进行自注意力的计算,最后再合并不同子空间中的注意力信息。\n", "\n", "所以,对于同一个输入向量,多个注意力机制可以同时对其进行处理,即利用并行计算加速处理过程,又在处理的时候更充分的分析和利用了向量特征。下图展示了多头注意力机制,其并行能力的主要体现在下图中的$a_1$和$a_2$是同一个向量进行分割获得的。\n", "\n", "\n", "\n", "以下是vision套件中的Multi-Head Attention代码,结合上文的解释,代码清晰的展现了这一过程。" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from mindspore import nn, ops\n", "\n", "class Attention(nn.Cell):\n", " def __init__(self,\n", " dim: int,\n", " num_heads: int = 8,\n", " keep_prob: float = 1.0,\n", " attention_keep_prob: float = 1.0):\n", " super(Attention, self).__init__()\n", "\n", " self.num_heads = num_heads\n", " head_dim = dim // num_heads\n", " self.scale = ms.Tensor(head_dim ** -0.5)\n", "\n", " self.qkv = nn.Dense(dim, dim * 3)\n", " self.attn_drop = nn.Dropout(attention_keep_prob)\n", " self.out = nn.Dense(dim, dim)\n", " self.out_drop = nn.Dropout(keep_prob)\n", "\n", " self.mul = ops.Mul()\n", " self.reshape = ops.Reshape()\n", " self.transpose = ops.Transpose()\n", " self.unstack = ops.Unstack(axis=0)\n", " self.attn_matmul_v = ops.BatchMatMul()\n", " self.q_matmul_k = ops.BatchMatMul(transpose_b=True)\n", " self.softmax = nn.Softmax(axis=-1)\n", "\n", " def construct(self, x):\n", " \"\"\"Attention construct.\"\"\"\n", " b, n, c = x.shape\n", "\n", " # 最初的输入向量首先会经过Embedding层映射成Q(Query),K(Key),V(Value)三个向量\n", " # 由于是并行操作,所以代码中是映射成为dim*3的向量然后进行分割\n", " qkv = self.qkv(x)\n", "\n", " #多头注意力机制就是将原本self-Attention处理的向量分割为多个Head进行处理\n", " qkv = self.reshape(qkv, (b, n, 3, self.num_heads, c // self.num_heads))\n", " qkv = self.transpose(qkv, (2, 0, 3, 1, 4))\n", " q, k, v = self.unstack(qkv)\n", "\n", " # 自注意力机制的自注意主要体现在它的Q,K,V都来源于其自身\n", " # 也就是该过程是在提取输入的不同顺序的向量的联系与特征\n", " # 最终通过不同顺序向量之间的联系紧密性(Q与K乘积经过Softmax的结果)来表现出来\n", " attn = self.q_matmul_k(q, k)\n", " attn = self.mul(attn, self.scale)\n", " attn = self.softmax(attn)\n", " attn = self.attn_drop(attn)\n", "\n", " # 其最终输出则是通过V这个映射后的向量与QK经过Softmax结果进行weight sum获得\n", " # 这个过程可以理解为在全局上进行自注意表示\n", " out = self.attn_matmul_v(attn, v)\n", " out = self.transpose(out, (0, 2, 1, 3))\n", " out = self.reshape(out, (b, n, c))\n", " out = self.out(out)\n", " out = self.out_drop(out)\n", "\n", " return out" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Transformer Encoder\n", "\n", "在了解了Self-Attention结构之后,通过与Feed Forward,Residual Connection等结构的拼接就可以形成Transformer的基础结构,接下来就利用Self-Attention来构建ViT模型中的TransformerEncoder部分,类似于构建了一个Transformer的编码器部分,如下图[1]所示:\n", "\n", "\n", "\n", "1. ViT模型中的基础结构与标准Transformer有所不同,主要在于Normalization的位置是放在Self-Attention和Feed Forward之前,其他结构如Residual Connection,Feed Forward,Normalization都如Transformer中所设计。\n", "\n", "2. 从Transformer结构的图片可以发现,多个子encoder的堆叠就完成了模型编码器的构建,在ViT模型中,依然沿用这个思路,通过配置超参数num_layers,就可以确定堆叠层数。\n", "\n", "3. Residual Connection,Normalization的结构可以保证模型有很强的扩展性(保证信息经过深层处理不会出现退化的现象,这是Residual Connection的作用),Normalization和dropout的应用可以增强模型泛化能力。\n", "\n", "从以下源码中就可以清晰看到Transformer的结构。将TransformerEncoder结构和一个多层感知器(MLP)结合,就构成了ViT模型的backbone部分。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "class TransformerEncoder(nn.Cell):\n", " def __init__(self,\n", " dim: int,\n", " num_layers: int,\n", " num_heads: int,\n", " mlp_dim: int,\n", " keep_prob: float = 1.,\n", " attention_keep_prob: float = 1.0,\n", " drop_path_keep_prob: float = 1.0,\n", " activation: nn.Cell = nn.GELU,\n", " norm: nn.Cell = nn.LayerNorm):\n", " super(TransformerEncoder, self).__init__()\n", " layers = []\n", "\n", " # 从vit_architecture图可以发现,多个子encoder的堆叠就完成了模型编码器的构建\n", " # 在ViT模型中,依然沿用这个思路,通过配置超参数num_layers,就可以确定堆叠层数\n", " for _ in range(num_layers):\n", " normalization1 = norm((dim,))\n", " normalization2 = norm((dim,))\n", " attention = Attention(dim=dim,\n", " num_heads=num_heads,\n", " keep_prob=keep_prob,\n", " attention_keep_prob=attention_keep_prob)\n", "\n", " feedforward = FeedForward(in_features=dim,\n", " hidden_features=mlp_dim,\n", " activation=activation,\n", " keep_prob=keep_prob)\n", "\n", " # ViT模型中的基础结构与标准Transformer有所不同\n", " # 主要在于Normalization的位置是放在Self-Attention和Feed Forward之前\n", " # 其他结构如Residual Connection,Feed Forward,Normalization都如Transformer中所设计\n", " layers.append(\n", " nn.SequentialCell([\n", " # Residual Connection,Normalization的结构可以保证模型有很强的扩展性\n", " # 保证信息经过深层处理不会出现退化的现象,这是Residual Connection的作用\n", " # Normalization和dropout的应用可以增强模型泛化能力\n", " ResidualCell(nn.SequentialCell([normalization1,\n", " attention])),\n", "\n", " ResidualCell(nn.SequentialCell([normalization2,\n", " feedforward]))\n", " ])\n", " )\n", " self.layers = nn.SequentialCell(layers)\n", "\n", " def construct(self, x):\n", " \"\"\"Transformer construct.\"\"\"\n", " return self.layers(x)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ViT模型的输入\n", "\n", "传统的Transformer结构主要用于处理自然语言领域的词向量(Word Embedding or Word Vector),词向量与传统图像数据的主要区别在于,词向量通常是1维向量进行堆叠,而图片则是二维矩阵的堆叠,多头注意力机制在处理1维词向量的堆叠时会提取词向量之间的联系也就是上下文语义,这使得Transformer在自然语言处理领域非常好用,而2维图片矩阵如何与1维词向量进行转化就成为了Transformer进军图像处理领域的一个小门槛。\n", "\n", "在ViT模型中:\n", "\n", "1. 通过将输入图像在每个channel上划分为16*16个patch,这一步是通过卷积操作来完成的,当然也可以人工进行划分,但卷积操作也可以达到目的同时还可以进行一次而外的数据处理;**例如一幅输入224 x 224的图像,首先经过卷积处理得到16 x 16个patch,那么每一个patch的大小就是14 x 14。**\n", "\n", "2. 再将每一个patch的矩阵拉伸成为一个1维向量,从而获得了近似词向量堆叠的效果。**上一步得到的14 x 14的patch就转换为长度为196的向量。**\n", "\n", "这是图像输入网络经过的第一步处理。具体Patch Embedding的代码如下所示:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "class PatchEmbedding(nn.Cell):\n", " MIN_NUM_PATCHES = 4\n", " def __init__(self,\n", " image_size: int = 224,\n", " patch_size: int = 16,\n", " embed_dim: int = 768,\n", " input_channels: int = 3):\n", " super(PatchEmbedding, self).__init__()\n", "\n", " self.image_size = image_size\n", " self.patch_size = patch_size\n", " self.num_patches = (image_size // patch_size) ** 2\n", "\n", " # 通过将输入图像在每个channel上划分为16*16个patch\n", " self.conv = nn.Conv2d(input_channels, embed_dim, kernel_size=patch_size, stride=patch_size, has_bias=True)\n", " self.reshape = P.Reshape()\n", " self.transpose = P.Transpose()\n", "\n", " def construct(self, x):\n", " \"\"\"Path Embedding construct.\"\"\"\n", " x = self.conv(x)\n", " b, c, h, w = x.shape\n", "\n", " # 再将每一个patch的矩阵拉伸成为一个1维向量,从而获得了近似词向量堆叠的效果;\n", " x = self.reshape(x, (b, c, h * w))\n", " x = self.transpose(x, (0, 2, 1))\n", "\n", " return x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "输入图像在划分为patch之后,会经过pos_embedding 和 class_embedding两个过程。\n", "\n", "1. class_embedding主要借鉴了BERT模型的用于文本分类时的思想,在每一个word vector之前增加一个类别值,通常是加在向量的第一位,**上一步得到的196维的向量加上class_embedding后变为197维。**\n", "\n", "2. 增加的class_embedding是一个可以学习的参数,经过网络的不断训练,最终以输出向量的第一个维度的输出来决定最后的输出类别;**由于输入是16 x 16个patch,所以输出进行分类时是取 16 x 16个class_embedding进行分类。**\n", "\n", "3. pos_embedding也是一组可以学习的参数,会被加入到经过处理的patch矩阵中。\n", "\n", "4. 由于pos_embedding也是可以学习的参数,所以它的加入类似于全链接网络和卷积的bias。**这一步就是创造一个长度维197的可训练向量加入到经过class_embedding的向量中。**\n", "\n", "实际上,pos_embedding总共有4种方案。但是经过作者的论证,只有加上pos_embedding和不加pos_embedding有明显影响,至于pos_embedding是1维还是2维对分类结果影响不大,所以,在我们的代码中,也是采用了1维的pos_embedding,由于class_embedding是加在pos_embedding之前,所以pos_embedding的维度会比patch拉伸后的维度加1。\n", "\n", "总的而言,ViT模型还是利用了Transformer模型在处理上下文语义时的优势,将图像转换为一种“变种词向量”然后进行处理,而这样转换的意义在于,多个patch之间本身具有空间联系,这类似于一种“空间语义”,从而获得了比较好的处理效果。\n", "\n", "### 整体构建ViT\n", "\n", "以下代码构建了一个完整的ViT模型。" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from typing import Optional\n", "\n", "class ViT(nn.Cell):\n", " def __init__(self,\n", " image_size: int = 224,\n", " input_channels: int = 3,\n", " patch_size: int = 16,\n", " embed_dim: int = 768,\n", " num_layers: int = 12,\n", " num_heads: int = 12,\n", " mlp_dim: int = 3072,\n", " keep_prob: float = 1.0,\n", " attention_keep_prob: float = 1.0,\n", " drop_path_keep_prob: float = 1.0,\n", " activation: nn.Cell = nn.GELU,\n", " norm: Optional[nn.Cell] = nn.LayerNorm,\n", " pool: str = 'cls') -> None:\n", " super(ViT, self).__init__()\n", "\n", " self.patch_embedding = PatchEmbedding(image_size=image_size,\n", " patch_size=patch_size,\n", " embed_dim=embed_dim,\n", " input_channels=input_channels)\n", " num_patches = self.patch_embedding.num_patches\n", "\n", " # 此处增加class_embedding和pos_embedding,如果不是进行分类任务\n", " # 可以只增加pos_embedding,通过pool参数进行控制\n", " self.cls_token = init(init_type=Normal(sigma=1.0),\n", " shape=(1, 1, embed_dim),\n", " dtype=ms.float32,\n", " name='cls',\n", " requires_grad=True)\n", "\n", " # pos_embedding也是一组可以学习的参数,会被加入到经过处理的patch矩阵中\n", " self.pos_embedding = init(init_type=Normal(sigma=1.0),\n", " shape=(1, num_patches + 1, embed_dim),\n", " dtype=ms.float32,\n", " name='pos_embedding',\n", " requires_grad=True)\n", "\n", " # axis=1定义了会在向量的开头加入class_embedding\n", " self.concat = P.Concat(axis=1)\n", "\n", " self.pool = pool\n", " self.pos_dropout = nn.Dropout(keep_prob)\n", " self.norm = norm((embed_dim,))\n", " self.tile = P.Tile()\n", " self.transformer = TransformerEncoder(dim=embed_dim,\n", " num_layers=num_layers,\n", " num_heads=num_heads,\n", " mlp_dim=mlp_dim,\n", " keep_prob=keep_prob,\n", " attention_keep_prob=attention_keep_prob,\n", " drop_path_keep_prob=drop_path_keep_prob,\n", " activation=activation,\n", " norm=norm)\n", "\n", " def construct(self, x):\n", " \"\"\"ViT construct.\"\"\"\n", " x = self.patch_embedding(x)\n", "\n", " # class_embedding主要借鉴了BERT模型的用于文本分类时的思想\n", " # 在每一个word vector之前增加一个类别值,通常是加在向量的第一位\n", " cls_tokens = self.tile(self.cls_token, (x.shape[0], 1, 1))\n", " x = self.concat((cls_tokens, x))\n", " x += self.pos_embedding\n", "\n", " x = self.pos_dropout(x)\n", " x = self.transformer(x)\n", " x = self.norm(x)\n", "\n", " # 增加的class_embedding是一个可以学习的参数,经过网络的不断训练\n", " # 最终以输出向量的第一个维度的输出来决定最后的输出类别;\n", " x = x[:, 0]\n", "\n", " return x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "整体流程图如下所示:\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 模型训练与推理\n", "\n", "### 模型训练\n", "\n", "模型开始训练前,需要设定损失函数,优化器,回调函数等,直接调用MindSpore Vision提供的接口可以方便完成实例化。\n", "\n", "完整训练ViT模型需要很长的时间,实际应用时建议根据项目需要调整epoch_size,当正常输出每个Epoch的step信息时,意味着训练正在进行,通过模型输出可以查看当前训练的loss值和时间等指标。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "346281984B [02:29, 2316804.40B/s] \n", "Epoch:[ 0/ 10], step:[ 1/ 125], loss:[1.908/1.908], time:9580.221 ms, lr:0.00300\n", "Epoch:[ 0/ 10], step:[ 2/ 125], loss:[13.546/7.727], time:756.571 ms, lr:0.00300\n", "Epoch:[ 0/ 10], step:[ 3/ 125], loss:[6.916/7.457], time:767.550 ms, lr:0.00300\n", "Epoch:[ 0/ 10], step:[ 4/ 125], loss:[10.087/8.114], time:767.763 ms, lr:0.00300\n", "Epoch:[ 0/ 10], step:[ 5/ 125], loss:[7.964/8.084], time:767.676 ms, lr:0.00300\n", "Epoch:[ 0/ 10], step:[ 6/ 125], loss:[7.451/7.979], time:773.207 ms, lr:0.00300\n", "Epoch:[ 0/ 10], step:[ 7/ 125], loss:[7.434/7.901], time:769.799 ms, lr:0.00300\n", "...\n", "Epoch:[ 9/ 10], step:[ 120/ 125], loss:[6.524/6.403], time:788.128 ms, lr:0.00293\n", "Epoch:[ 9/ 10], step:[ 121/ 125], loss:[6.467/6.404], time:788.564 ms, lr:0.00293\n", "Epoch:[ 9/ 10], step:[ 122/ 125], loss:[6.483/6.405], time:788.837 ms, lr:0.00293\n", "Epoch:[ 9/ 10], step:[ 123/ 125], loss:[6.233/6.403], time:788.030 ms, lr:0.00293\n", "Epoch:[ 9/ 10], step:[ 124/ 125], loss:[6.781/6.406], time:791.158 ms, lr:0.00293\n", "Epoch:[ 9/ 10], step:[ 125/ 125], loss:[6.584/6.408], time:1614.970 ms, lr:0.00293\n", "Epoch time: 99938.756 ms, per step time: 799.510 ms, avg loss: 6.408\n" ] } ], "source": [ "import mindspore.nn as nn\n", "\n", "from mindvision.classification.models import vit_b_16\n", "from mindvision.engine.callback import LossMonitor\n", "from mindvision.engine.loss import CrossEntropySmooth\n", "\n", "# 定义超参数\n", "epoch_size = 10\n", "momentum = 0.9\n", "step_size = dataset_train.get_dataset_size()\n", "num_classes = 1000\n", "\n", "# 构建模型\n", "network = vit_b_16(num_classes=num_classes, image_size=resize, pretrained=True)\n", "\n", "# 定义递减的学习率\n", "lr = nn.cosine_decay_lr(min_lr=float(0),\n", " max_lr=0.003,\n", " total_step=epoch_size * step_size,\n", " step_per_epoch=step_size,\n", " decay_epoch=90)\n", "\n", "# 定义优化器\n", "network_opt = nn.Adam(network.trainable_params(), lr, momentum)\n", "\n", "# 定义损失函数\n", "network_loss = CrossEntropySmooth(sparse=True,\n", " reduction=\"mean\",\n", " smooth_factor=0.1,\n", " classes_num=num_classes)\n", "\n", "# 设定checkpoint\n", "ckpt_config = ms.CheckpointConfig(save_checkpoint_steps=step_size, keep_checkpoint_max=100)\n", "ckpt_callback = ms.ModelCheckpoint(prefix='vit_b_16', directory='./ViT', config=ckpt_config)\n", "\n", "# 初始化模型\n", "model = ms.Model(network, loss_fn=network_loss, optimizer=network_opt, metrics={\"acc\"})\n", "\n", "# 训练\n", "model.train(epoch_size,\n", " dataset_train,\n", " callbacks=[ckpt_callback, LossMonitor(lr)],\n", " dataset_sink_mode=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 模型验证\n", "\n", "模型验证过程主要应用了Model,ImageNet,CrossEntropySmooth和vit_b_16等接口。\n", "\n", "Model主要用于编译模型。\n", "\n", "ImageNet主要用于读取数据集。\n", "\n", "CrossEntropySmooth是损失函数实例化接口。\n", "\n", "vit_b_16是MindSpore Vision提供的模型实例化接口。\n", "\n", "通过改变ImageNet接口的split参数即可调用验证集。\n", "\n", "与训练过程相似,首先调用vit_b_16接口定义网络结构,加载预训练模型参数。随后设置损失函数,评价指标等,编译模型后进行验证。本案例采用了业界通用的评价标准Top_1_Accuracy和Top_5_Accuracy评价指标来评价模型表现。\n", "\n", "在本案例中,这两个指标代表了在输出的1000维向量中,以最大值或前5的输出值所代表的类别为预测结果时,模型预测的准确率。这两个指标的值越大,代表模型准确率越高。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'Top_1_Accuracy': 0.73524, 'Top_5_Accuracy': 0.91756}\n" ] } ], "source": [ "dataset_analyse = ImageNet(data_url,\n", " split=\"val\",\n", " num_parallel_workers=1,\n", " resize=resize,\n", " batch_size=batch_size)\n", "dataset_eval = dataset_analyse.run()\n", "\n", "network = vit_b_16(num_classes=num_classes, image_size=resize, pretrained=True)\n", "\n", "network_loss = CrossEntropySmooth(sparse=True,\n", " reduction=\"mean\",\n", " smooth_factor=0.1,\n", " classes_num=num_classes)\n", "\n", "# 定义评价指标\n", "eval_metrics = {'Top_1_Accuracy': nn.Top1CategoricalAccuracy(),\n", " 'Top_5_Accuracy': nn.Top5CategoricalAccuracy()}\n", "\n", "model = ms.Model(network, network_loss, metrics=eval_metrics)\n", "\n", "# 评估模型\n", "result = model.eval(dataset_eval)\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "从结果可以看出,由于我们加载了预训练模型参数,模型的Top_1_Accuracy和Top_5_Accuracy达到了很高的水平,实际项目中也可以以此准确率为标准。如果未使用预训练模型参数,则需要更多的epoch来训练。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 模型推理\n", "\n", "在进行模型推理之前,首先要定义一个对推理图片进行数据预处理的方法。该方法可以对我们的推理图片进行resize和normalize处理,这样才能与我们训练时的输入数据匹配。\n", "\n", "本案例采用了一张Doberman的图片作为推理图片来测试模型表现,期望模型可以给出正确的预测结果。" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "import mindspore.dataset.vision as vision\n", "\n", "# 数据预处理操作\n", "def infer_transform(dataset, columns_list, resize):\n", "\n", " mean = [0.485 * 255, 0.456 * 255, 0.406 * 255]\n", " std = [0.229 * 255, 0.224 * 255, 0.225 * 255]\n", "\n", " trans = [vision.Decode(),\n", " vision.Resize([resize, resize]),\n", " vision.Normalize(mean=mean, std=std),\n", " vision.HWC2CHW()]\n", "\n", " dataset = dataset.map(operations=trans,\n", " input_columns=columns_list[0],\n", " num_parallel_workers=1)\n", " dataset = dataset.batch(1)\n", "\n", " return dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "接下来,我们将调用模型的predict方法进行模型推理,利用read_dataset接口读推理图片路径,利用GeneratorDataset来生成测试集。\n", "\n", "在推理过程中,ImageNet接口主要负责对原数据集标签和模型输出进行配对。通过index2label就可以获取对应标签,再通过show_result接口将结果写在对应图片上。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{236: 'Doberman'}\n" ] } ], "source": [ "import numpy as np\n", "\n", "import mindspore.dataset as ds\n", "\n", "from mindvision.dataset.generator import DatasetGenerator\n", "from mindvision.dataset.download import read_dataset\n", "from mindvision.classification.utils.image import show_result\n", "\n", "# 读取推理图片\n", "image_list, image_label = read_dataset('./dataset/infer')\n", "columns_list = ('image', 'label')\n", "\n", "dataset_infer = ds.GeneratorDataset(DatasetGenerator(image_list, image_label),\n", " column_names=list(columns_list),\n", " num_parallel_workers=1)\n", "\n", "dataset_infer = infer_transform(dataset_infer, columns_list, resize)\n", "\n", "# 读取数据进行推理\n", "for i, image in enumerate(dataset_infer.create_dict_iterator(output_numpy=True)):\n", " image = image[\"image\"]\n", " image = ms.Tensor(image)\n", " prob = model.predict(image)\n", " label = np.argmax(prob.asnumpy(), axis=1)\n", "\n", " predict = dataset_analyse.index2label[int(label)]\n", " output = {int(label): predict}\n", " print(output)\n", " show_result(img=image_list[i], result=output, out_file=image_list[i])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "推理过程完成后,在推理文件夹下可以找到图片的推理结果,可以看出预测结果是Doberman,与期望结果相同,验证了模型的准确性。\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 总结\n", "\n", "本案例完成了一个ViT模型在ImageNet数据上进行训练,验证和推理的过程,其中,对关键的ViT模型结构和原理作了讲解。通过学习本案例,理解源码可以帮助用户掌握Multi-Head Attention,TransformerEncoder,pos_embedding等关键概念,如果要详细理解ViT的模型原理,建议基于源码更深层次的详细阅读,可以参考vision套件:\n", "\n", "<https://gitee.com/mindspore/vision/tree/r0.1/examples/classification/vit>。\n", "\n", "## 引用\n", "\n", "[1] Dosovitskiy, Alexey, et al. \\\"An image is worth 16x16 words: Transformers for image recognition at scale.\\\" arXiv preprint arXiv:2010.11929 (2020).\n", "\n", "[2] Vaswani, Ashish, et al. \\\"Attention is all you need.\\\"Advances in Neural Information Processing Systems. (2017)." ] } ], "metadata": { "interpreter": { "hash": "52ba6bae6fdbb5820a190a51a3809013ce8929101c66f6533d58a760dc505d49" }, "kernelspec": { "display_name": "MindSpore", "language": "python", "name": "mindspore" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 2 }