{ "cells": [ { "cell_type": "markdown", "source": [ "# 使用SentimentNet实现情感分类\n", "\n", "`GPU` `CPU` `进阶` `自然语言处理` `全流程`\n", "\n", "[![](https://gitee.com/mindspore/docs/raw/r1.5/resource/_static/logo_modelarts.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9taW5kc3BvcmUtd2Vic2l0ZS5vYnMuY24tbm9ydGgtNC5teWh1YXdlaWNsb3VkLmNvbS9ub3RlYm9vay9tYXN0ZXIvdHV0b3JpYWxzL3poX2NuL21pbmRzcG9yZV9zZW50aW1lbnRuZXQuaXB5bmI=&imageid=65f636a0-56cf-49df-b941-7d2a07ba8c8c) [![](https://gitee.com/mindspore/docs/raw/r1.5/resource/_static/logo_notebook.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r1.5/tutorials/zh_cn/mindspore_sentimentnet.ipynb) [![](https://gitee.com/mindspore/docs/raw/r1.5/resource/_static/logo_download_code.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r1.5/tutorials/zh_cn/mindspore_sentimentnet.py) [![](https://gitee.com/mindspore/docs/raw/r1.5/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r1.5/tutorials/source_zh_cn/intermediate/text/sentimentnet.ipynb)" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 概述\n", "\n", "情感分类是自然语言处理中文本分类问题的子集，属于自然语言处理最基础的应用。它是对带有感情色彩的主观性文本进行分析和推理的过程，即分析说话人的态度，是倾向正面还是反面。\n", "\n", "通常情况下，我们会把情感类别分为正面、反面和中性三类。虽然“面无表情”的评论也有不少；不过，大部分时候会只采用正面和反面的案例进行训练，下面这个数据集就是很好的例子。\n", "\n", "传统的文本主题分类问题的典型参考数据集为[20 Newsgroups](http://qwone.com/~jason/20Newsgroups/)，该数据集由20组新闻数据组成，包含约20000个新闻文档。\n", "其主题列表中有些类别的数据比较相似，例如comp.sys.ibm.pc.hardware和comp.sys.mac.hardware都是和电脑系统硬件相关的题目，相似度比较高。而有些主题类别的数据相对来说就毫无关联，例如misc.forsale和soc.religion.christian。\n", "\n", "就网络本身而言，文本主题分类的网络结构和情感分类的网络结构大致相似。在掌握了情感分类网络如何构造之后，可以很容易构造一个类似的网络，稍作调参即可用于文本主题分类任务。\n", "\n", "但在业务上下文侧，文本主题分类是分析文本讨论的客观内容，而情感分类是要从文本中得到它是否支持某种观点的信息。比如，“《阿甘正传》真是好看极了，影片主题明确，节奏流畅。”这句话，在文本主题分类是要将其归为类别为“电影”主题，而情感分类则要挖掘出这一影评的态度是正面还是负面。\n", "\n", "相对于传统的文本主题分类，情感分类较为简单，实用性也较强。常见的购物网站、电影网站都可以采集到相对高质量的数据集，也很容易给业务领域带来收益。例如，可以结合领域上下文，自动分析特定类型客户对当前产品的意见，可以分主题分用户类型对情感进行分析，以作针对性的处理，甚至基于此进一步推荐产品，提高转化率，带来更高的商业收益。\n", "\n", "特殊领域中，某些非极性词也充分表达了用户的情感倾向，比如下载使用APP时，“卡死了”、“下载太慢了”就表达了用户的负面情感倾向；股票领域中，“看涨”、“牛市”表达的就是用户的正面情感倾向。所以，本质上，我们希望模型能够在垂直领域中，挖掘出一些特殊的表达，作为极性词给情感分类系统使用：\n", "\n", "$垂直极性词 = 通用极性词 + 领域特有极性词$\n", "\n", "按照处理文本的粒度不同，情感分析可分为词语级、短语级、句子级、段落级以及篇章级等几个研究层次。这里以“段落级”为例，输入为一个段落，输出为影评是正面还是负面的信息。\n", "\n", "接下来，以IMDB影评情感分类为例来体验MindSpore在自然语言处理上的应用。\n", "\n", "> 本篇基于，GPU/CPU环境运行。" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 整体流程\n", "\n", "1. 准备环节。\n", "2. 加载数据集，进行数据处理。\n", "3. 定义网络。\n", "4. 定义优化器和损失函数。\n", "5. 使用网络训练数据，生成模型。\n", "6. 得到模型之后，使用验证数据集，查看模型精度情况。" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 准备环节\n", "\n", "### 下载数据集\n", "\n", "本次体验采用IMDB影评数据集作为实验数据。\n", "\n", "1. 下载IMDB影评数据集。\n", "\n", " 以下是负面影评（Negative）和正面影评（Positive）的案例。\n", "\n", " | Review | Label | \n", " |:---|:---:|\n", " | \"Quitting\" may be as much about exiting a pre-ordained identity as about drug withdrawal. As a rural guy coming to Beijing, class and success must have struck this young artist face on as an appeal to separate from his roots and far surpass his peasant parents' acting success. Troubles arise, however, when the new man is too new, when it demands too big a departure from family, history, nature, and personal identity. The ensuing splits, and confusion between the imaginary and the real and the dissonance between the ordinary and the heroic are the stuff of a gut check on the one hand or a complete escape from self on the other. | Negative | \n", " | This movie is amazing because the fact that the real people portray themselves and their real life experience and do such a good job it's like they're almost living the past over again. Jia Hongsheng plays himself an actor who quit everything except music and drugs struggling with depression and searching for the meaning of life while being angry at everyone especially the people who care for him most. | Positive |\n", " \n", " 将下载好的数据集解压并放在当前工作目录下的`datasets`目录下，由于数据集文件较多，解压过程耗时大约15分钟。其中，参数`--checkpoint=1000 --checkpoint-action=dot`表示每解压1000个文件将在底部追加打印一个黑点，如下所示。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 1, "source": [ "!wget https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/aclImdb_v1.tar.gz -N --no-check-certificate\n", "!mkdir -p datasets\n", "!if [ ! -d \"datasets/aclImdb\" ];then tar -C datasets --checkpoint=1000 --checkpoint-action=dot -xzf aclImdb_v1.tar.gz;fi" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "2. 下载GloVe文件\n", "\n", " 下载并解压GloVe文件到当前工作目录下的`datasets`目录下，并在所有Glove文件开头处添加如下所示新的一行，意思是总共读取400000个单词，每个单词用300纬度的词向量表示。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 2, "source": [ "!wget -N https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/glove.6B.zip --no-check-certificate\n", "!unzip -o glove.6B.zip -d datasets/glove\n", "!sed -i '1i 400000 300' datasets/glove/*\n", "!mkdir -p preprocess ckpt" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "3. 在当前工作目录创建名为`preprocess`的空目录，该目录将用于存储在数据集预处理操作中IMDB数据集转换为MindRecord格式后的文件。此时当前工作目录结构如下所示。\n", "\n", " ```text\n", " .\n", " ├── aclImdb_v1.tar.gz\n", " ├── ckpt\n", " ├── datasets\n", " │ ├── aclImdb\n", " │ │ ├── imdbEr.txt\n", " │ │ ├── imdb.vocab\n", " │ │ ├── README\n", " │ │ ├── test\n", " │ │ └── train\n", " │ └── glove\n", " │ ├── glove.6B.100d.txt\n", " │ ├── glove.6B.200d.txt\n", " │ ├── glove.6B.300d.txt\n", " │ └── glove.6B.50d.txt\n", " ├── glove.6B.zip\n", " ├── nlp_application.ipynb\n", " └── preprocess\n", " ```" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### 确定评价标准\n", "\n", "作为典型的分类问题，情感分类的评价标准可以比照普通的分类问题处理。常见的精度（Accuracy）、精准度（Precision）、召回率（Recall）和F_beta分数都可以作为参考。\n", "\n", "$精度（Accuracy）= 分类正确的样本数目 / 总样本数目$\n", "\n", "$精准度（Precision）= 真阳性样本数目 / 所有预测类别为阳性的样本数目$\n", "\n", "$召回率（Recall）= 真阳性样本数目 / 所有真实类别为阳性的样本数目$ \n", "\n", "$F1分数 = (2 * Precision * Recall) / (Precision + Recall)$\n", "\n", "在IMDB这个数据集中，正负样本数差别不大，可以简单地用精度（accuracy）作为分类器的衡量标准。\n", "\n", "### 确定网络\n", "\n", "我们使用基于LSTM构建的SentimentNet网络进行自然语言处理。\n", "\n", "> LSTM（Long short-term memory，长短期记忆）网络是一种时间循环神经网络，适合于处理和预测时间序列中间隔和延迟非常长的重要事件。" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### 配置运行信息和SentimentNet网络参数" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "1. 使用`parser`模块传入运行必要的信息。\n", " \n", " - `preprocess`：是否预处理数据集，默认为否。\n", " - `aclimdb_path`：数据集存放路径。\n", " - `glove_path`：GloVe文件存放路径。\n", " - `preprocess_path`：预处理数据集的结果文件夹。\n", " - `ckpt_path`：CheckPoint文件路径。\n", " - `pre_trained`：预加载CheckPoint文件。\n", " - `device_target`：指定GPU或CPU环境。\n", "\n", "2. 进行训练前，需要配置必要的信息，包括环境信息、执行的模式、后端信息及硬件信息。 \n", "\n", "运行以下一段代码中配置训练所需相关参数（详细的接口配置信息，请参见MindSpore官网`context.set_context`API接口说明）。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 4, "source": [ "import argparse\n", "from mindspore import context\n", "from easydict import EasyDict as edict\n", "\n", "\n", "# LSTM CONFIG\n", "lstm_cfg = edict({\n", " 'num_classes': 2,\n", " 'learning_rate': 0.1,\n", " 'momentum': 0.9,\n", " 'num_epochs': 10,\n", " 'batch_size': 64,\n", " 'embed_size': 300,\n", " 'num_hiddens': 100,\n", " 'num_layers': 2,\n", " 'bidirectional': True,\n", " 'save_checkpoint_steps': 390,\n", " 'keep_checkpoint_max': 10\n", "})\n", "\n", "cfg = lstm_cfg\n", "\n", "parser = argparse.ArgumentParser(description='MindSpore LSTM Example')\n", "parser.add_argument('--preprocess', type=str, default='false', choices=['true', 'false'],\n", " help='whether to preprocess data.')\n", "parser.add_argument('--aclimdb_path', type=str, default=\"./datasets/aclImdb\",\n", " help='path where the dataset is stored.')\n", "parser.add_argument('--glove_path', type=str, default=\"./datasets/glove\",\n", " help='path where the GloVe is stored.')\n", "parser.add_argument('--preprocess_path', type=str, default=\"./preprocess\",\n", " help='path where the pre-process data is stored.')\n", "parser.add_argument('--ckpt_path', type=str, default=\"./models/ckpt/nlp_application\",\n", " help='the path to save the checkpoint file.')\n", "parser.add_argument('--pre_trained', type=str, default=None,\n", " help='the pretrained checkpoint file path.')\n", "parser.add_argument('--device_target', type=str, default=\"GPU\", choices=['GPU', 'CPU'],\n", " help='the target device to run, support \"GPU\", \"CPU\". Default: \"GPU\".')\n", "args = parser.parse_args(['--device_target', 'GPU', '--preprocess', 'true'])\n", "\n", "context.set_context(\n", " mode=context.GRAPH_MODE,\n", " save_graphs=False,\n", " device_target=args.device_target)\n", "\n", "print(\"Current context loaded:\\n mode: {}\\n device_target: {}\".format(context.get_context(\"mode\"), context.get_context(\"device_target\")))" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Current context loaded:\n", " mode: 0\n", " device_target: GPU\n" ] } ], "metadata": { "scrolled": true } }, { "cell_type": "markdown", "source": [ "安装`gensim`依赖包。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 5, "source": [ "!pip install gensim" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n", "Requirement already satisfied: gensim in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (3.8.3)\n", "Requirement already satisfied: numpy>=1.11.3 in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from gensim) (1.17.5)\n", "Requirement already satisfied: six>=1.5.0 in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from gensim) (1.15.0)\n", "Requirement already satisfied: smart-open>=1.8.1 in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from gensim) (4.0.1)\n", "Requirement already satisfied: scipy>=0.18.1 in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from gensim) (1.3.3)\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 数据处理\n", "\n", "### 预处理数据集\n", "\n", "执行数据集预处理：\n", "\n", "- 定义`ImdbParser`类解析文本数据集，包括编码、分词、对齐、处理GloVe原始数据，使之能够适应网络结构。\n", "- 定义`convert_to_mindrecord`函数将数据集格式转换为MindRecord格式，便于MindSpore读取。函数`_convert_to_mindrecord`中`weight.txt`为数据预处理后自动生成的weight参数信息文件。\n", "- 调用`convert_to_mindrecord`函数执行数据集预处理。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 6, "source": [ "import os\n", "from itertools import chain\n", "import numpy as np\n", "import gensim\n", "from mindspore.mindrecord import FileWriter\n", "\n", "\n", "class ImdbParser():\n", " \"\"\"\n", " 按照下面的流程解析原始数据集，获得features与labels：\n", " sentence->tokenized->encoded->padding->features\n", " \"\"\"\n", "\n", " def __init__(self, imdb_path, glove_path, embed_size=300):\n", " self.__segs = ['train', 'test']\n", " self.__label_dic = {'pos': 1, 'neg': 0}\n", " self.__imdb_path = imdb_path\n", " self.__glove_dim = embed_size\n", " self.__glove_file = os.path.join(glove_path, 'glove.6B.' + str(self.__glove_dim) + 'd.txt')\n", "\n", " # properties\n", " self.__imdb_datas = {}\n", " self.__features = {}\n", " self.__labels = {}\n", " self.__vacab = {}\n", " self.__word2idx = {}\n", " self.__weight_np = {}\n", " self.__wvmodel = None\n", "\n", " def parse(self):\n", " \"\"\"\n", " 解析imdb data\n", " \"\"\"\n", " self.__wvmodel = gensim.models.KeyedVectors.load_word2vec_format(self.__glove_file)\n", "\n", " for seg in self.__segs:\n", " self.__parse_imdb_datas(seg)\n", " self.__parse_features_and_labels(seg)\n", " self.__gen_weight_np(seg)\n", "\n", " def __parse_imdb_datas(self, seg):\n", " \"\"\"\n", " 从原始文本中加载数据\n", " \"\"\"\n", " data_lists = []\n", " for label_name, label_id in self.__label_dic.items():\n", " sentence_dir = os.path.join(self.__imdb_path, seg, label_name)\n", " for file in os.listdir(sentence_dir):\n", " with open(os.path.join(sentence_dir, file), mode='r', encoding='utf8') as f:\n", " sentence = f.read().replace('\\n', '')\n", " data_lists.append([sentence, label_id])\n", " self.__imdb_datas[seg] = data_lists\n", "\n", " def __parse_features_and_labels(self, seg):\n", " \"\"\"\n", " 解析features与labels\n", " \"\"\"\n", " features = []\n", " labels = []\n", " for sentence, label in self.__imdb_datas[seg]:\n", " features.append(sentence)\n", " labels.append(label)\n", "\n", " self.__features[seg] = features\n", " self.__labels[seg] = labels\n", "\n", " self.__updata_features_to_tokenized(seg)\n", " self.__parse_vacab(seg)\n", " self.__encode_features(seg)\n", " self.__padding_features(seg)\n", "\n", " def __updata_features_to_tokenized(self, seg):\n", " \"\"\"\n", " 切分原始语句\n", " \"\"\"\n", " tokenized_features = []\n", " for sentence in self.__features[seg]:\n", " tokenized_sentence = [word.lower() for word in sentence.split(\" \")]\n", " tokenized_features.append(tokenized_sentence)\n", " self.__features[seg] = tokenized_features\n", "\n", " def __parse_vacab(self, seg):\n", " \"\"\"\n", " 构建词汇表\n", " \"\"\"\n", " tokenized_features = self.__features[seg]\n", " vocab = set(chain(*tokenized_features))\n", " self.__vacab[seg] = vocab\n", "\n", " # word_to_idx: {'hello': 1, 'world':111, ... '': 0}\n", " word_to_idx = {word: i + 1 for i, word in enumerate(vocab)}\n", " word_to_idx[''] = 0\n", " self.__word2idx[seg] = word_to_idx\n", "\n", " def __encode_features(self, seg):\n", " \"\"\" 词汇编码 \"\"\"\n", " word_to_idx = self.__word2idx['train']\n", " encoded_features = []\n", " for tokenized_sentence in self.__features[seg]:\n", " encoded_sentence = []\n", " for word in tokenized_sentence:\n", " encoded_sentence.append(word_to_idx.get(word, 0))\n", " encoded_features.append(encoded_sentence)\n", " self.__features[seg] = encoded_features\n", "\n", " def __padding_features(self, seg, maxlen=500, pad=0):\n", " \"\"\"\n", " 将所有features填充到相同的长度\n", " \"\"\"\n", " padded_features = []\n", " for feature in self.__features[seg]:\n", " if len(feature) >= maxlen:\n", " padded_feature = feature[:maxlen]\n", " else:\n", " padded_feature = feature\n", " while len(padded_feature) < maxlen:\n", " padded_feature.append(pad)\n", " padded_features.append(padded_feature)\n", " self.__features[seg] = padded_features\n", "\n", " def __gen_weight_np(self, seg):\n", " \"\"\"\n", " 使用gensim获取权重\n", " \"\"\"\n", " weight_np = np.zeros((len(self.__word2idx[seg]), self.__glove_dim), dtype=np.float32)\n", " for word, idx in self.__word2idx[seg].items():\n", " if word not in self.__wvmodel:\n", " continue\n", " word_vector = self.__wvmodel.get_vector(word)\n", " weight_np[idx, :] = word_vector\n", "\n", " self.__weight_np[seg] = weight_np\n", "\n", " def get_datas(self, seg):\n", " \"\"\"\n", " 返回 features, labels, weight\n", " \"\"\"\n", " features = np.array(self.__features[seg]).astype(np.int32)\n", " labels = np.array(self.__labels[seg]).astype(np.int32)\n", " weight = np.array(self.__weight_np[seg])\n", " return features, labels, weight\n", "\n", "\n", "\n", "def _convert_to_mindrecord(data_home, features, labels, weight_np=None, training=True):\n", " \"\"\"\n", " 将原始数据集转换为mindrecord格式\n", " \"\"\"\n", " if weight_np is not None:\n", " np.savetxt(os.path.join(data_home, 'weight.txt'), weight_np)\n", "\n", " # 写入mindrecord\n", " schema_json = {\"id\": {\"type\": \"int32\"},\n", " \"label\": {\"type\": \"int32\"},\n", " \"feature\": {\"type\": \"int32\", \"shape\": [-1]}}\n", "\n", " data_dir = os.path.join(data_home, \"aclImdb_train.mindrecord\")\n", " if not training:\n", " data_dir = os.path.join(data_home, \"aclImdb_test.mindrecord\")\n", "\n", " def get_imdb_data(features, labels):\n", " data_list = []\n", " for i, (label, feature) in enumerate(zip(labels, features)):\n", " data_json = {\"id\": i,\n", " \"label\": int(label),\n", " \"feature\": feature.reshape(-1)}\n", " data_list.append(data_json)\n", " return data_list\n", "\n", " writer = FileWriter(data_dir, shard_num=4)\n", " data = get_imdb_data(features, labels)\n", " writer.add_schema(schema_json, \"nlp_schema\")\n", " writer.add_index([\"id\", \"label\"])\n", " writer.write_raw_data(data)\n", " writer.commit()\n", "\n", "\n", "def convert_to_mindrecord(embed_size, aclimdb_path, preprocess_path, glove_path):\n", " \"\"\"\n", " 将原始数据集转换为mindrecord格式\n", " \"\"\"\n", " parser = ImdbParser(aclimdb_path, glove_path, embed_size)\n", " parser.parse()\n", "\n", " if not os.path.exists(preprocess_path):\n", " print(f\"preprocess path {preprocess_path} is not exist\")\n", " os.makedirs(preprocess_path)\n", "\n", " train_features, train_labels, train_weight_np = parser.get_datas('train')\n", " _convert_to_mindrecord(preprocess_path, train_features, train_labels, train_weight_np)\n", "\n", " test_features, test_labels, _ = parser.get_datas('test')\n", " _convert_to_mindrecord(preprocess_path, test_features, test_labels, training=False)\n", "\n", "if args.preprocess == \"true\":\n", " os.system(\"rm -f ./preprocess/aclImdb* weight*\")\n", " print(\"============== Starting Data Pre-processing ==============\")\n", " convert_to_mindrecord(cfg.embed_size, args.aclimdb_path, args.preprocess_path, args.glove_path)\n", " print(\"======================= Successful =======================\")\n", "\n" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "============== Starting Data Pre-processing ==============\n", "======================= Successful =======================\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "转换成功后会在`preprocess`目录下生成MindRecord文件，通常该操作在数据集不变的情况下，无需每次训练都执行，此时查看`preprocess`文件目录结构。\n", "\n", "```text\n", "preprocess\n", "├── aclImdb_test.mindrecord0\n", "├── aclImdb_test.mindrecord0.db\n", "├── aclImdb_test.mindrecord1\n", "├── aclImdb_test.mindrecord1.db\n", "├── aclImdb_test.mindrecord2\n", "├── aclImdb_test.mindrecord2.db\n", "├── aclImdb_test.mindrecord3\n", "├── aclImdb_test.mindrecord3.db\n", "├── aclImdb_train.mindrecord0\n", "├── aclImdb_train.mindrecord0.db\n", "├── aclImdb_train.mindrecord1\n", "├── aclImdb_train.mindrecord1.db\n", "├── aclImdb_train.mindrecord2\n", "├── aclImdb_train.mindrecord2.db\n", "├── aclImdb_train.mindrecord3\n", "├── aclImdb_train.mindrecord3.db\n", "└── weight.txt\n", "```" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "此时`preprocess`目录下的文件为：\n", "\n", "- 名称包含`aclImdb_train.mindrecord`的为转换后的MindRecord格式的训练数据集。\n", "- 名称包含`aclImdb_test.mindrecord`的为转换后的MindRecord格式的测试数据集。\n", "- `weight.txt`为预处理后自动生成的weight参数信息文件。" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "创建训练集：\n", "- 定义创建数据集函数`lstm_create_dataset`，创建训练集`ds_train`。\n", "- 通过`create_dict_iterator`方法创建字典迭代器，读取已创建的数据集`ds_train`中的数据。\n", "\n", "运行以下一段代码，创建数据集并读取第1个`batch`中的`label`数据列表，和第1个`batch`中第1个元素的`feature`数据。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 8, "source": [ "import os\n", "import mindspore.dataset as ds\n", "\n", "\n", "def lstm_create_dataset(data_home, batch_size, repeat_num=1, training=True):\n", " \"\"\"创建数据集\"\"\"\n", " ds.config.set_seed(1)\n", " data_dir = os.path.join(data_home, \"aclImdb_train.mindrecord0\")\n", " if not training:\n", " data_dir = os.path.join(data_home, \"aclImdb_test.mindrecord0\")\n", "\n", " data_set = ds.MindDataset(data_dir, columns_list=[\"feature\", \"label\"], num_parallel_workers=4)\n", "\n", " # 对数据集进行shuffle、batch与repeat操作\n", " data_set = data_set.shuffle(buffer_size=data_set.get_dataset_size())\n", " data_set = data_set.batch(batch_size=batch_size, drop_remainder=True)\n", " data_set = data_set.repeat(count=repeat_num)\n", "\n", " return data_set\n", "\n", "ds_train = lstm_create_dataset(args.preprocess_path, cfg.batch_size)\n", "\n", "iterator = next(ds_train.create_dict_iterator())\n", "first_batch_label = iterator[\"label\"].asnumpy()\n", "first_batch_first_feature = iterator[\"feature\"].asnumpy()[0]\n", "print(f\"The first batch contains label below:\\n{first_batch_label}\\n\")\n", "print(f\"The feature of the first item in the first batch is below vector:\\n{first_batch_first_feature}\")" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "The first batch contains label below:\n", "[0 0 1 1 1 0 0 0 0 0 0 1 0 0 1 0 1 1 0 0 0 0 1 1 1 0 1 1 1 0 0 1 0 0 1 0 1\n", " 0 0 0 0 1 0 0 1 1 1 0 0 0 1 1 1 1 0 1 0 0 1 1 0 1 1 0]\n", "\n", "The feature of the first item in the first batch is below vector:\n", "[249996 54143 184172 203651 229589 221693 185989 118515 64846 54704\n", " 19712 140286 54143 10035 223633 182804 110279 20992 185989 118515\n", " 54143 229589 124426 189682 129826 98619 251411 16315 100038 112995\n", " 237022 116461 30735 229874 38533 25750 44090 30219 30735 229874\n", " 171780 118515 65081 44090 74354 128277 82354 118515 215392 61497\n", " 212639 923 210633 105168 249996 54143 185745 184172 187822 185213\n", " 223619 100038 65443 73067 129442 44090 118515 156542 82301 111804\n", " 66658 184172 42988 95885 185989 76874 13192 171920 229589 156542\n", " 45558 5290 52959 80287 91542 91662 114496 112876 42988 192087\n", " 185507 186212 66658 233582 230976 143758 128277 215027 229589 154143\n", " 246234 167821 184159 40065 100038 112995 238258 180552 118515 95633\n", " 128277 118515 99327 98619 184172 24185 98619 184172 88217 128277\n", " 159969 128277 98619 96460 44090 118515 130663 710 128277 247284\n", " 118515 90362 185989 118515 90745 100038 112995 187822 42867 249652\n", " 118515 123509 239643 184172 118515 212864 185989 98619 161660 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0]\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 定义网络\n", "\n", "1. 导入初始化网络所需模块。\n", "2. 定义需要单层LSTM小算子堆叠的设备类型。\n", "3. 定义`lstm_default_state`函数来初始化网络参数及网络状态。\n", "4. 定义`stack_lstm_default_state`函数来初始化小算子堆叠需要的初始化网络参数及网络状态。\n", "5. 针对CPU场景，自定义单层LSTM小算子堆叠，来实现多层LSTM大算子功能。\n", "6. 使用`Cell`方法，定义网络结构（`SentimentNet`网络）。\n", "7. 实例化`SentimentNet`，创建网络，最后输出网络中加载的参数。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 9, "source": [ "import math\n", "import numpy as np\n", "from mindspore import Tensor, nn, context, Parameter, ParameterTuple\n", "from mindspore.common.initializer import initializer\n", "import mindspore.ops as ops\n", "\n", "# 当设备类型为CPU时采用堆叠类型的LSTM\n", "STACK_LSTM_DEVICE = [\"CPU\"]\n", "\n", "# 将短期记忆(h)和长期记忆(c)初始化为0\n", "def lstm_default_state(batch_size, hidden_size, num_layers, bidirectional):\n", " \"\"\"LSTM网络输入初始化\"\"\"\n", " num_directions = 2 if bidirectional else 1\n", " h = Tensor(np.zeros((num_layers * num_directions, batch_size, hidden_size)).astype(np.float32))\n", " c = Tensor(np.zeros((num_layers * num_directions, batch_size, hidden_size)).astype(np.float32))\n", " return h, c\n", "\n", "def stack_lstm_default_state(batch_size, hidden_size, num_layers, bidirectional):\n", " \"\"\"STACK LSTM网络输入初始化\"\"\"\n", " num_directions = 2 if bidirectional else 1\n", "\n", " h_list = c_list = []\n", " for _ in range(num_layers):\n", " h_list.append(Tensor(np.zeros((num_directions, batch_size, hidden_size)).astype(np.float32)))\n", " c_list.append(Tensor(np.zeros((num_directions, batch_size, hidden_size)).astype(np.float32)))\n", " h, c = tuple(h_list), tuple(c_list)\n", " return h, c\n", "\n", "\n", "class StackLSTM(nn.Cell):\n", " \"\"\"\n", " 实现堆叠LSTM\n", " \"\"\"\n", "\n", " def __init__(self,\n", " input_size,\n", " hidden_size,\n", " num_layers=1,\n", " has_bias=True,\n", " batch_first=False,\n", " dropout=0.0,\n", " bidirectional=False):\n", " super(StackLSTM, self).__init__()\n", " self.num_layers = num_layers\n", " self.batch_first = batch_first\n", " self.transpose = ops.Transpose()\n", "\n", " num_directions = 2 if bidirectional else 1\n", "\n", " input_size_list = [input_size]\n", " for i in range(num_layers - 1):\n", " input_size_list.append(hidden_size * num_directions)\n", "\n", " # LSTMCell为单层RNN结构，通过堆叠LSTMCell可完成StackLSTM\n", " layers = []\n", " for i in range(num_layers):\n", " layers.append(nn.LSTMCell(input_size=input_size_list[i],\n", " hidden_size=hidden_size,\n", " has_bias=has_bias,\n", " batch_first=batch_first,\n", " bidirectional=bidirectional,\n", " dropout=dropout))\n", "\n", " # 权重初始化\n", " weights = []\n", " for i in range(num_layers):\n", " weight_size = (input_size_list[i] + hidden_size) * num_directions * hidden_size * 4\n", " if has_bias:\n", " bias_size = num_directions * hidden_size * 4\n", " weight_size = weight_size + bias_size\n", "\n", " stdv = 1 / math.sqrt(hidden_size)\n", " w_np = np.random.uniform(-stdv, stdv, (weight_size, 1, 1)).astype(np.float32)\n", "\n", " weights.append(Parameter(initializer(Tensor(w_np), w_np.shape), name=\"weight\" + str(i)))\n", "\n", " self.lstms = layers\n", " self.weight = ParameterTuple(tuple(weights))\n", "\n", " def construct(self, x, hx):\n", " \"\"\"构建网络\"\"\"\n", " if self.batch_first:\n", " x = self.transpose(x, (1, 0, 2))\n", " h, c = hx\n", " hn = cn = None\n", " for i in range(self.num_layers):\n", " x, hn, cn, _, _ = self.lstms[i](x, h[i], c[i], self.weight[i])\n", " if self.batch_first:\n", " x = self.transpose(x, (1, 0, 2))\n", " return x, (hn, cn)\n", "\n", "\n", "class SentimentNet(nn.Cell):\n", " \"\"\"构建SentimentNet\"\"\"\n", "\n", " def __init__(self,\n", " vocab_size,\n", " embed_size,\n", " num_hiddens,\n", " num_layers,\n", " bidirectional,\n", " num_classes,\n", " weight,\n", " batch_size):\n", " super(SentimentNet, self).__init__()\n", " # 对数据中的词汇进行降维\n", " self.embedding = nn.Embedding(vocab_size,\n", " embed_size,\n", " embedding_table=weight)\n", " self.embedding.embedding_table.requires_grad = False\n", " self.trans = ops.Transpose()\n", " self.perm = (1, 0, 2)\n", "\n", " # 判断是否需要堆叠LSTM\n", " if context.get_context(\"device_target\") in STACK_LSTM_DEVICE:\n", "\n", " self.encoder = StackLSTM(input_size=embed_size,\n", " hidden_size=num_hiddens,\n", " num_layers=num_layers,\n", " has_bias=True,\n", " bidirectional=bidirectional,\n", " dropout=0.0)\n", " self.h, self.c = stack_lstm_default_state(batch_size, num_hiddens, num_layers, bidirectional)\n", " else:\n", " self.encoder = nn.LSTM(input_size=embed_size,\n", " hidden_size=num_hiddens,\n", " num_layers=num_layers,\n", " has_bias=True,\n", " bidirectional=bidirectional,\n", " dropout=0.0)\n", " self.h, self.c = lstm_default_state(batch_size, num_hiddens, num_layers, bidirectional)\n", "\n", " self.concat = ops.Concat(1)\n", " if bidirectional:\n", " self.decoder = nn.Dense(num_hiddens * 4, num_classes)\n", " else:\n", " self.decoder = nn.Dense(num_hiddens * 2, num_classes)\n", "\n", " def construct(self, inputs):\n", " # input：(64,500,300)\n", " embeddings = self.embedding(inputs)\n", " embeddings = self.trans(embeddings, self.perm)\n", " output, _ = self.encoder(embeddings, (self.h, self.c))\n", " # states[i] size(64,200) -> encoding.size(64,400)\n", " encoding = self.concat((output[0], output[499]))\n", " outputs = self.decoder(encoding)\n", " return outputs\n", "\n", "embedding_table = np.loadtxt(os.path.join(args.preprocess_path, \"weight.txt\")).astype(np.float32)\n", "network = SentimentNet(vocab_size=embedding_table.shape[0],\n", " embed_size=cfg.embed_size,\n", " num_hiddens=cfg.num_hiddens,\n", " num_layers=cfg.num_layers,\n", " bidirectional=cfg.bidirectional,\n", " num_classes=cfg.num_classes,\n", " weight=Tensor(embedding_table),\n", " batch_size=cfg.batch_size)\n", "\n", "print(network.parameters_dict(recurse=True))" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "OrderedDict([('embedding.embedding_table', Parameter (name=embedding.embedding_table, value=Tensor(shape=[252193, 300], dtype=Float32, value=\n", "[[ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00 ... 0.00000000e+00, 0.00000000e+00, 0.00000000e+00],\n", " [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00 ... 0.00000000e+00, 0.00000000e+00, 0.00000000e+00],\n", " [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00 ... 0.00000000e+00, 0.00000000e+00, 0.00000000e+00],\n", " ...\n", " [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00 ... 0.00000000e+00, 0.00000000e+00, 0.00000000e+00],\n", " [-2.64310002e-01, 2.03539997e-01, -1.07670002e-01 ... 3.17510009e-01, -6.45749986e-01, 4.42129999e-01],\n", " [-2.82150000e-01, 2.53950000e-01, 3.94300014e-01 ... 1.75999999e-01, 7.86110014e-02, -7.89420009e-02]]))), ('encoder.weight', Parameter (name=encoder.weight, value=Tensor(shape=[563200, 1, 1], dtype=Float32, value=\n", "[[[-1.65955983e-02]],\n", " [[ 4.40648980e-02]],\n", " [[-9.99771282e-02]],\n", " ...\n", " [[-6.54547513e-02]],\n", " [[ 1.46641862e-02]],\n", " [[-2.03442890e-02]]]))), ('decoder.weight', Parameter (name=decoder.weight, value=Tensor(shape=[2, 400], dtype=Float32, value=\n", "[[ 8.68825766e-04, 1.55616635e-02, -3.46743106e-03 ... -1.70452073e-02, 6.96127317e-05, -1.37791187e-02],\n", " [ 5.52378222e-03, -2.03212705e-02, 1.68735497e-02 ... 1.62047185e-02, 5.66494651e-03, -1.49743268e-02]]))), ('decoder.bias', Parameter (name=decoder.bias, value=Tensor(shape=[2], dtype=Float32, value= [ 0.00000000e+00, 0.00000000e+00])))])\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 训练并保存模型\n", "\n", "运行以下一段代码，创建优化器和损失函数模型，加载训练数据集（`ds_train`）并配置好`CheckPoint`生成信息，然后使用`model.train`接口，进行模型训练。根据输出可以看到loss值随着训练逐步降低，最后达到0.262左右。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 10, "source": [ "from mindspore import Model\n", "from mindspore.train.callback import CheckpointConfig, ModelCheckpoint, TimeMonitor, LossMonitor\n", "from mindspore.nn import Accuracy\n", "from mindspore import nn\n", "\n", "os.system(\"rm -f {0}/*.ckpt {0}/*.meta\".format(args.ckpt_path))\n", "loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean')\n", "opt = nn.Momentum(network.trainable_params(), cfg.learning_rate, cfg.momentum)\n", "model = Model(network, loss, opt, {'acc': Accuracy()})\n", "loss_cb = LossMonitor(per_print_times=78)\n", "print(\"============== Starting Training ==============\")\n", "config_ck = CheckpointConfig(save_checkpoint_steps=cfg.save_checkpoint_steps,\n", " keep_checkpoint_max=cfg.keep_checkpoint_max)\n", "ckpoint_cb = ModelCheckpoint(prefix=\"lstm\", directory=args.ckpt_path, config=config_ck)\n", "time_cb = TimeMonitor(data_size=ds_train.get_dataset_size())\n", "if args.device_target == \"CPU\":\n", " model.train(cfg.num_epochs, ds_train, callbacks=[time_cb, ckpoint_cb, loss_cb], dataset_sink_mode=False)\n", "else:\n", " model.train(cfg.num_epochs, ds_train, callbacks=[time_cb, ckpoint_cb, loss_cb])\n", "print(\"============== Training Success ==============\")" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "============== Starting Training ==============\n", "epoch: 1 step: 78, loss is 0.2971678\n", "epoch: 1 step: 156, loss is 0.30519545\n", "epoch: 1 step: 234, loss is 0.2370582\n", "epoch: 1 step: 312, loss is 0.25823578\n", "epoch: 1 step: 390, loss is 0.2899053\n", "Epoch time: 27745.798, per step time: 71.143\n", "epoch: 2 step: 78, loss is 0.20885809\n", "epoch: 2 step: 156, loss is 0.2168142\n", "epoch: 2 step: 234, loss is 0.14624771\n", "epoch: 2 step: 312, loss is 0.2152691\n", "epoch: 2 step: 390, loss is 0.3756763\n", "Epoch time: 27407.312, per step time: 70.275\n", "epoch: 3 step: 78, loss is 0.116764486\n", "epoch: 3 step: 156, loss is 0.20790516\n", "epoch: 3 step: 234, loss is 0.2118046\n", "epoch: 3 step: 312, loss is 0.18587393\n", "epoch: 3 step: 390, loss is 0.25241128\n", "Epoch time: 27251.069, per step time: 69.875\n", "epoch: 4 step: 78, loss is 0.11729147\n", "epoch: 4 step: 156, loss is 0.16071466\n", "epoch: 4 step: 234, loss is 0.43869072\n", "epoch: 4 step: 312, loss is 0.37149796\n", "epoch: 4 step: 390, loss is 0.18670222\n", "Epoch time: 27441.597, per step time: 70.363\n", "epoch: 5 step: 78, loss is 0.08070815\n", "epoch: 5 step: 156, loss is 0.143559\n", "epoch: 5 step: 234, loss is 0.292204\n", "epoch: 5 step: 312, loss is 0.07726648\n", "epoch: 5 step: 390, loss is 0.15458854\n", "Epoch time: 27602.059, per step time: 70.775\n", "epoch: 6 step: 78, loss is 0.16412595\n", "epoch: 6 step: 156, loss is 0.1664415\n", "epoch: 6 step: 234, loss is 0.1091502\n", "epoch: 6 step: 312, loss is 0.112443276\n", "epoch: 6 step: 390, loss is 0.14458877\n", "Epoch time: 27568.301, per step time: 70.688\n", "epoch: 7 step: 78, loss is 0.110504806\n", "epoch: 7 step: 156, loss is 0.079935536\n", "epoch: 7 step: 234, loss is 0.29199448\n", "epoch: 7 step: 312, loss is 0.1512347\n", "epoch: 7 step: 390, loss is 0.3185295\n", "Epoch time: 27512.058, per step time: 70.544\n", "epoch: 8 step: 78, loss is 0.22663717\n", "epoch: 8 step: 156, loss is 0.21799277\n", "epoch: 8 step: 234, loss is 0.13152371\n", "epoch: 8 step: 312, loss is 0.168206\n", "epoch: 8 step: 390, loss is 0.1784227\n", "Epoch time: 27545.180, per step time: 70.629\n", "epoch: 9 step: 78, loss is 0.27715153\n", "epoch: 9 step: 156, loss is 0.085485235\n", "epoch: 9 step: 234, loss is 0.35549596\n", "epoch: 9 step: 312, loss is 0.1265975\n", "epoch: 9 step: 390, loss is 0.081303015\n", "Epoch time: 27582.971, per step time: 70.726\n", "epoch: 10 step: 78, loss is 0.19696395\n", "epoch: 10 step: 156, loss is 0.03179455\n", "epoch: 10 step: 234, loss is 0.11651886\n", "epoch: 10 step: 312, loss is 0.050257515\n", "epoch: 10 step: 390, loss is 0.025655827\n", "Epoch time: 27546.935, per step time: 70.633\n", "============== Training Success ==============\n" ] } ], "metadata": { "scrolled": true } }, { "cell_type": "markdown", "source": [ "## 模型验证\n", "\n", "创建并加载验证数据集（`ds_eval`），加载由**训练**保存的CheckPoint文件，进行验证，查看模型质量，此步骤用时约30秒。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 11, "source": [ "from mindspore import load_checkpoint, load_param_into_net\n", "args.ckpt_path_saved = f'{args.ckpt_path}/lstm-{cfg.num_epochs}_390.ckpt'\n", "print(\"============== Starting Testing ==============\")\n", "ds_eval = lstm_create_dataset(args.preprocess_path, cfg.batch_size, training=False)\n", "param_dict = load_checkpoint(args.ckpt_path_saved)\n", "load_param_into_net(network, param_dict)\n", "if args.device_target == \"CPU\":\n", " acc = model.eval(ds_eval, dataset_sink_mode=False)\n", "else:\n", " acc = model.eval(ds_eval)\n", "print(\"============== {} ==============\".format(acc))\n" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "============== Starting Testing ==============\n", "============== {'acc': 0.8476362179487179} ==============\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### 训练结果评价\n", "\n", "根据以上一段代码的输出可以看到，在经历了10轮epoch之后，使用验证的数据集，对文本的情感分析正确率在85%左右，达到一个基本满意的结果。" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 总结\n", "\n", "以上便完成了MindSpore自然语言处理应用的体验，我们通过本次体验全面了解了如何使用MindSpore进行自然语言中处理情感分类问题，理解了如何通过定义和初始化基于LSTM的`SentimentNet`网络进行训练模型及验证正确率。" ], "metadata": {} } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }