{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 图数据加载与处理\n", "\n", "[![在线运行](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/resource/_static/logo_modelarts.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9vYnMuZHVhbHN0YWNrLmNuLW5vcnRoLTQubXlodWF3ZWljbG91ZC5jb20vbWluZHNwb3JlLXdlYnNpdGUvbm90ZWJvb2svcjIuMC90dXRvcmlhbHMvemhfY24vYWR2YW5jZWQvZGF0YXNldC9taW5kc3BvcmVfYXVnbWVudF9ncmFwaF9kYXRhLmlweW5i=&imageid=e225a9aa-230a-4ea5-a538-b5faed64a6a6) [![下载Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/resource/_static/logo_notebook.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r2.0/tutorials/zh_cn/advanced/dataset/mindspore_augment_graph_data.ipynb) \n", "[![下载样例代码](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/resource/_static/logo_download_code.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r2.0/tutorials/zh_cn/advanced/dataset/mindspore_augment_graph_data.py) \n", "[![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r2.0/tutorials/source_zh_cn/advanced/dataset/augment_graph_data.ipynb)\n", "\n", "MindSpore提供的`mindspore.dataset`模块可以帮助用户构建数据集对象，分批次地读取文本数据。同时，在各个数据集类中还内置了数据处理和数据分词算子，使得数据在训练过程中能够像经过pipeline管道的水一样源源不断地流向训练系统，提升数据训练效果。\n", "\n", "本章将简要演示如何使用MindSpore加载和处理图数据。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 图的概念\n", "\n", "在介绍图数据的读取及增强之前，先介绍图的基本概念进行，有助于后续内容更好地理解。通常一个图（graph) `G`是由一系列的节点(vertices) `V`以及边（edges）`E`组成的，每条边都连接着图中的两个节点，用公式可表述为：\n", "\n", "$$G = F(V, E)$$\n", "\n", "简单的图如下所示。\n", "\n", "![basicGraph.png](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/tutorials/source_zh_cn/advanced/dataset/images/basic_graph.png)\n", "\n", "图中包含节点V = {a, b, c, d}，和边E = {(a, b), (b, c), (c, d), (d, b)}，针对图中的连接关系通常需借助数学的方式进行描述，如常用的基于邻接矩阵的方式，用于描述上述图连接关系的矩阵C如下，其中a、 b、c、d对应为第1、2、 3、4个节点。\n", "\n", "$$C=\\begin{bmatrix}\n", "1&1&0&0\\\\\n", "1&1&1&1\\\\\n", "0&1&1&1\\\\\n", "0&1&1&1\\\\\n", "\\end{bmatrix}$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 数据集准备环节\n", "\n", "1. 数据集介绍\n", "\n", "常用的图数据集包含Cora、Citeseer、PubMed等，在本文中我们基于Cora数据集进行介绍。\n", "\n", "> 原始数据集可以从[ucsc网站](https://linqs-data.soe.ucsc.edu/public/lbc/cora.tgz)进行下载，本文采用kimiyoung提供的[预处理后的版本](https://github.com/kimiyoung/planetoid)[[1]](#参考文献)。\n", "\n", "其中，Cora数据集主体部分(`cora.content`)包含2708条样本，每条样本描述1篇科学论文的信息，论文都属于7个类别中的一个。每条样本数据包含三部分，依次为论文编号、论文的词向量（一个1433位的二进制）、论文的类别；引用数据集部分(`cora.cites`)包含5429行，每行包含两个论文编号，表示第二篇论文对第一篇论文进行了引用。\n", "\n", "2. 数据集下载\n", "\n", "以下示例代码将cora数据集下载并解压到指定位置：" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from download import download\n", "import os\n", "import shutil\n", "\n", "if not os.path.exists(\"./cora\"):\n", " os.mkdir(\"./cora\")\n", " if not os.path.exists(\"./planetoid\"):\n", " url = \"https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/planetoid.tar.gz\"\n", " download(url, \"./\", kind=\"tar.gz\", replace=True)\n", " content = os.listdir(\"./planetoid/data\")\n", " new_content = []\n", " for name in content:\n", " if \"cora\" in name:\n", " new_content.append(name)\n", " for name in new_content:\n", " path = \"./planetoid/data/\"+name\n", " shutil.copy(path, \"./cora\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "下载预处理后的cora数据集目录如下所示。\n", "\n", "```text\n", "./cora\n", "├── ind.cora.allx\n", "├── ind.cora.ally\n", "├── ind.cora.graph\n", "├── ind.cora.test.index\n", "├── ind.cora.tx\n", "├── ind.cora.ty\n", "├── ind.cora.x\n", "├── ind.cora.y\n", "├── trans.cora.graph\n", "├── trans.cora.tx\n", "├── trans.cora.ty\n", "├── trans.cora.x\n", "└── trans.cora.y\n", "```\n", "\n", "3. 数据集格式转换\n", "\n", "将数据集转换为MindSpore Record格式，可借助models仓库提供的转换脚本进行转换，生成的MindSpore Record文件在`./cora_mindrecord`路径下。" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "if not os.path.exists(\"./cora_mindrecord\"):\n", " os.mkdir(\"./cora_mindrecord\")\n", " url = \"https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/graph_to_mindrecord.tar.gz\"\n", " download(url, \"./\", kind=\"tar.gz\", replace=True)\n", " os.system('python graph_to_mindrecord/writer.py --mindrecord_script cora --mindrecord_file \"./cora_mindrecord/cora_mr\" --mindrecord_partitions 1 --mindrecord_header_size_by_bit 18 --mindrecord_page_size_by_bit 20 --graph_api_args \"./cora\"')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 加载数据集\n", "\n", "MindSpore目前支持加载文本领域常用的经典数据集和多种数据存储格式下的数据集，用户也可以通过构建自定义数据集类实现自定义方式的数据加载。\n", "\n", "下面演示使用`MindSpore.dataset`模块中的`MindDataset`类加载上述已转换成MindSpore Record格式的cora数据集。\n", "\n", "1. 配置数据集目录，创建数据集对象。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import mindspore.dataset as ds\n", "import numpy as np\n", "\n", "data_file = \"./cora_mindrecord/cora_mr\"\n", "dataset = ds.GraphData(data_file)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "2. 访问对应的接口，获取图信息及特性、标签内容。" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "graph info: {'node_type': [0], 'edge_type': [0], 'node_num': {0: 2708}, 'edge_num': {0: 10858}, 'node_feature_type': [1, 2], 'edge_feature_type': [], 'graph_feature_type': []}\n", "node shape: 2708\n", "features shape: (2708, 1433)\n", "labels shape: (2708,)\n", "labels: [3 4 4 ... 3 3 3]\n" ] } ], "source": [ "# 查看图中结构信息\n", "graph = dataset.graph_info()\n", "print(\"graph info:\", graph)\n", "\n", "# 获取所有的节点信息\n", "nodes = dataset.get_all_nodes(0)\n", "nodes_list = nodes.tolist()\n", "print(\"node shape:\", len(nodes_list))\n", "\n", "# 获取特征和标签信息，总共2708条数据\n", "# 每条数据中特征信息是用于描述论文i，长度为1433的二进制表示，标签信息指的是论文所属的种类\n", "raw_tensor = dataset.get_node_feature(nodes_list, [1, 2])\n", "features, labels = raw_tensor[0], raw_tensor[1]\n", "\n", "print(\"features shape:\", features.shape)\n", "print(\"labels shape:\", labels.shape)\n", "print(\"labels:\", labels)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 数据处理\n", "\n", "下面演示构建pipeline，对节点进行采样等操作。\n", "\n", "1. 获取节点的邻居节点，构造邻接矩阵。" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "neighbor:\n", " [[ 0 633 1862 ... -1 -1 -1]\n", " [ 1 2 652 ... -1 -1 -1]\n", " [ 2 1986 332 ... -1 -1 -1]\n", " ...\n", " [2705 287 -1 ... -1 -1 -1]\n", " [2706 165 2707 ... -1 -1 -1]\n", " [2707 598 165 ... -1 -1 -1]]\n" ] } ], "source": [ "neighbor = dataset.get_all_neighbors(nodes_list, 0)\n", "\n", "# neighbor的第一列是node_id，第二列到最后一列存储的是第一列的邻居节点，如果不存在这么多，则用-1补齐。\n", "print(\"neighbor:\\n\", neighbor)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "2. 依据节点的邻居节点信息，构造邻接矩阵。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "adj:\n", " [[0. 0. 0. ... 0. 0. 0.]\n", " [0. 0. 1. ... 0. 0. 0.]\n", " [0. 1. 0. ... 0. 0. 0.]\n", " ...\n", " [0. 0. 0. ... 0. 0. 0.]\n", " [0. 0. 0. ... 0. 0. 1.]\n", " [0. 0. 0. ... 0. 1. 0.]]\n" ] } ], "source": [ "nodes_num = labels.shape[0]\n", "node_map = {node_id: index for index, node_id in enumerate(nodes_list)}\n", "adj = np.zeros([nodes_num, nodes_num], dtype=np.float32)\n", "\n", "for index, value in np.ndenumerate(neighbor):\n", " # neighbor的第一列是node_id，第二列到最后一列存储的是第一列的邻居节点，如果不存在这么多，则用-1补齐。\n", " if value >= 0 and index[1] > 0:\n", " adj[node_map[neighbor[index[0], 0]], node_map[value]] = 1\n", "\n", "print(\"adj:\\n\", adj)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "3. 节点采样，支持常见的多次跳跃采样与随机游走采样方法等。\n", " 多跳邻接点采样如（a）图所示，当次采样的节点将作为下次采样的起始点；随机游走方式如（b）图所示，随机选择一条路径依次遍历相邻的节点，对应图中则选择了从Vi到Vj的游走路径。\n", "\n", "![graph](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0/tutorials/source_zh_cn/advanced/dataset/images/graph_sample.png)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "neighbor:\n", " [[ 0 1862 633]\n", " [ 1 654 2]\n", " [ 2 1666 1]\n", " [ 3 2544 2544]\n", " [ 4 1256 1761]\n", " [ 5 1659 1629]\n", " [ 6 1416 373]\n", " [ 7 208 208]\n", " [ 8 281 1996]\n", " [ 9 723 2614]\n", " [ 10 2545 476]\n", " [ 11 1655 1839]\n", " [ 12 2662 1001]\n", " [ 13 1810 1701]\n", " [ 14 2668 2077]\n", " [ 15 1093 1271]\n", " [ 16 2444 970]\n", " [ 17 2140 1315]\n", " [ 18 2082 1560]\n", " [ 19 1939 1939]\n", " [ 20 2375 2269]]\n", "walks:\n", " [[ 0 1862]\n", " [ 1 654]\n", " [ 2 1666]\n", " [ 3 2544]\n", " [ 4 2176]\n", " [ 5 1659]\n", " [ 6 1042]\n", " [ 7 208]\n", " [ 8 281]\n", " [ 9 723]\n", " [ 10 2545]\n", " [ 11 1839]\n", " [ 12 2662]\n", " [ 13 1701]\n", " [ 14 2034]\n", " [ 15 1271]\n", " [ 16 2642]\n", " [ 17 2140]\n", " [ 18 2145]\n", " [ 19 1939]\n", " [ 20 2269]]\n" ] } ], "source": [ "# 基于多次跳跃进行节点采样\n", "neighbor = dataset.get_sampled_neighbors(node_list=nodes_list[0:21], neighbor_nums=[2], neighbor_types=[0])\n", "print(\"neighbor:\\n\", neighbor)\n", "\n", "# 基于随机游走进行节点采样\n", "meta_path = [0]\n", "walks = dataset.random_walk(nodes_list[0:21], meta_path)\n", "print(\"walks:\\n\", walks)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 由于上面代码中游走采样存在随机性，因此在执行时可能会出现不同的打印结果。\n", "\n", "4. 通过节点获取边/通过边获取节点。" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "part edges: [0 1 2 3 4 5 6 7 8 9]\n", "nodes: [[ 0 633]\n", " [ 0 1862]\n", " [ 0 2582]\n", " [ 1 2]\n", " [ 1 652]\n", " [ 1 654]\n", " [ 2 1986]\n", " [ 2 332]\n", " [ 2 1666]\n", " [ 2 1]]\n", "edges: [ 0 4 7 11]\n" ] } ], "source": [ "# 通过边获取节点\n", "part_edges = dataset.get_all_edges(0)[:10]\n", "nodes = dataset.get_nodes_from_edges(part_edges)\n", "print(\"part edges:\", part_edges)\n", "print(\"nodes:\", nodes)\n", "\n", "# 通过节点获取边\n", "nodes_pair_list = [(0, 633), (1, 652), (2, 332), (3, 2544)]\n", "edges = dataset.get_edges_from_nodes(nodes_pair_list)\n", "print(\"edges:\", edges)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 参考文献\n", "\n", "[1] Yang Z, Cohen W, Salakhudinov R. [Revisiting semi-supervised learning with graph embeddings](http://proceedings.mlr.press/v48/yanga16.pdf)." ] } ], "metadata": { "kernelspec": { "display_name": "MindSpore", "language": "python", "name": "mindspore" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5" } }, "nbformat": 4, "nbformat_minor": 4 }