Building Custom Operators Offline

Overview

Our Conversion Tool is a highly flexible tool. In addition to the basic ability of model converter, we have designed a set of registration mechanism, which allows users to expand, including node-parse extension, model-parse extension and graph-optimization extension. The users can combined them as needed to achieve their own intention.

node-parse extension: The users can define the process to parse a certain node of a model by themselves, which only support ONNX, CAFFE, TF and TFLITE. The related interface is NodeParser, NodeParserRegistry. model-parse extension: The users can define the process to parse a model by themselves, which only support ONNX, CAFFE, TF and TFLITE. The related interface is ModelParser, ModelParserRegistry. graph-optimization extension: After parsing a model, a graph structure defined by MindSpore will show up and then, the users can define the process to optimize the parsed graph. The related interfaces are PassBase, PassPosition, PassRegistry.

The node-parse extension needs to rely on the flatbuffers, protobuf and the serialization files of third-party frameworks, at the same time, the version of flatbuffers and the protobuf needs to be consistent with that of the released package, the serialized files must be compatible with that used by the released package. Note that the flatbuffers, protobuf and the serialization files are not provided in the released package, users need to compile and generate the serialized files by themselves. The users can obtain the basic information about flabuffers, probobuf, ONNX prototype file, CAFFE prototype file, TF prototype file and TFLITE prototype file from the MindSpore WareHouse.

MindSpore Lite alse providers a series of registration macros to facilitate user access. These macros include node-parse registration REG_NODE_PARSER, model-parse registration REG_MODEL_PARSER, graph-optimization registration REG_PASS and graph-optimization scheduled registration REG_SCHEDULED_PASS

The expansion capability of MindSpore Lite conversion tool only support on Linux system currently.

In this chapter, we will show the users a sample of extending Mindspore Lite converter tool, covering the example of expanding node, example of optimizing graph, compiling and linking. The example will help the users understand the extension ability as soon as possible.

Due to that model-parse extension is a modular extension ability, the chapter will not introduce in details. However, we still provide the users with a simplified unit case for inference.

The chapter takes a add.tflite, which only includes an opreator of adding, as an example. We will show the users how to convert the single operator of adding to that of Custom and finally obtain a model which only includs a single operator of custom.

The code related to the example can be obtained from the directory mindspore/lite/examples/converter_extend.

Node Extension

Self-defined node-parse: The users need to inherit the base class NodeParser, and then, choose a interface to override according to model frameworks.
Node-parse Registration: The users can directly call the registration interface REG_NODE_PARSER, so that the self-defined node-parse will be registered in the converter tool of MindSpore Lite.

class AddParserTutorial : public NodeParser {  // inherit the base class
 public:
  AddParserTutorial() = default;
  ~AddParserTutorial() = default;
  ops::PrimitiveC *Parse(const std::unique_ptr<tflite::OperatorT> &tflite_op,            // override interface
                         const std::unique_ptr<tflite::SubGraphT> &tflite_subgraph,
                         const std::unique_ptr<tflite::ModelT> &tflite_model) override;
};

REG_NODE_PARSER(kFmkTypeTflite, ADD, std::make_shared<AddParserTutorial>());     // call the registration macro

For the sample code, please refer to node_parser.

Model Extension

For the sample code, please refer to the unit case ModelParserRegistryTest.

Optimization Extension

Self-defined Pass: The users need to inherit the base class PassBase, and override the interface function Execute.
Pass Registration: The users can directly call the registration interface REG_PASS, so that the self-defined pass can be registered in the converter tool of MindSpore Lite.

class PassTutorial : public registry::PassBase {  // inherit the base class
 public:
  PassTutorial() : PassBase("PassTutorial") {}

  ~PassTutorial() = default;

  bool Execute(const api::FuncGraphPtr &func_graph) override;     // override interface

 private:
  AnfNodePtr CreateCustomOp(const api::FuncGraphPtr func_graph, const CNodePtr &cnode);
};

using mindspore::registry::POSITION_BEGIN;            // choose a scheduling position
REG_PASS(PassTutorial, opt::PassTutorial)             // register PassBase's subclass
REG_SCHEDULED_PASS(POSITION_BEGIN, {"PassTutorial"})  // register scheduling logic

For the sample code, please refer to pass.

In the offline phase of conversion, we will infer the basic information of output tensors of each node of the model, including the format, data type and shape. So, in this phase, users need to provide the inferring process of self-defined operator. Here, users can refer to Operator Infershape Extension.

Example

Compile

Environment Requirements
- System environment: Linux x86_64; Recommend Ubuntu 18.04.02LTS
- compilation dependencies:
  - CMake >= 3.18.3
  - GCC >= 7.3.0
Compilation preparation

The release package of MindSpore Lite doesn't provide serialized files of other frameworks, therefore, users need to compile and obtain by yourselves. Here, please refer to Overview.

The case is a tflite model, users need to compile flatbuffers and combine the TFLITE Proto File to generate the serialized file.

After generating, users need to create a directory schema under the directory of mindspore/lite/examples/converter_extend and then place the serialized file in it.
Compilation and Build

Execute the script build.sh in the directory of mindspore/lite/examples/converter_extend. And then, the released package of Mindspore Lite will be downloaded and the demo will be compiled automatically.
```
bash build.sh
```
If the automatic download is failed, users can download the specified package manually, of which the hardware platform is CPU and the system is Ubuntu-x64 mindspore-lite-{version}-linux-x64.tar.gz, After unzipping, please copy the directory of tools/converter/lib and tools/converter/include to the directory of mindspore/lite/examples/converter_extend.

After manually downloading and storing the specified file, users need to execute the build.sh script to complete the compilation and build process.
Compilation Result

The dynamic library libconverter_extend_tutorial.so will be generated in the directory of mindspore/lite/examples/converter_extend/build.

Executing Program

Copy library

Copy the dynamic library libconverter_extend_tutorial.so to the directory of tools/converter/lib of the released package.
Enter the conversion directory of the released package.
```
cd ${PACKAGE_ROOT_PATH}/tools/converter/converter
```

Create extension configuration file(converter.cfg, please refer to Extension Configuration), the content is as follows:

[registry]
plugin_path=libconverter_extend_tutorial.so      # users need to configure the correct path of the dynamic library

Add the required dynamic library to the environment variable LD_LIBRARY_PATH

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${PACKAGE_ROOT_PATH}/tools/converter/lib

Execute the script

./converter_lite --fmk=TFLITE --modelFile=add.tflite --configFile=converter.cfg --outputFile=add_extend

The model file add_extend.ms will be generated, the place of which is up to the parameter outputFile.

Extension Configuration

To load the extension module when converting, users need to configure the path of extended dynamic library. The parameters related to the extension include plugin_path, disable_fusion. The detailed description of the parameters is as follows:

Parameter	Attribute	Function Description	Parameter Type	Default Value	Value Range
plugin_path	Optional	Third-party library path	String	-	If there are more than one, please use `;` to separate.
disable_fusion	Optional	Indicate whether to close fusion	String	off	off or on.
fusion_blacklists	Optional	Specified fusion operator names to be closed	String	-	If there are more than one, please use `,` to separate

We have generated the default configuration file (converter.cfg). The content is as follows:

[registry]
plugin_path=libconverter_extend_tutorial.so      # users need to configure the correct path of the dynamic library

If the user needs to turn off the specified operator fusions, the fusion configuration of the the specified operator names to be closed are as follows:

[registry]
# When parameter `disable_fusion` is configured as `off`, the user can turn off the specified operator fusions by configuring parameter `fusion_blacklists`. While parameter `disable_fusion` is configured as `on`, the parameter `fusion_blacklists` does not work.
disable_fusion=off
fusion_blacklists=ConvActivationFusion,MatMulActivationFusion

The operator fusion names are as follows:

No	the operator fusion name
1	AddConcatActivationFusion
2	SqueezeFusion
3	TransposeFusion
4	ReshapeReshapeFusion
5	ConvBiasaddFusion
6	ConvBatchNormFusion
7	ConvScaleFusion
8	GroupNormFusion
9	TfNormFusion
10	OnnxLayerNormFusion
11	OnnxLayerNormFusion2
12	BatchMatMulFusion
13	BatchNormToScaleFusion
14	SigmoidMulFusion
15	ActivationFusion
16	ConvActivationFusion
17	ConvTupleGetItemFusion
18	ConvTupleActivationFusion
19	TfliteLstmCellFusion
20	TfLstmCellFusion
21	TfBidirectionGruFusion
22	TfGeLUFusion
23	OnnxGeLUFusion
24	TfliteRelPosMultiHeadAttentionFusion
25	GLUFusion
26	ConstFoldPass
27	AffineFusion
28	AffineActivationFusion
29	ConvConvFusion
30	ConvPadFusion
31	MatMulAddFusion
32	MatMulMulFusion
33	TransposeMatMulFusion
34	MulAddFusion
35	ScaleActivationFusion
36	ScaleScaleFusion
37	FullConnectedFusion
38	FullconnectedAddFusion
39	TensorDotFusion
40	MatMulActivationFusion