Ascend Conversion Tool Description
Introduction
This article introduces the related features of the cloud-side inference model conversion tool in Ascend back-end, such as profile options, dynamic shape, AOE, custom operators.
Configuration File
Table 1: Configure [ascend_context] parameter
Parameters |
Attributes |
Functions Description |
Types |
Values Description |
---|---|---|---|---|
|
Optional |
Specify the model input format. |
String |
Options: |
|
Optional |
Specify the model input Shape. input_name must be the input name in the network model before conversion, in the order of the inputs, separated by |
String |
Such as |
|
Optional |
Specify the dynamic BatchSize and dynamic resolution parameters. |
String |
|
|
Optional |
Configure the model accuracy mode. |
String |
Options: |
|
Optional |
Configure the operator selection mode. |
String |
Optioans: |
|
Optional |
Specify the data type of network output |
String |
Options: |
|
Optional |
Configure the Fusion Switch Configuration File file path and file name. |
String |
Specify the configuration file for the fusion switch |
|
Optional |
Model insertion AIPP operator |
String |
Path of AIPP configuration file |
|
Optional |
AOE auto-tuning mode |
String |
Options: "subgraph tuning", "operator tuning" or "subgraph tuning, operator tuning". Default: Not enabled |
|
Optional |
Enable Ascend backend fusion optimization to generate custom operators |
String |
The available options are |
|
Optional |
Specify custom operator structures in the enabling model |
String |
|
|
Optional |
Specify custom operator attributes for fusion |
String |
|
Table 2: Configure [acl_init_options] parameter
Parameters |
Attributes |
Functions Description |
Types |
Values Description |
---|---|---|---|---|
|
Optional |
Set the core type used by the network model. |
String |
Options: |
|
Optional |
Version of the Ascend AI processor. |
String |
Options: |
|
Optional |
Data cache optimization switch. |
String |
Options: |
|
Optional |
Data compression can be performed on Weight to improve performance. |
String |
Options: |
|
Optional |
The path of the configuration file for the node list to be compressed is mainly composed of the conv operator and the fc operator. |
String |
path of config file |
|
Optional |
Select the operator precision mode. |
String |
Options: |
|
Optional |
Memory reuse switch. |
String |
Options: |
|
Optional |
Whether to enable a model to use only one stream. |
String |
Options: |
|
Optional |
Set the number of AI cores used during compilation. |
String |
Default: |
|
Optional |
Fusion configuration file path. |
String |
path of config file |
|
Optional |
Whether to enable the optimization of small channel. |
String |
Options: |
|
Optional |
Select the operator implementation mode. |
String |
Options: |
|
Optional |
A list of operators that uses the mode specified by the |
String |
Operator type |
|
Optional |
Configure operator to compile Disk buffer mode. |
String |
Options: |
|
Optional |
Configure operator mutation Disk buffer directory. |
String |
Options: |
|
Optional |
Configure the path to save debugging related process files generated by operator compilation. |
String |
Default Generate Current Path |
|
Optional |
Operator debug function switch. |
String |
Options: |
|
Optional |
Configure a mixed precision list. |
String |
path of config file |
|
Optional |
Enable global sparsity characteristics. |
String |
Options: |
|
Optional |
Whether to save the weights of constant nodes separately in a file. |
String |
Options: |
|
Optional |
Whether to enable deterministic calculation. |
String |
Options: |
|
Optional |
Support for inconsistency between the compilation environment operating system and the runtime environment. |
String |
Options: |
|
Optional |
Support for inconsistency between the operating system architecture and the runtime environment in the compilation environment. |
String |
Options: |
|
Optional |
Whether the offline model is supported to run on virtual devices generated by the Ascend virtualization instance feature. |
String |
Options: |
|
Optional |
Compression optimization feature configuration file path. |
String |
path of config file |
Table 3: Configure [acl_build_options] parameter
Parameters |
Attributes |
Functions Description |
Types |
Values Description |
---|---|---|---|---|
|
Optional |
Specify the model input format. |
String |
Options: |
|
Optional |
Specify the model input shape. After the model is converted, it can be obtained using the Model.get_model_info ("input_shape") method. This parameter is consistent with the command line input_shape. |
String |
For example: |
|
Optional |
Specify the model shape rang. |
String |
For example: |
|
Optional |
Extension operator mapping configuration file path. |
String |
path of config file |
|
Optional |
Set dynamic batch gear parameters. |
String |
This parameter needs to be used in conjunction with the |
|
Optional |
Set dynamic resolution parameters for input images. |
String |
This parameter needs to be used in conjunction with the |
|
Optional |
Set the gear of dynamic dimensions in ND format. After the model is converted, it can be obtained using the Model.get_model_info ("dynamic_dims") method |
String |
This parameter needs to be used in conjunction with the |
|
Optional |
Enter the configuration file path for the preprocessing operator. |
String |
path of config file |
|
Optional |
Enter the configuration file path for the preprocessing operator. |
String |
Options: |
|
Optional |
Memory reuse switch. |
String |
Options: |
|
Optional |
Network output data type. |
String |
Options: |
|
Optional |
Specify output nodes. |
String |
For example: |
|
Optional |
Specify the input node name with input data type FP16. |
String |
|
|
Optional |
Set Log Level. |
String |
Options: |
|
Optional |
Configure operator to compile disk buffer mode. |
String |
Options: |
|
Optional |
Configure operator mutation disk buffer directory. |
String |
Options: |
|
Optional |
Configure the path to save debugging related process files generated by operator compilation. |
String |
Default Generate Current Path |
|
Optional |
Operator debug function switch. |
String |
Options: |
|
Optional |
Path to custom knowledge base after loading model tuning. |
String |
This parameter needs to be used in conjunction with the |
|
Optional |
Customizing the knowledge base path after loading operator tuning. |
String |
Knowledge Base Path |
|
Optional |
Configure mixed precision list. |
String |
path of config file |
|
Optional |
Set the precision mode of a specific operator and use this parameter to set the configuration file path. |
String |
path of config file |
|
Optional |
Shape compilation method during image compilation. |
String |
Options: |
|
Optional |
Memory detection function switch. |
String |
path of config file |
|
Optional |
Do you want to save the weights of constant nodes separately in a file. |
String |
Options: |
|
Optional |
Set the network model not to use one or some acceleration engines. |
String |
Options: |
Dynamic Shape Configuration
In some inference scenarios, such as detecting a target and then executing the target recognition network, the number of targets is not fixed resulting in a variable input BatchSize for the target recognition network. If each inference is computed at the maximum BatchSize or maximum resolution, it will result in wasted computational resources. Therefore, it needs to support dynamic BatchSize and dynamic resolution scenarios during inference. Lite inference on Ascend supports dynamic BatchSize and dynamic resolution scenarios. The dynamic_dims dynamic parameter in [ascend_context] is configured via configFile in the convert phase, and the model Resize is used during inference, to change the input shape.
Dynamic Batch Size
Parameter Name
dynamic_dims
Functions
Set the dynamic batch profile parameter for scenarios where the number of images processed at a time is not fixed during inference. This parameter needs to be used in conjunction with input_shape, and the position of -1 in input_shape is the dimension where the dynamic batch is located.
Value
Support up to 100 profiles configuration. Each profile is separated by English comma. The value limit of each profile: [1~2048]. For example, the parameters in the configuration file are configured as follows:
[ascend_context] input_shape=input:[-1,64,64,3] dynamic_dims=[1],[2]
"-1" in input_shape means setting dynamic batch, and the profile can take the value of "1,2", that is, support profile 0: [1,64,64,3], profile 1: [2,64,64,3].
If more than one input exists, the profiles corresponding to the different inputs needs to be the same and separated by
;
.[ascend_context] input_shape=input1:[-1,64,64,3];input2:[-1,256,256,3] dynamic_dims=[1],[2];[1],[2]
converter
./converter_lite --fmk=ONNX --modelFile=${model_name}.onnx --configFile=./config.txt --optimize=ascend_oriented --outputFile=${model_name}
Note: When enabling dynamic BatchSize, you do not need to specify the inputShape parameter, and only need to configure the [ascend_context] dynamic batch size through configFile, that is, the configuration content in the previous section.
Inference
Enable dynamic BatchSize. When the model inference is performed, the input shape can only choose the set value of the profile at the time of the converter. If you want to switch to the input shape corresponding to another profile, use the model Resize function.
Precautions
If the user performs inference operations and the number of images processed is not fixed at a time, this parameter can be configured to dynamically allocate the number of images processed at a time. For example, if a user needs to process 2, 4, or 8 images each time to perform inference, it can be configured as 2, 4, and 8. Once the profile is requested, memory will be requested based on the actual profile during model inference.
If the profile value set by the user is too large or the profiles are too many, it may cause model compilation failure, in which case the user is advised to reduce the profiles or turn down the profile value.
If the profile value set by the user is too large or the profiles are too many, when performing inference in the runtime environment, it is recommended that the swapoff -a command be executed to turn off the swap interval as memory, to avoid that the swap space is continued to be called as memory, resulting in an unusually slow running environment due to the lack of memory.
Dynamic Resolution
Parameter Name
dynamic_dims
Function
Set the dynamic resolution parameter of the input image. For scenarios where the width and height of the image are not fixed each time during inference. This parameter needs to be used in conjunction with input_shape, and the position of -1 in input_shape is the dimension where the dynamic resolution is located.
Value
Support up to 100 profiles configuration. Each profile is separated by English comma, such as "[imagesize1_height,imagesize1_width],[imagesize2_height,imagesize2_width]". For example, the parameters in the configuration file are configured as follows:
[ascend_context] input_format=NHWC input_shape=input:[1,-1,-1,3] dynamic_dims=[64,64],[19200,960]
"-1" in input_shape means setting the dynamic resolution, i.e., it supports profile 0: [1,64,64,3] and profile 1: [1,19200,960,3].
converter
./converter_lite --fmk=ONNX --modelFile=${model_name}.onnx --configFile=./config.txt --optimize=ascend_oriented --outputFile=${model_name}
Note: When enabling dynamic BatchSize, you do not need to specify the inputShape parameter, and only need to configure the [ascend_context] dynamic resolution through configFile, that is, the configuration content in the previous section.
Inference
By enabling dynamic resolution, when model inference is performed, the input shape can only select the set profile value at the time of the converter. If you want to switch to the input shape corresponding to another profile, use the model Resize function.
Precautions
If the resolution value set by the user is too large or the profiles are too many, it may cause model compilation failure, in which case the user is advised to reduce the profiles or turn down the profile value.
If the user sets a dynamic resolution, the size of the dataset images used for the actual inference needs to match the specific resolution used.
If the resolution value set by the user is too large or the profiles are too many, when performing inference in the runtime environment, it is recommended that the swapoff -a command be executed to turn off the swap interval as memory, to avoid that the swap space is continued to be called as memory, resulting in an unusually slow running environment due to the lack of memory.
Dynamic dimension
Parameter Name
ge.dynamicDims
Function
Set the gear of the dynamic dimension input in ND format. Applicable to scenarios where any dimension is processed each time reasoning is performed, This parameter needs to be used in conjunction with input_shape, and the position of -1 in input_shape is the dimension where the dynamic dim is located.
Value
Up to 100 configurations are supported, each separated by an English comma. For example, the parameters in the configuration file are configured as follows:
[acl_build_options] input_format="ND" input_shape="input1:1,-1,-1;input2:1,-1" ge.dynamicDims="32,32,24;64,64,36"
The "-1" in the shape indicates the setting of dynamic dimensions, which supports gear 0: input1:1,32,32; input2:1,24, gear 1:1, 64,64; input2:1,36.
converter
./converter_lite --fmk=ONNX --modelFile=${model_name}.onnx --configFile=./config.txt --optimize=ascend_oriented --outputFile=${model_name}
Note: When enabling dynamic dimension,
input_format
must be set toND
.Inference
By enabling dynamic dimension, when model inference is performed, the input shape can only select the set profile value at the time of the converter. If you want to switch to the input shape corresponding to another profile, use the model Resize function.
Precautions
If the resolution value set by the user is too large or the profiles are too many, it may cause model compilation failure, in which case the user is advised to reduce the profiles or turn down the profile value.
If the user sets a dynamic dimension, the dimension of the inputs for the actual inference needs to match the specific resolution used.
If the resolution value set by the user is too large or the profiles are too many, when performing inference in the runtime environment, it is recommended that the swapoff -a command be executed to turn off the swap interval as memory, to avoid that the swap space is continued to be called as memory, resulting in an unusually slow running environment due to the lack of memory.
AOE Auto-tuning
AOE is a computational graph performance auto-tuning tool built specifically for the Davinci platform. Lite enables AOE ability to integrate the AOE offline executable in the converter phase, to perform performance tuning of the graph, generate a knowledge base, and save the offline model. This function supports subgraph tuning and operator tuning. The function supports subgraph tuning and operator tuning. The specific use process is as follows:
AOE Tool Tuning
Configure environment variables
${LOCAL_ASCEND}
is the path where the Ascend package is installedexport LOCAL_ASCEND=/usr/local/Ascend source ${LOCAL_ASCEND}/latest/bin/setenv.bash
Confirm that the AOE executable program can be found and run in the environment:
aoe -h
Specify the knowledge base path
AOE tuning generates an operator knowledge base. The default path:
${HOME}/Ascend/latest/data/aoe/custom/graph(op)/${soc_version}
(Optional) You can also customize the knowledge base path with the
export TUNE_BANK_PATH
environment variable.Clear the cache
In order for the model compilation to get the knowledge base generated by AOE, it is best to delete the compilation cache before AOE is enabled to avoid cache reuse. Taking Atlas inference series environment with user as root for example, delete
/root/atc_data/kernel_cache/Ascend310P3
and/root/atc_data/fuzzy_kernel_cache/Ascend310P3
directories.Specified options of the configuration file
Specify the AOE tuning mode in the
[ascend_context]
configuration file of the conversion tool config. In the following example, the subgraph tuning will be executed first, and then the operator tuning.[ascend_context] aoe_mode="subgraph tuning, operator tuning"
The performance improvements will vary from environment to environment, and the actual latency reduction percentage is not exactly the same as the results shown in the tuning logs.
AOE tuning generates
aoe_workspace
directory in the current directory where the task is executed, which is used to save the models before and after tuning for performance improvement comparison, as well as the process data and result files necessary for tuning. This directory will occupy additional disk space, e.g., 2~10GB for a 500MB raw model, depending on the model size, operator type structure, input shape size and other factors. Therefore, it is recommended to reserve enough disk space, otherwise it may lead to tuning failure.The
aoe_workspace
directory needs to be deleted manually to free up disk space.
AOE API Tuning
For Ascend inference, when the runtime specifies provider
as ge
, multiple models within one device can share weights, and some the weights in the model can be updated, that is, variables. Currently, only AOE API tuning supports variables exists in the model, and the default AOE tool tuning does not support that. The environment variables, setting and use of knowledge base paths, and AOE tuning cache are consistent with AOE tool tuning in the previous section. For details, please refer to AOE tuning.
AOE API tuning needs to be done through converter tool. When optimize=ascend_oriented
, in the configuration file, there is provider=ge
in [ascend_context]
, and there is a valid aoe_mode
in [ascend_context]
or acl_option_cfg_param]
, or there is a valid job_type
in [aoe_global_options]
, AOE API tuning will be performed. AOE API tuning only generates a knowledge base and does not generate an optimized model.
Specify
provider
asge
[ascend_context] provider=ge
AOE options
The options in
[aoe_global_options]
will be passed through to the global options of the AOE API. The options in[aoe_tuning_options]
will be passed through to the tuning options of the AOE API.We will extract the options in sections
[acl_option_cfg_param]
,[ascend_context]
,[ge_session_options]
and[ge_graph_options]
and convert them into AOE options to avoid the need for users to manually convert these options. The extracted options includeinput_format
,input_shape
,dynamic_dims
andprecision_mode
. When the same option exists in multiple configuration sections at the same time, the priority ranges from low to high, with options in[aoe_global_options]
and[aoe_tuning_options]
having the highest priority. It is recommended to use[ge_graph_options]
andaoe_uning_options
.AOE tuning mode
The
aoe_mode
is currently limited tosubgraph tuning
oroperator tuning
. Currently,subgraph tuning, operator tuning
is not supported, which means that subgraph and operator tuning is not supported in the same tuning process. If necessary, subgraph and operator tuning can be performed separately.In
[aoe_global_options]
, when the value ofjob_type
is1
, it means subgraph tuning, and when the value is2
, it means operator tuning.[ascend_context] aoe_mode="operator tuning"
[acl_option_cfg_param] aoe_mode="operator tuning"
[aoe_global_options] job_type=2
Dynamic dimension profiles
Dynamic dimension profiles can be set in
[acl_option_cfg_param]
,[ascend_context]
,[ge_graph_options]
,[aoe_tuning_options]
, with priority ranging from low to high. The following settings are equivalent. Setting the dynamic dimension profiles in[ascend_context]
can refer to Dynamic Shape Configuration. Setting the dynamic dimension profiles in[acl_option_cfg_param]
,[ge_graph_options]
and[aoe_tuning_options]
can refer to dynamic_dims, dynamic_batch_size, dynamic_image_size. Note that the[ge_graph_options]
only supports thege.dynamicDims
and does not support the forms ofdynamic_batch_size
anddynamic_image_size
.input_format
is used to specify the input dimension layout for dynamic profiles. When usingdynamic_image_size
, it is necessary to specifyinput_format
asNCHW
orNHWC
to indicate the location of theH
andW
dimensions.[ascend_context] input_shape=x1:[-1,3,224,224];x2:[-1,3,1024,1024] dynamic_dims=[1],[2],[3],[4];[1],[2],[3],[4]
[acl_option_cfg_param] input_shape=x1:-1,3,224,224;x2:-1,3,1024,1024 dynamic_dims=1,1;2,2;3,3;4,4
[ge_graph_options] ge.inputShape=x1:-1,3,224,224;x2:-1,3,1024,1024 ge.dynamicDims=1,1;2,2;3,3;4,4
[aoe_tuning_options] input_shape=x1:-1,3,224,224;x2:-1,3,1024,1024 dynamic_dims=1,1;2,2;3,3;4,4
Precision mode
Precision mode can be set in
[acl_option_cfg_param]
,[ascend_context]
,[ge_graph_options]
,[aoe_tuning_options]
, with priority ranging from low to high. The following settings are equivalent. Setting the precision mode in[ascend_context]
and[acl_option_cfg_param]
can refer to ascend_context - precision_mode. Setting the precision mode in[ge_graph_options]
and[aoe_tuning_options]
can refer to precision_mode.[ascend_context] precision_mode=preferred_fp32
[acl_option_cfg_param] precision_mode=preferred_fp32
[ge_graph_options] precision_mode=allow_fp32_to_fp16
[aoe_tuning_options] precision_mode=allow_fp32_to_fp16
Deploying Ascend Custom Operators
MindSpore Lite converter supports converting models with MindSpore Lite custom Ascend operators to MindSpore Lite models. Custom operators can be used to optimize model inference performance in special scenarios, such as using custom MatMul to achieve higher matrix multiplication, using the transformer fusion operators provided by MindSpore Lite to improve transformer model performance (to be launched) and using the AKG graph fusion operator to automatically fuse models to improve inference performance.
If MindSpore Lite converts Ascend models with custom operators, user needs to deploy the custom operators to the ACL operator library before calling the converter in order to complete the conversion properly. The following describes the key steps to deploy Ascend custom operators:
Configure environment variables
${ASCEND_OPP_PATH}
is the operator library path of Ascend software CANN package, usually under Ascend software installation path. The default is usually/usr/local/Ascend/latest/opp
.export ASCEND_OPP_PATH=/usr/local/Ascend/latest/opp
Obtain Ascend custom operator package
MindSpore Lite cloud-side inference package will contain Ascend custom operator package directory whose relative directory is
${LITE_PACKAGE_PATH}/tools/custom_kernels/ascend
. After unzip the Mindspore Lite cloud-side inference package, enter the corresponding directory.tar zxf mindspore-lite-{version}-linux-{arch}.tar.gz cd tools/custom_kernels/ascend
Run install.sh script to deploy custom operator
Run the installation script in the operator package directory to deploy the custom operator.
bash install.sh
Check the Ascend library directory to see if the installation is successful
After deploying the custom operator, go to the Ascend operator library directory
/usr/local/Ascend/latest/opp/vendors/
and check whether there are corresponding custom operator files in the directory. At present, we mainly provide the basic operator sample and the AKG graph fusion operator implementation. The specific file structure is as follows:/usr/local/Ascend/latest/opp/vendors/ ├── config.ini # Custom operator vendor configuration file, define the priority between different vendors, which needs to have vendor configuration of mslite └── mslite # Custom operator directory provided by mslite ├── framework # Third-party framework adaptation configuration │ └── tensorflow # tensorflow adaptation configuration, not required │ └── npu_supported_ops.json ├── op_impl # Custom operator implementation directory │ ├── ai_core # Run operator implementation directory in ai_core │ │ └── tbe # tbe operator implementation directory │ │ ├── config # Operator configurations for different chips │ │ │ ├── ascend310 # Operator configuration of Atlas 200/300/500 inference product chip │ │ │ └── aic_ascend310-ops-info.json │ │ │ ├── ascend310p # Operator configuration of Atlas inference series chip │ │ │ └── aic_ascend310p-ops-info.json │ │ │ ├── ascend910 # Operator configuration of Atlas training series chip │ │ │ └── aic_ascend910-ops-info.json │ │ └── mslite_impl # Implementation logic directory of operators │ │ ├── add_dsl.py # add sample logic implementation file based on dsl development │ │ ├── add_tik.py # add sample logic implementation file based on tik development │ │ ├── compiler.py # Operator compilation logic file needed for akg graph │ │ ├── custom.py # akg custom operator implementation file │ │ ├── matmul_tik.py # matmul sample logic implementation file based on tik development │ ├── cpu # aicpu custom operator subdirectory, not required │ │ └── aicpu_kernel │ │ └── impl │ └── vector_core # Run operator implementation directory in vector_core │ └── tbe # tbe operator implementation directory │ └── mslite_impl # Implementation logic directory of operators │ ├── add_dsl.py # add sample logic implementation file based on dsl development │ ├── add_tik.py # add sample logic implementation file based on tik development │ └── matmul_tik.py # matmul sample logic implementation file based on tik development └── op_proto # Operator prototype definition package directory └── libcust_op_proto.so # operator prototype definition so file. akg custom operator is registered by default, and do not need this file