Building Cloud-side MindSpore Lite

View Source On Gitee

This section describes how to quickly compile MindSpore Lite.

Cloud-side MindSpore Lite contains modules:

Modules

Supported Platforms

Description

converter

Linux

Model Converter

runtime(cpp, java)

Linux

Model Inference Framework

benchmark

Linux

Benchmarking Tool

minddata

Linux

Image Processing Library

akg

Linux

Polyhedral-based deep learning operator compiler(Auto Kernel Generator

Environment Requirements

  • System Environment: Linux x86_64 or arm64, Ubuntu 18.04.02LTS recommended

  • C++ compilation dependencies

  • Compilation dependency of the Java API module (optional), which is not compiled if the JAVA_HOME environment variable is not set.

    • Gradle >= 6.6.1

      • Configure environment variables: export GRADLE_HOME=GRADLE path, and export GRADLE_USER_HOME=GRADLE path

      • Add the bin directory to the PATH: export PATH=${GRADLE_HOME}/bin:$PATH

    • Maven >= 3.3.1

      • Configure environment variables: export MAVEN_HOME=MAVEN path

      • Add the bin directory to the PATH: export PATH=${MAVEN_HOME}/bin:$PATH

    • OpenJDK between 1.8 and 1.15

      • Configure environment variables: export JAVA_HOME=JDK path

      • Add the bin directory to the PATH: export PATH=${JAVA_HOME}/bin:$PATH

  • Compilation dependency for the Python API module (optional), which is not compiled if Python3 or NumPy is not installed.

    • Python >= 3.7.0

    • NumPy >= 1.17.0 (If installation with pip fails, please upgrade the pip version first: python -m pip install -U pip)

    • wheel >= 0.32.0 (If installation with pip fails, please upgrade the pip version first: python -m pip install -U pip)

  • Compilation dependency for AKG (optional, compiled by default), which is not compiled if LLVM-12 or Python3 is not installed. To compile the AKG for the Ascend backend, git-lfs must be installed.

Gradle recommends using gradle-6.6.1-complete, and configuring other versions of gradle will use the gradle wrapper mechanism to automatically download gradle-6.6.1-complete.

You can also directly use Docker compiling images that have been configured with the above dependencies.

  • Download images: docker pull swr.cn-south-1.myhuaweicloud.com/mindspore-build/mindspore-lite:ubuntu18.04.2-20210530

  • Create a container: docker run -tid --net=host --name=docker01 swr.cn-south-1.myhuaweicloud.com/mindspore-build/mindspore-lite:ubuntu18.04.2-20210530

  • Enter the container: docker exec -ti -u 0 docker01 bash

Compilation Options

The build.sh script in the MindSpore root directory can be used to compile cloud-side MindSpore Lite.

Instructions for Using the Parameters of build.sh

Parameters

Description of the parameters

Range of values

Default values

-I

Select target architecture

arm64, x86_64

None

-d

Set this parameter to compile the Debug version, otherwise compile the Release version

None

None

-i

Set this parameter for incremental compilation, otherwise for full compilation

None

None

-j[n]

Set the number of threads used at compile time, otherwise the default setting is 8 threads

Integer

8

-K

Set whether to compile AKG during compilation, otherwise the default setting is on

on, off

on

  • If the JAVA_HOME environment variable is configured and Gradle is installed, the JAR package is compiled at the same time.

  • Add the -i parameter for incremental compilation does not take effect when the -I parameter changes, e.g. -I x86_64 becomes -I arm64.

  • Cross-compilation is not supported, i.e. the arm64 version needs to be compiled in the arm environment.

Module Build Compilation Options

The building of modules is controlled through environment variables, and the users can control the build compilation modules by declaring the relevant environment variables. After modifying the compilation options, the -i parameter can not be added for incremental compilation when compiling with the build.sh script in order for the options to take effect.

General module compilation options:

Options

Description of the parameters

Range of values

Default values

MSLITE_GPU_BACKEND

Set GPU backend, only tensorrt is valid at -I x86_64.

tensorrt, off

off in -I x86_64

MSLITE_ENABLE_TOOLS

Whether to compile the accompanying benchmarking tool

on, off

on

MSLITE_ENABLE_TESTCASES

Whether to compile test cases

on, off

off

MSLITE_ENABLE_ACL

Whether to enable Ascend ACL

on, off

off

MSLITE_ENABLE_CLOUD_INFERENCE

Whether to enable cloud-side inference

on, off

off

MSLITE_ENABLE_SSE

Whether to enable SSE instruction set, only valid for -I x86_64

on, off

off

MSLITE_ENABLE_AVX512

Whether to enable AVX512 instruction set, only valid for -I x86_64

on, off

off

  • The cloud-side inference version relies on the model converter, so when MSLITE_ENABLE_CLOUD_INFERENCE is configured to on, it will compile converter at the same time.

  • If the environment only supports the SSE instruction set, the AVX512 instruction set needs to be configured as off.

Compilation Examples

First, you need to download the source code from the MindSpore code repository before compiling.

git clone -b r2.3.q1 https://gitee.com/mindspore/mindspore.git

Environment Preparation

Ascend

  • Verify the installation of the Ascend AI processor package.

    • The Ascend package is available in both commercial and community versions.

      1. Please refer to the Ascend Data Center Solution 23.0.RC3 Installation Guide document for the download link and installation method of the commercial version.

      2. There is no restriction on downloading the Community Edition. Please go to CANN Community Edition, select 7.0.RC1.beta1 version, and get the corresponding firmware and driver installation packages from the Firmware and Driver. For package selection and installation, please refer to the commercial version installation guide document above.

    • The default installation path for the installation package is /usr/local/Ascend. After installation, make sure the current user has permission to access the installation path of the Ascend AI processor companion package. If you don’t have permission, the root user needs to add the current user to the user group where /usr/local/Ascend is located.

    • Install the whl package included with the Ascend AI processor companion software. If you have previously installed the package included with the Ascend AI processor, you need to uninstall the corresponding whl package first by using the following command.

    pip uninstall te topi -y
    

    The default installation path is installed via the following command. If the installation path is not the default path, you need to replace the path in the command with the installation path.

    pip install /usr/local/Ascend/ascend-toolkit/latest/lib64/topi-{version}-py3-none-any.whl
    pip install /usr/local/Ascend/ascend-toolkit/latest/lib64/te-{version}-py3-none-any.whl
    
  • Configure environment variables

    • After installing Ascend package, you need to export Runtime-related environment variables. The /LOCAL_ASCEND=/usr/local/Ascend in the following command indicates the installation path of the package, so you need to change it to the actual installation path of the package.

      # control log level. 0-EBUG, 1-INFO, 2-WARNING, 3-ERROR, 4-CRITICAL, default level is WARNING.
      export GLOG_v=2
      
      # Conda environmental options
      LOCAL_ASCEND=/usr/local/Ascend # the root directory of run package
      
      # lib libraries that the run package depends on
      export LD_LIBRARY_PATH=${LOCAL_ASCEND}/ascend-toolkit/latest/lib64:${LOCAL_ASCEND}/driver/lib64:${LOCAL_ASCEND}/ascend-toolkit/latest/opp/op_impl/built-in/ai_core/tbe/op_tiling:${LD_LIBRARY_PATH}
      
      # Environment variables that must be configured
      export TBE_IMPL_PATH=${LOCAL_ASCEND}/ascend-toolkit/latest/opp/op_impl/built-in/ai_core/tbe            # TBE operator implementation tool path
      export ASCEND_OPP_PATH=${LOCAL_ASCEND}/ascend-toolkit/latest/opp                                       # OPP path
      export PATH=${LOCAL_ASCEND}/ascend-toolkit/latest/compiler/ccec_compiler/bin/:${PATH}                  # TBE operator compilation tool path
      export PYTHONPATH=${TBE_IMPL_PATH}:${PYTHONPATH}                                                       # Python library that TBE implementation depends on
      

GPU

GPU environment compilation. Using TensorRT requires integration with CUDA, TensorRT. The current version is adapted to CUDA 11.1 and TensorRT 8.5.1.

Install the appropriate version of CUDA and set the installed directory to the environment variable ${CUDA_HOME}. The build script will use this environment variable to find CUDA.

Download the corresponding version of the TensorRT archive and set the directory where the archive was unzipped to the environment variable ${TENSORRT_PATH}. The build script will use this environment variable to find TensorRT.

CPU

Use x86_64 or ARM64 environment.

Installing LLVM-optional

The CPU backend of the graph kernel fusion in the converter needs to rely on LLVM-12. Run the following commands to install LLVM to enable CPU backend. If LLVM-12 is not installed, the graph kernel fusion can only support GPU and Ascend backend.

wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
sudo add-apt-repository "deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-12 main"
sudo apt-get update
sudo apt-get install llvm-12-dev -y

Executing Compilation

Three-backend-unification packages need to configure the following environment variables:

export MSLITE_ENABLE_CLOUD_INFERENCE=on
export MSLITE_GPU_BACKEND=tensorrt
export MSLITE_ENABLE_ACL=on
  • If you don’t need Ascend backend, you can configure export MSLITE_ENABLE_ACL=off.

  • If you don’t need GPU backend, you can configure export MSLITE_GPU_BACKEND=off.

Execute the following command in the source root directory to compile different versions of MindSpore Lite.

  • Compile the x86_64 architecture version while set the number of threads.

    bash build.sh -I x86_64 -j32
    
  • Compile the arm64 architecture version while set the number of threads.

    bash build.sh -I arm64 -j32
    
  • Compile the x86_64 architecture version while setting the number of threads, but do not compile AKG.

    bash build.sh -I x86_64 -j32 -K off
    

Finally, the following file will be generated in the output/ directory:

  • mindspore-lite-{version}-{os}-{arch}.tar.gz: contains runtime and companion tools.

  • mindspore-lite-{version}-{python}-{os}-{arch}.whl: contains the Whl package for runtime (Python).

  • version: The version number of the output, which corresponds to the version of the branch code compiled.

  • python: The output Python version, e.g. Python 3.7 for cp37-cp37m.

  • os: The operating system to which the output piece should be deployed

  • arch: The system architecture in which the output pieces should be deployed.

To experience the Python interface, you need to move to the output/ directory and use the following command to install the Whl installer.

pip install mindspore-lite-{version}-{python}-{os}-{arch}.whl

After installation, you can use the following command to check whether the installation is successful: if no error is reported, the installation is successful.

python -c "import mindspore_lite"

After installation, you can use the following command to check if the built-in AKG in MindSpore Lite is installed successfully: if no error is reported, the installation is successful.

python -c "import mindspore_lite.akg"

After successful installation, you can use the pip show mindspore_lite command to see where the Python modules for MindSpore Lite are installed.

Directory Structure

mindspore-lite-{version}-linux-{arch}
├── runtime
│   ├── include
│   ├── lib
│   │   ├── libascend_kernel_plugin.so # Ascend Kernel Plugin Dynamic Library
│   │   ├── libdvpp_utils.so           # DVPP Image Preprocessing Tools Dynamic Library
│   │   ├── libminddata-lite.a         # Image Processing Static Library
│   │   ├── libminddata-lite.so        # Image Processing Dynamic Library
│   │   ├── libmindspore-core.so       # MindSpore Core Dynamic Library
│   │   ├── libmindspore-glog.so.0     # glog Dynamic Library
│   │   ├── libmindspore-lite-jni.so   # jni dynamic library of MindSpore Lite inference framework
│   │   ├── libmindspore-lite.so       # MindSpore Lite Inference Framework Dynamic Library
│   │   ├── libmsplugin-ge-litert.so   # GE LiteRT Plugin Dynamic Library
│   │   └── mindspore-lite-java.jar    # MindSpore Lite Inference framework jar package
│   └── third_party
│       ├── glog
│       ├── libjpeg-turbo
│       └── securec
└── tools
    ├── akg
    |    └── akg-{version}-{python}-{os}-{arch}.whl # AKG Python whl package
    ├── benchmark              # Benchmarking Tools
    │   └── benchmark          # Benchmarking tool executable file
    └── converter              # Model converter
        ├── converter
        │   └── converter_lite # Converter executable file
        ├── include
        └── lib
            ├── libascend_pass_plugin.so
            ├── libmindspore_converter.so
            ├── libmindspore_core.so
            ├── libmindspore_glog.so.0
            ├── libmslite_shared_lib.so
            ├── libmslite_converter_plugin.so
            ├── libopencv_core.so.4.5
            ├── libopencv_imgcodecs.so.4.5
            └── libopencv_imgproc.so.4.5