Semi-automatic Parallel

View Source on Gitee

Semi-automatic parallel supports the automatic mixing of multiple parallel modes, including:

  • Operator-level parallel: Operator-level parallel refers to slicing the input tensor and model parameters into multiple devices for computation on an operator basis to improve overall speed.

  • Higher-order Operator-level Parallelism: Higher-order operator-level parallelism refers to operator-level parallelism that allows customized device layout with tensor layout for more complex sharding logic.

  • Optimizer parallel: Optimizer parallel reduces redundant computations on multiple devices for the same weight updates, spreading the computation over multiple devices.

  • Pipeline parallel: Pipeline parallel means that the model is sliced by layer, with each device processing only a certain part of the model.