mindspore.context

The context of mindspore, used to configure the current execution environment, includes the execution mode, execution backend and other feature switches.

class mindspore.context.ParallelMode

Parallel mode options.

There are five kinds of parallel modes, “STAND_ALONE”, “DATA_PARALLEL”, “HYBRID_PARALLEL”, “SEMI_AUTO_PARALLEL” and “AUTO_PARALLEL”. Default: “STAND_ALONE”.

• STAND_ALONE: Only one processor is working.

• DATA_PARALLEL: Distributes the data across different processors.

• HYBRID_PARALLEL: Achieves data parallelism and model parallelism manually.

• SEMI_AUTO_PARALLEL: Achieves data parallelism and model parallelism by setting parallel strategies.

• AUTO_PARALLEL: Achieves parallelism automatically.

MODE_LIST: The list of all supported parallel modes.

mindspore.context.get_auto_parallel_context(attr_key)

Get auto parallel context attribute value according to the key.

Parameters

attr_key (str) – The key of the attribute.

Returns

Returns attribute value according to the key.

Raises

ValueError – If input key is not attribute in auto parallel context.

Examples

>>> from mindspore import context
>>> parallel_mode = context.get_auto_parallel_context("parallel_mode")
>>> dataset_strategy = context.get_auto_parallel_context("dataset_strategy")

mindspore.context.get_context(attr_key)

Get context attribute value according to the input key. If some attributes are not set, they will be automatically obtained.

Parameters

attr_key (str) – The key of the attribute.

Returns

Object, The value of given attribute key.

Raises

ValueError – If input key is not an attribute in context.

Examples

>>> from mindspore import context
>>> context.get_context("device_target")
>>> context.get_context("device_id")

mindspore.context.get_ps_context(attr_key)

Get parameter server training mode context attribute value according to the key.

Parameters

attr_key (str) –

The key of the attribute:

• enable_ps (bool): Whether to enable parameter server training mode.

Returns

Returns attribute value according to the key.

Raises

ValueError – If input key is not attribute in auto parallel context.

Examples

>>> from mindspore import context
>>> context.get_ps_context("enable_ps")

mindspore.context.reset_auto_parallel_context()

Reset auto parallel context attributes to the default values:

• device_num: 1.

• global_rank: 0.

• gradients_mean: False.

• gradient_fp32_sync: True.

• parallel_mode: ‘stand_alone’.

• search_mode: ‘dynamic_programming’.

• auto_parallel_search_mode: ‘dynamic_programming’.

• parameter_broadcast: False.

• strategy_ckpt_load_file: ‘’.

• strategy_ckpt_save_file: ‘’.

• full_batch: False.

• enable_parallel_optimizer: False.

• enable_alltoall: False.

• pipeline_stages: 1.

• fusion_threshold: 64.

mindspore.context.reset_ps_context()

Reset parameter server training mode context attributes to the default values:

• enable_ps: False.

mindspore.context.set_auto_parallel_context(**kwargs)

Set auto parallel context, which is valid only for Ascend and GPU target.

Auto parallel context should be configured before the initialization of your network.

Note

Attribute name is required for setting attributes. If a program has tasks on different parallel modes, before setting a new parallel mode for the next task, interface mindspore.context.reset_auto_parallel_context() should be called to reset the configuration. Setting or changing parallel modes must be called before creating any Initializer, otherwise, it may have RuntimeError when compiling the network.

Some configurations are parallel mode specific, see the below table for details:

Common	AUTO_PARALLEL
device_num	gradient_fp32_sync
global_rank	loss_repeated_mean
gradients_mean	search_mode
parallel_mode	strategy_ckpt_load_file
all_reduce_fusion_config	strategy_ckpt_save_file
enable_parallel_optimizer	dataset_strategy
parallel_optimizer_config	pipeline_stages
enable_alltoall	grad_accumulation_step
	auto_parallel_search_mode
	comm_fusion

Parameters

• device_num (int) – Available device number, the value must be in [1, 4096]. Default: 1.

• global_rank (int) – Global rank id, the value must be in [0, 4095]. Default: 0.

• gradients_mean (bool) – Whether to perform mean operator after allreduce of gradients. “stand_alone” do not support gradients_mean. Default: False.

• gradient_fp32_sync (bool) – Run allreduce of gradients in fp32. “stand_alone”, “data_parallel” and “hybrid_parallel” do not support gradient_fp32_sync. Default: True.

• parallel_mode (str) –

  There are five kinds of parallel modes, “stand_alone”, “data_parallel”, “hybrid_parallel”, “semi_auto_parallel” and “auto_parallel”. Note the pynative mode only supports the “stand_alone” and “data_parallel” mode. Default: “stand_alone”.

  • stand_alone: Only one processor is working.

  • data_parallel: Distributes the data across different processors.

  • hybrid_parallel: Achieves data parallelism and model parallelism manually.

  • semi_auto_parallel: Achieves data and model parallelism by setting parallel strategies.

  • auto_parallel: Achieving parallelism automatically.

• search_mode (str) –

  There are three kinds of shard strategy search modes: “recursive_programming”, “dynamic_programming” and “sharding_propagation”. Default: “dynamic_programming”.

  • recursive_programming: Recursive programming search mode.

  • dynamic_programming: Dynamic programming search mode.

  • sharding_propagation: Propagate shardings from configured ops to non-configured ops.

• auto_parallel_search_mode (str) – This is the old version of ‘search_mode’. Here, remaining this attribute is for forward compatibility, and this attribute will be deleted in a future MindSpore version.

• parameter_broadcast (bool) – Whether to broadcast parameters before training. Before training, in order to have the same network initialization parameter values for all devices, broadcast the parameters on device 0 to other devices. Parameter broadcasting in different parallel modes is different, data_parallel mode, all parameters are broadcast except for the parameter whose attribute layerwise_parallel is True. Hybrid_parallel, semi_auto_parallel and auto_parallel mode, the segmented parameters do not participate in broadcasting. Default: False.

• strategy_ckpt_load_file (str) – The path to load parallel strategy checkpoint. Default: ‘’

• strategy_ckpt_save_file (str) – The path to save parallel strategy checkpoint. Default: ‘’

• full_batch (bool) – If you load whole batch datasets in auto_parallel mode, this parameter should be set as True. Default: False. The interface is not to be recommended currently, it is better using ‘dataset_strategy’ to replace it.

• dataset_strategy (Union[str, tuple]) – Dataset sharding strategy. Default: “data_parallel”. dataset_strategy=”data_parallel” is equal to full_batch=False, dataset_strategy=”full_batch” is equal to full_batch=True. For dataset load into net by model parallel strategy likes ds_stra ((1, 8), (1, 8)), it requires using set_auto_parallel_context(dataset_strategy=ds_stra).

• enable_parallel_optimizer (bool) – This is a developing feature, which shards the weight update computation for data parallel training in the benefit of time and memory saving. Currently, auto and semi auto parallel mode support all optimizers in both Ascend and GPU. Data parallel mode only supports Lamb and AdamWeightDecay in Ascend . Default: False.

• enable_alltoall (bool) – A switch that allows AllToAll operators to be generated during communication. If its value is False, there will be a combination of operators such as AllGather, Split and Concat instead of AllToAll. Default: False.

• all_reduce_fusion_config (list) – Set allreduce fusion strategy by parameters indices. Only support ReduceOp.SUM and HCCL_WORLD_GROUP/NCCL_WORLD_GROUP. No Default, if it is not set, the fusion is closed.

• pipeline_stages (int) – Set the stage information for pipeline parallel. This indicates how the devices are distributed alone in the pipeline. The total devices will be divided into ‘pipeline_stags’ stages. Currently, this could only be used when parallel mode semi_auto_parallel is enabled. Default: 1.

• grad_accumulation_step (int) – Set the accumulation steps of gradients in auto and semi auto parallel mode. This should be a positive int. Default: 1.

• parallel_optimizer_config (dict) –

  A dict contains the keys and values for setting the parallel optimizer configure. The configure provides more detailed behavior control about parallel training when parallel optimizer is enabled. Currently it supports the key gradient_accumulation_shard. The configure will be effective when we use context.set_auto_parallel_context(enable_parallel_optimizer=True). It supports the following keys.

  • gradient_accumulation_shard(bool): If true, the accumulation gradient parameters will be sharded across the data parallel devices. This will introduce additional communication(ReduceScatter) at each step when accumulate the gradients, but saves a lot of device memories, thus can make model be trained with larger batch size. This configure is effective only when the model runs on pipeline training or gradient accumulation with data parallel. Default True.

  • parallel_optimizer_threshold(int): Set the threshold of parallel optimizer. When parallel optimizer is enabled, parameters with size smaller than this threshold will not be sharded across the devices. Parameter size = shape[0] * … * shape[n] * size(dtype). Non-negative. Unit: KB. Default: 64.

• comm_fusion (dict) –

  A dict contains the types and configurations for setting the communication fusion. each communication fusion config has two keys: “mode” and “config”. It supports following communication fusion types and configurations:

  • allreduce: If communication fusion type is allreduce. The mode contains: auto, size and index. In auto mode, AllReduce fusion is configured by gradients size and the default fusion threshold is 64 MB. In ‘size’ mode, AllReduce fusion is configured by gradients size manually, and the fusion threshold must be larger than 0 MB. In index mode, it is same as all_reduce_fusion_config.

  • allgather: If communication fusion type is allgather. The mode contains: auto, size. In auto mode, AllGather fusion is configured by gradients size, and the default fusion threshold is 64 MB. In ‘size’ mode, AllGather fusion is configured by gradients size manually, and the fusion threshold must be larger than 0 MB.

  • reducescatter: If communication fusion type is reducescatter. The mode contains: auto and size. Config is same as allgather.

Raises

ValueError – If input key is not attribute in auto parallel context.

Examples

>>> from mindspore import context
>>> context.set_auto_parallel_context(device_num=8)
>>> context.set_auto_parallel_context(global_rank=0)
>>> context.set_auto_parallel_context(gradients_mean=True)
>>> context.set_auto_parallel_context(gradient_fp32_sync=False)
>>> context.set_auto_parallel_context(parallel_mode="auto_parallel")
>>> context.set_auto_parallel_context(search_mode="dynamic_programming")
>>> context.set_auto_parallel_context(auto_parallel_search_mode="dynamic_programming")
>>> context.set_auto_parallel_context(parameter_broadcast=False)
>>> context.set_auto_parallel_context(strategy_ckpt_load_file="./strategy_stage1.ckpt")
>>> context.set_auto_parallel_context(strategy_ckpt_save_file="./strategy_stage1.ckpt")
>>> context.set_auto_parallel_context(dataset_strategy=((1, 8), (1, 8)))
>>> context.set_auto_parallel_context(enable_parallel_optimizer=False)
>>> context.set_auto_parallel_context(enable_alltoall=False)
>>> context.set_auto_parallel_context(all_reduce_fusion_config=[8, 160])
>>> context.set_auto_parallel_context(pipeline_stages=2)
>>> parallel_config = {"gradient_accumulation_shard": True, "parallel_optimizer_threshold": 24}
>>> context.set_auto_parallel_context(parallel_optimizer_config=parallel_config, enable_parallel_optimizer=True)
>>> config = {"allreduce": {"mode": "size", "config": 32}, "allgather": {"mode": "size", "config": 32}}
>>> context.set_auto_parallel_context(comm_fusion=config)

mindspore.context.set_context(**kwargs)

Set context for running environment.

Context should be configured before running your program. If there is no configuration, it will be automatically set according to the device target by default.

Note

Attribute name is required for setting attributes. The mode is not recommended to be changed after net was initialized because the implementations of some operations are different in graph mode and pynative mode. Default: GRAPH_MODE.

Some configurations are device specific, see the below table for details:

Function Classification	Configuration Parameters	Hardware Platform Support
System Configuration	device_id	CPU/GPU/Ascend
	device_target	CPU/GPU/Ascend
	max_device_memory	GPU/Ascend
	variable_memory_max_size	Ascend
	mempool_block_size	GPU/Ascend
Debug Configuration	save_graphs	CPU/GPU/Ascend
	save_graphs_path	CPU/GPU/Ascend
	enable_dump	Ascend
	save_dump_path	Ascend
	enable_profiling	Ascend
	profiling_options	Ascend
	print_file_path	Ascend
	env_config_path	CPU/GPU/Ascend
	precompile_only	CPU/GPU/Ascend
	reserve_class_name_in_scope	CPU/GPU/Ascend
	pynative_synchronize	GPU/Ascend
Executive Control	mode	CPU/GPU/Ascend
	enable_graph_kernel	Ascend/GPU
	graph_kernel_flags	Ascend/GPU
	enable_reduce_precision	Ascend
	auto_tune_mode	Ascend
	check_bprop	CPU/GPU/Ascend
	max_call_depth	CPU/GPU/Ascend
	enable_sparse	CPU/GPU/Ascend
	grad_for_scalar	CPU/GPU/Ascend
	enable_compile_cache	CPU/GPU/Ascend
	runtime_num_threads	CPU/GPU/Ascend
	compile_cache_path	CPU/GPU/Ascend

Parameters

• device_id (int) – ID of the target device, the value must be in [0, device_num_per_host-1], while device_num_per_host should be no more than 4096. Default: 0.

• device_target (str) – The target device to run, support “Ascend”, “GPU”, and “CPU”. If device target is not set, the version of MindSpore package is used.

• max_device_memory (str) – Set the maximum memory available for devices. The format is “xxGB”. Default: “1024GB”. The actual used memory size is the minimum of the available memory of the device and max_device_memory.

• variable_memory_max_size (str) – This parameter is deprecated, and will be removed in a future version. Please use parameter ‘max_device_memory’ instead.

• mempool_block_size (str) – Set the size of the memory pool block in PyNative mode for devices. The format is “xxGB”. Default: “1GB”. Minimum size is “1G”. The actual used memory block size is the minimum of the available memory of the device and mempool_block_size.

• save_graphs (bool) – Whether to save graphs. Default: False. When the save_graphs attribute is set as True, attribute of save_graphs_path is used to set the intermediate compilation graph storage path. By default, the graphs are saved in the current directory.

• save_graphs_path (str) – Path to save graphs. Default: “.”. If the specified directory does not exist, the system will automatically create the directory. During distributed training, graphs will be saved to the directory of save_graphs_path/rank_${rank_id}/. rank_id is the ID of the current device in the cluster.

• enable_dump (bool) – This parameters is deprecated, and will be deleted in the next version.

• save_dump_path (str) – This parameters is deprecated, and will be deleted in the next version.

• enable_profiling (bool) – This parameters is deprecated, and will be deleted in the next version. Please use mindspore.profiler.Profiler api instead.

• profiling_options (str) – This parameters is deprecated, and will be deleted in the next version. Please use mindspore.profiler.Profiler api instead.

• print_file_path (str) – The path of saving print data. If this parameter is set, print data is saved to a file by default, and print_file_path is not set, the screen will be displayed. If the saved file already exists, the timestamp suffix will be added to the file. Saving data to a file solves the problem of data loss in screen printing when a large amount of data is generated. If it is not set, an error will be reported: prompt to set the upper absolute path.

• env_config_path (str) –

  Config path for DFX. Through context.set_context(env_config_path=”./mindspore_config.json”)

  configure RDR:

  • enable: controls whether the RDR is enabled to collect the key data during training and save key data in the fault scenario. When set to true, the RDR will be turned on. When set to false, the RDR will be turned off.

  • mode: sets the mode of RDR on exporting data. When set to 1, the RDR only exports data in the fault scenario. When set to 2, the RDR exports data in the fault scenario and the normal end scenario. Default is 1.

  • path: sets the path where RDR saves data. The current path must be absolute.

  Memory reuse:

  • mem_Reuse: controls whether the memory reuse function is turned on. When set to True,

  • the memory reuse function is turned on. When set to False, the memory reuse function is turned off.

• precompile_only (bool) – Whether to only precompile the network. Default: False. If set to True, the network will only be compiled, not executed.

• reserve_class_name_in_scope (bool) –

  Whether to save the network class name in the scope. Default: True. Each node has a scope. A scope of a subnode is the name of its parent node. If reserve_class_name_in_scope is set to True, the class name will be saved after keyword ‘net-’ in the scope. For example:

  Default/net-Net1/net-Net2 (reserve_class_name_in_scope=True)

  Default/net/net (reserve_class_name_in_scope=False)

• pynative_synchronize (bool) – Whether to enable synchronous execution of the device in PyNative mode. Default: False. When the value is set to False, the operator is executed asynchronously on the device. When an error occurs in the execution of the operator, the specific error script code location cannot be located, when the value is set to True, the operator is executed synchronously on the device. It will reduce the execution performance of the program. At this time, when an error occurs in the execution of the operator, the location of the error script code can be located according to the call stack of the error.

• mode (int) – Running in GRAPH_MODE(0) or PYNATIVE_MODE(1). Default: GRAPH_MODE(0). GRAPH_MODE or PYNATIVE_MODE can be set by mode attribute and both modes support all backends, default mode is GRAPH_MODE.

• enable_graph_kernel (bool) – Whether to enable graph kernel fusion to optimize network execution performance. Default: False. Indicates whether to enable image-computing convergence to optimize network execution performance. If enable_graph_kernel is set to True, acceleration can be enabled. For details of graph kernel fusion, please check Enabling Graph Kernel Fusion.

• graph_kernel_flags (str) –

  Optimization options of graph kernel fusion, and the priority is higher when it conflicts with enable_graph_kernel. Only for experienced users. For example, context.set_context(graph_kernel_flags=”–opt_level=2 –dump_as_text”). Some general options:

  • opt_level: Set the optimization level. Default: 2. Graph kernel fusion can be enabled equivalently by setting opt_level greater than 0. Available values are:

    • 0: disables graph kernel fusion;

    • 1: enables the basic fusion of operators;

    • 2: includes all optimizations of level 1, and turns on more optimizations such as CSE, arithmetic simplification and so on;

    • 3: includes all optimizations of level 2, and turns on more optimizations such as SitchingFusion, ParallelFusion and so on. Optimizations of this level are radical and unstable in some scenarios. Be caution when using this level.

  • dump_as_text: dumps detail info as text files. Default: false.

  More options can refer to the implementation code.

• enable_reduce_precision (bool) – Whether to enable precision reduction. If the operator does not support the user-specified precision, the precision will be changed automatically. Default: True.

• auto_tune_mode (str) –

  The mode of auto tune when op building, get the best tiling performance. Default: NO_TUNE. The value must be in [‘RL’, ‘GA’, ‘RL,GA’].

  • RL: Reinforcement Learning tune.

  • GA: Genetic Algorithm tune.

  • RL,GA: When both RL and GA optimization are enabled, the tool automatically selects RL or GA based on different types of operators in the network model. The sequence of RL and GA is not differentiated. (Automatic selection).

  For more information about the enable operator tuning tool settings, please check Enable the operator optimization tool.

• check_bprop (bool) – Whether to check back propagation nodes. The checking ensures that the shape and dtype of back propagation node outputs is the same as input parameters. Default: False.

• max_call_depth (int) – Specify the maximum depth of function call. Must be positive integer. Default: 1000. The max_call_depth parameter needs to be set when the nested call is too deep or the number of subgraphs is too large. If max_call_depth is set larger than before, the system max stack depth should be set larger too, otherwise a core dumped exception may be raised because of system stack overflow.

• enable_sparse (bool) – Whether to enable sparsity feature. Default: False. For details of sparsity and sparse tensor, please check sparse tensor.

• grad_for_scalar (bool) – Whether to get gradient for scalar. Default: False. When grad_for_scalar is set to True, the function’s scalar input can be derived. The default value is False. Because the back-end does not support scaling operations currently, this interface only supports simple operations that can be deduced by the front-end.

• enable_compile_cache (bool) – Whether to save or load the cache of the graph compiled by front-end. After enable_compile_cache is set to True, during the first execution, a hardware-independent compilation cache is generated and exported to a MINDIR file. When the network is executed again, if enable_compile_cache is still set to True and the network scripts are not changed, the compile cache is loaded. Note that only limited automatic detection for the changes of python scripts is supported by now, which means that there is a correctness risk. Default: False. This is an experimental prototype that is subject to change and/or deletion.

• compile_cache_path (str) – Path to save the cache of the graph compiled by front-end. Default: “.”. If the specified directory does not exist, the system will automatically create the directory. The cache will be saved to the directory of compile_cache_path/rank_${rank_id}/. The rank_id is the ID of the current device in the cluster.

• runtime_num_threads (int) – The thread pool number of cpu kernel and actor used in runtime, which must bigger than 0. Default value is 30, if you run many processes at the same time, you should set the value smaller to avoid thread contention.

Raises

ValueError – If input key is not an attribute in context.

Examples

>>> from mindspore import context
>>> context.set_context(mode=context.PYNATIVE_MODE)
>>> context.set_context(precompile_only=True)
>>> context.set_context(device_target="Ascend")
>>> context.set_context(device_id=0)
>>> context.set_context(save_graphs=True, save_graphs_path="./model.ms")
>>> context.set_context(enable_reduce_precision=True)
>>> context.set_context(enable_dump=True, save_dump_path=".")
>>> context.set_context(enable_graph_kernel=True)
>>> context.set_context(graph_kernel_flags="--opt_level=2 --dump_as_text")
>>> context.set_context(reserve_class_name_in_scope=True)
>>> context.set_context(variable_memory_max_size="6GB")
>>> context.set_context(enable_profiling=True,
...                     profiling_options='{"output":"/home/data/output","training_trace":"on"}')
>>> context.set_context(check_bprop=True)
>>> context.set_context(max_device_memory="3.5GB")
>>> context.set_context(mempool_block_size="1GB")
>>> context.set_context(print_file_path="print.pb")
>>> context.set_context(enable_sparse=True)
>>> context.set_context(max_call_depth=80)
>>> context.set_context(env_config_path="./env_config.json")
>>> context.set_context(auto_tune_mode="GA,RL")
>>> context.set_context(grad_for_scalar=True)
>>> context.set_context(enable_compile_cache=True, compile_cache_path="./cache.ms")
>>> context.set_context(pynative_synchronize=True)
>>> context.set_context(runtime_num_threads=10)

mindspore.context.set_ps_context(**kwargs)

Set parameter server training mode context.

Note

Some other environment variables should also be set for parameter server training mode. These environment variables are listed below:

MS_SERVER_NUM: Server number

MS_WORKER_NUM: Worker number

MS_SCHED_HOST: Scheduler IP address

MS_SCHED_PORT: Scheduler port

MS_ROLE: The role of this process:

MS_SCHED: represents the scheduler,

MS_WORKER: represents the worker,

MS_PSERVER/MS_SERVER: represents the Server

Parameters

• enable_ps (bool) – Whether to enable parameter server training mode. Only after enable_ps is set True, the environment variables will be effective. Default: False.

• config_file_path (string) – Configuration file path used by recovery, parameter server training mode only supports Server disaster recovery currently. Default: ‘’.

• scheduler_manage_port (int) – Scheduler manage port used to scale out/in. Default: 11202.

• enable_ssl (bool) – Set PS SSL mode enabled or disabled. Default: False.

• client_password (str) – Password to decrypt the secret key stored in the client certificate. Default: ‘’.

• server_password (str) – Password to decrypt the secret key stored in the server certificate. Default: ‘’.

Raises

ValueError – If input key is not the attribute in parameter server training mode context.

Examples

>>> from mindspore import context
>>> context.set_ps_context(enable_ps=True, enable_ssl=True, client_password='123456', server_password='123456')