IR File Analysis

Overview

When a model compiled using MindSpore runs in the graph mode set_context(mode=GRAPH_MODE) and set_context(save_graphs=2) is set in the configuration, some intermediate files will be generated during graph compliation. These intermediate files are called IR files. Currently, there are two IR files:

.ir file: An IR file that describes the model structure in text format and can be directly viewed using any text editors.
.dot file: When set_context(save_graphs=3) is set in the configuration, an IR file that describes the topology relationships between different nodes. You can use this file by graphviz as the input to generate images for users to view the model structure. For models with multiple operators, it is recommended using the visualization component MindSpore Insight to visualize computing graphs.

Saving IR

set_context(save_graphs=2) is used to save the intermediate code in each compilation phase. The intermediate code can be saved in two formats, and the .ir file with the extension '.ir' is saved by default. If set_context(save_graphs=3) is set, a graphical .ir file with the extension .dot is printed. When the network scale is small, you are advised to use the graphical format that is more intuitive. When the network scale is large, you are advised to use the text format that is more efficient.

You can run the graphviz command to convert a .dot file to the picture format. For example, you can run the dot -Tpng *.dot -o *.png command to convert a .dot file to a .png file.

In the training script train.py, we add the following code to the set_context function, when running the training script, MindSpore will automatically store the IR file generated during compilation to the specified path.

if __name__ == "__main__":
    set_context(save_graphs=3, save_graphs_path="path/to/ir/files")

After the training command is executed, several files were generated under the specified path.

.
├──00_bootstrap_0000.ir
├──00_bootstrap_0001.dot
├──01_type_inference_0002.ir
├──01_type_inference_0003.dot
├──02_graph_reusing_0004.ir
├──02_graph_reusing_0005.dot
├──03_auto_monad_0006.ir
├──03_auto_monad_0007.dot
...

The IR files starting with digits and underscores are generated during the ME graph compilation. The compute graph is saved in each phase of the pipeline. Let's see the important phases.

The bootstrap phase parses the entrance function, this phase initially generates MindIR. If you view the IR file, you can see that there is a foundational resolve node represent the entry function of the graph, and the corresponding call node with parameters.
The type_inference phase performs both type deduction and symbol resolution. It recursively parses the program's entrance functions, resolving references to other functions and objects, and deducing the data type and shape information for all nodes. Errors related to unsupported syntax or unresolved references are flagged during this phase, providing early feedback for developers.
The optimize phase refers hardware-independent optimization is performed. The automatic differential and automatic parallel functions are also performed. This stage can be subdivided into several substages. In the list of IR files, where the files prefixed with opt_pass_ [ordinal] are IR files saved after the end of these sub-stages, non-framework developers do not need to pay too much attention.
The validate phase will verify the compiled compute graph and check the temporary operators which should be removed in the prior phase. If any temporary operator exists, the process will report an error and exit.
The task_emit phase will transfer the compute graph to the backend for further processing.
The execute phase will execute the compute graph. The IR graph in this stage is the final graph in the phase of frontend.

In addition, because the backend is closer to the bottom layer, non-framework developers do not need to pay much attention to other IR files saved during the backend optimization process (such as files that begin with hwopt). Non-framework developers only need to look at the file named graph_build_[Graph Sequence Number]_[IR File Sequence Number].ir, i.e. IR after all front and back end optimizations.

As the IR file number is located at the end of the file, when the files are sorted by file name, the IR files are not sorted by the sequence in which the IR files are generated. To sort IR files according to their generation order, you can utilize the Linux awk command find . -name '*ir' | awk --field-separator="_" '{print $(NF) "--->" $0}' | sort -n.

Multiple files may be saved because the backend is optimized on subgraphs, which is different from the mechanism by which multiple subgraphs on the front end are saved in the same file.

IR File Contents Introduction

The following is an example to describe the contents of the IR file. Run the script:

import mindspore as ms
import mindspore.nn as nn
from mindspore import ops

ms.set_context(mode=ms.GRAPH_MODE)
ms.set_context(save_graphs=2, save_graphs_path="./ir")

class Net(nn.Cell):
    def __init__(self):
        super().__init__()

    def func(x, y):
        return ops.div(x, y)

    def construct(self, x, y):
        a = ops.sub(x, 1)
        b = ops.add(a, y)
        if b :
            b = ops.mul(b, self.func(a, b))
        return b

input1 = ms.Tensor(3, ms.float32)
input2 = ms.Tensor(2, ms.float32)
net = Net()
out = net(input1, input2)
print(out)

ir Introduction

Use a text editing software (for example, vi) to open the 18_execute_0161.ir file output after execution. The file contents are as follows (Here is MindSpore 2.3, and the content may have some imperceptible changes with the version upgrade):

# IR entry: @19_1___main___Net_construct_304
# Total subgraphs: 3
  3
# Attrs:
has_shard: 0
has_attached: 1
jit_level:
check_set_strategy_valid_once_only: 1
FLASH_SP_RUN_ONCE_ONLY: 1
pynative_run_in_graph: 0
less_bn: 0
auto_parallel_finish_pre_action: 1
 13
# Total params: 2
# Params:
%para1_x: <Tensor[Float32], ()> : []
%para2_y: <Tensor[Float32], ()> : []
 18
Node counting information:
Total number of nodes: 29
Total number of cnodes: 12
 22
subgraph attr:
has_shard: 0
has_attached: 1
jit_level:
check_set_strategy_valid_once_only: 1
FLASH_SP_RUN_ONCE_ONLY: 1
pynative_run_in_graph: 0
less_bn: 0
auto_parallel_finish_pre_action: 1
subgraph instance: 19_1___main___Net_construct_304 : 0x135400418
# In file t6.py:15~20, 4~16/    def construct(self, x, y):/
subgraph @19_1___main___Net_construct_304() {
 %0(CNode_310$a) = PrimFunc_Sub(%para1_x, Tensor(shape=[], dtype=Float32, value=1)) cnode_attrs: {checkpoint: Bool(1), is_dynamic_len: Bool(0)}
     : (<Tensor[Float32], ()>, <Tensor[Float32], (), value=...>) -> (<Tensor[Float32], ()>)
     # Fullname with scope: (Default/Sub-op1)
     # In file t6.py:15~20, 4~16/    def construct(self, x, y):/
     # In file t6.py:16, 12~25/        a = ops.sub(x, 1)/
     # In file t6.py:16, 12~19/        a = ops.sub(x, 1)/<~~This line of code can be shared by multiple nodes, and may be duplicated./
     # In file /workspace/mindspore/build/package/mindspore/ops/auto_generate/gen_ops_def.py:5251~5294, 0~31/def sub(input, other):/
     # In file /workspace/mindspore/build/package/mindspore/ops/auto_generate/gen_ops_def.py:5294, 11~31/    return sub_op(input, other)/
 %1(CNode_309$b) = PrimFunc_Add(%0, %para2_y) cnode_attrs: {checkpoint: Bool(1), is_dynamic_len: Bool(0)}
     : (<Tensor[Float32], ()>, <Tensor[Float32], ()>) -> (<Tensor[Float32], ()>)
     # Fullname with scope: (Default/Add-op1)
     # In file t6.py:15~20, 4~16/    def construct(self, x, y):/
     # In file t6.py:17, 12~25/        b = ops.add(a, y)/
     # In file t6.py:17, 12~19/        b = ops.add(a, y)/<~~This line of code can be shared by multiple nodes, and may be duplicated./
     # In file /workspace/mindspore/build/package/mindspore/ops/auto_generate/gen_ops_def.py:183~241, 0~31/def add(input, other):/
     # In file /workspace/mindspore/build/package/mindspore/ops/auto_generate/gen_ops_def.py:241, 11~31/    return add_op(input, other)/
 %2(CNode_308) = PrimFunc_Cast(%1, I64(30)) primitive_attrs: {output_names: [output], input_names: [x, dst_type]} cnode_attrs: {checkpoint: Bool(1), is_dynamic_len: Bool(0)}
     : (<Tensor[Float32], ()>, <Int64, NoShape>) -> (<Tensor[Bool], ()>)
     # Fullname with scope: (Default/Cast-op1)
     # In file /workspace/mindspore/build/package/mindspore/_extends/parse/standard_method.py:2747~2749, 0~23/def bool_(x):/
     # In file /workspace/mindspore/build/package/mindspore/_extends/parse/standard_method.py:2749, 11~23/    return x.__bool__()/
     # In file /workspace/mindspore/build/package/mindspore/_extends/parse/standard_method.py:2749, 11~21/    return x.__bool__()/<~~This line of code can be shared by multiple nodes, and may be duplicated./
     # In file /workspace/mindspore/build/package/mindspore/_extends/parse/standard_method.py:3267~3272, 0~34/def tensor_bool(x):/
     # In file /workspace/mindspore/build/package/mindspore/_extends/parse/standard_method.py:3270~3271, 4~38/    if is_cond and F.isconstant(x):/
     # In file /workspace/mindspore/build/package/mindspore/_extends/parse/standard_method.py:3272, 11~34/    return F.cast(x, mstype.bool_)/<~~This line of code can be shared by multiple nodes, and may be duplicated./
 %3(CNode_317) = Partial(@20_4_✓__main___Net_construct_311, %1, %0) primitive_attrs: {side_effect_propagate: I64(1)} cnode_attrs: {checkpoint: Bool(1)}
     : (<Func, NoShape>, <Tensor[Float32], ()>, <Tensor[Float32], ()>) -> (<Func, NoShape>)
     # Fullname with scope: (Default/Partial-op0)
 %4(CNode_316) = Partial(@21_14_✗__main___Net_construct_314, %1) primitive_attrs: {side_effect_propagate: I64(1)} cnode_attrs: {checkpoint: Bool(1)}
     : (<Func, NoShape>, <Tensor[Float32], ()>) -> (<Func, NoShape>)
     # Fullname with scope: (Default/Partial-op1)
 %5(ValueNode_307) = Switch(%2, %3, %4) cnode_attrs: {checkpoint: Bool(1)}
     : (<Tensor[Bool], ()>, <Func, NoShape>, <Func, NoShape>) -> (<Func, NoShape>)
     # Fullname with scope: (Default/Switch-op4)
     # In file t6.py:15~20, 4~16/    def construct(self, x, y):/
     # In file t6.py:18~19, 8~43/        if b :/
 %6(CNode_306) = %5[@FuncUnion(@20_4_✓__main___Net_construct_311, @21_14_✗__main___Net_construct_314)]()
     : () -> (<Tensor[Float32], ()>)
     # Fullname with scope: (5)
     # In file t6.py:15~20, 4~16/    def construct(self, x, y):/
     # In file t6.py:18~19, 8~43/        if b :/
 Return(%6) cnode_attrs: {checkpoint: Bool(1)}
     : (<Tensor[Float32], ()>)
     # Fullname with scope: (Default/Return-op19)
     # In file t6.py:15~20, 4~16/    def construct(self, x, y):/
     # In file t6.py:18~19, 8~43/        if b :/
}
 82
 83
indirect: 1
subgraph attr:
defer_inline: 0
undeterminate: 0
subgraph instance: 20_4_✓__main___Net_construct_311 : 0x135400a18
# Parameters: 2, (<Tensor[Float32], ()>, <Tensor[Float32], ()>)
# In file t6.py:15~20, 4~16/    def construct(self, x, y):/
subgraph @20_4_✓__main___Net_construct_311(%para3_Parameter_320, %para4_Parameter_319) {
 %0(output) = PrimFunc_Div(%para4_Parameter_319, %para3_Parameter_320)
     : (<Tensor[Float32], ()>, <Tensor[Float32], ()>) -> (<Tensor[Float32], ()>)
     # Fullname with scope: (Default/Div-op1)
     # In file t6.py:15~20, 4~16/    def construct(self, x, y):/
     # In file t6.py:19, 27~42/            b = ops.mul(b, self.func(a, b))/
     # In file t6.py:19, 27~36/            b = ops.mul(b, self.func(a, b))/<~~This line of code can be shared by multiple nodes, and may be duplicated./
     # In file t6.py:12~13, 4~28/    def func(x, y):/
     # In file t6.py:13, 15~28/        return ops.div(x, y)/
     # In file t6.py:13, 15~22/        return ops.div(x, y)/<~~This line of code can be shared by multiple nodes, and may be duplicated./
     # In file /workspace/mindspore/build/package/mindspore/ops/function/math_func.py:707~766, 0~17/def div(input, other, *, rounding_mode=None):/
     # In file /workspace/mindspore/build/package/mindspore/ops/function/math_func.py:762~765, 4~38/    if rounding_mode:/
     # In file /workspace/mindspore/build/package/mindspore/ops/function/math_func.py:765, 17~38/        output = P.Div()(input, other)/<~~This line of code can be shared by multiple nodes, and may be duplicated./
 %1(CNode_313$b) = PrimFunc_Mul(%para3_Parameter_320, %0) cnode_attrs: {is_dynamic_len: Bool(0)}
     : (<Tensor[Float32], ()>, <Tensor[Float32], ()>) -> (<Tensor[Float32], ()>)
     # Fullname with scope: (Default/Mul-op1)
     # In file t6.py:15~20, 4~16/    def construct(self, x, y):/
     # In file t6.py:19, 16~43/            b = ops.mul(b, self.func(a, b))/
     # In file t6.py:19, 16~23/            b = ops.mul(b, self.func(a, b))/<~~This line of code can be shared by multiple nodes, and may be duplicated./
     # In file /workspace/mindspore/build/package/mindspore/ops/auto_generate/gen_ops_def.py:3471~3518, 0~31/def mul(input, other):/
     # In file /workspace/mindspore/build/package/mindspore/ops/auto_generate/gen_ops_def.py:3518, 11~31/    return mul_op(input, other)/
 Return(%1)
     : (<Tensor[Float32], ()>)
     # Fullname with scope: (Default/Return-op20)
     # In file t6.py:15~20, 4~16/    def construct(self, x, y):/
     # In file t6.py:19, 12~43/            b = ops.mul(b, self.func(a, b))/
}
118
119
indirect: 1
subgraph attr:
defer_inline: 0
undeterminate: 0
subgraph instance: 21_14_✗__main___Net_construct_314 : 0x1353ff218
# Parameters: 1, (<Tensor[Float32], ()>)
# In file t6.py:15~20, 4~16/    def construct(self, x, y):/
subgraph @21_14_✗__main___Net_construct_314(%para5_Parameter_322) {
 Return(%para5_Parameter_322)
     : (<Tensor[Float32], ()>)
     # Fullname with scope: (Default/Return-op21)
     # In file t6.py:15~20, 4~16/    def construct(self, x, y):/
     # In file t6.py:18~19, 8~43/        if b :/
}

The above contents can be divided into two parts. The first part is the input list and the second part is the graph structure:

Line 1 tells us @19_1___main___Net_construct_304, the name of the top MindSpore graph about the network, which is the entry graph.
Line 2 tells us the number of subgraph parsed by the network. There are 3 graphs in this IR. Line 23 is the entry graph @19_1___main___Net_construct_304. Line 84 is graph 20_4_✓__main___Net_construct_311, parsed from the block when the condition of the if statement in the network is true. Line 120 is graph 21_14_✗__main___Net_construct_314, parsed from the block when the condition of the if statement in the network is false.
Line 14 tells us how many inputs are in the network.
Line 16 to 17 are the input list, which is in the format of %para[No.]_[name] : <[data_type], (shape)>.

Taking graph @19_1___main___Net_construct_304 as an example:

Line 23 to 81 indicate the graph structure, which contains several nodes, namely, CNode. In this example, there are Sub, Add, Mul defined in the function __init__.

The CNode (check the design of ANF-IR) information format is as follows: from left to right, the ordinal number, node name - debug_name, operator name - op_name, input node - arg, attributes of the node - primitive_attrs, input and output specifications, source code parsing call stack and other information. Because the ANF graph is a unidirectional acyclic graph, the connection between nodes is displayed only based on the input relationship. The corresponding source code reflects the relationship between the CNode and the script source code. For example, line 75 is parsed from if b.

%[No.]([debug_name]) = [op_name]([arg], ...) primitive_attrs: {[key]: [value], ...}
    : (<[input data_type]x[input shape]>, ...) -> (<[output data_type]x[output shape]>, ...)
    # Corresponding source code

About the corresponding source code:

The source code information includes the file path, start position, and end position. For example, # In file /workspace/mindspore/build/package/mindspore/nn/wrap/cell_wrapper.py:437~441, 8~45 indicates the file path /workspace/mindspore/build/package/mindspore/nn/wrap/cell_wrapper.py, the code starts from row 437 and column 8, and ends at row 441 and column 45. If the code does not span lines, the end row information is not displayed. For example, # In file /workspace/mindspore/build/package/mindspore/nn/wrap/cell_wrapper.py:418, 19~37.
There are two mode for the corresponding source code displaying. The first mode is to display the complete call stack, and the second mode only displays one code line for reducing the size of the IR file, which eliminates the call stack. The first mode is used by default. The code lines of the complete call stacks are displayed in all ir files.
If the operator is a back propagation operator, the associated code line will not only display its own code, but also the corresponding forward code, identified by "Corresponding forward node candidate:".
If the operator is a fusion operator, the associated code line will display the fusion related code, identified by "Corresponding code candidate:", where the separator "-" is used to distinguish different codes.

After several optimizations by the compiler, the node may undergo several changes (such as operator splitting and operator merging). The source code parsing call stack information of the node may not be in a one-to-one correspondence with the script. This is only an auxiliary method.

After the kernel select phase at the backend, two lines of input and output specification information (that is, the content after :) will appear. The first line represents the specifications on the HOST side, and the second line represents the specifications on the DEVICE side.

dot Introduction

We can use this file by graphviz as the input to generate images for users to view the model structure. For example, under the Linux operating system, we can convert a PNG image by the following command.

dot -Tpng -o 01_type_inference_0003.png 01_type_inference_0003.dot

After the conversion, we obtain a model diagram similar to the one below, which allows us to observe the structure of the constructed static graph model. The different black boxes distinguish different subgraphs, and the blue arrows between graphs represent calling another graph. The blue area represents the parameter, the rectangle represents the parameter list of the graph, the hexagon and the black arrow represent the parameter as the input of the CNode to participate in the calculation process. The yellow rectangle represents the CNode. As can be seen from the picture, the CNode input starts from index 0, and the 0th input (that is, the purple or green area) represents what calculation the operator will perform, which is connected by a dotted arrow. The type is usually an operator primitive, or it can also be another graph. The rest inputs are the parameters required for the calculation.

For models with multiple operators, the picture will be very large. It is recommended by using the visualization component MindSpore Insight to visualize compute graphs.

How to derive the cause of the failure based on the analyze_fail.ir file analysis graph

In the graph compilation process, MindSpore often reports a graph derivation failure in the type_inference phase. But we can find the reason by analyzing the exception information and analyze_fail.ir.

Example 1: parameters number mismatch

import mindspore as ms
import mindspore.nn as nn
from mindspore import ops
  4
ms.set_context(mode=ms.GRAPH_MODE)
ms.set_context(save_graphs=2, save_graphs_path="./ir")
  7
class Net(nn.Cell):
   def __init__(self):
       super().__init__()
 11
   def func(x, y):
       return ops.div(x, y)
 14
   def construct(self, x, y):
       a = ops.sub(x, 1)
       b = ops.add(a, y)
       c = ops.mul(b, self.func(a, a, b))
 19
input1 = ms.Tensor(3, ms.float32)
input2 = ms.Tensor(2, ms.float32)
net = Net()
out = net(input1, input2)
print(out)

An error happens.

Traceback (most recent call last):
 File "t2.py", line 23, in <module>
   out = net(input1, input2)
 File "/workspace/mindspore/build/package/mindspore/nn/cell.py", line 701, in __call__
   out = self.compile_and_run(*args, **kwargs)
 File "/workspace/mindspore/build/package/mindspore/nn/cell.py", line 1051, in compile_and_run
   self.compile(*args, **kwargs)
 File "/workspace/mindspore/build/package/mindspore/nn/cell.py", line 1034, in compile
   _cell_graph_executor.compile(self, *self._compile_args, phase=self.phase,
 File "/workspace/mindspore/build/package/mindspore/common/api.py", line 1815, in compile
   result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
TypeError: The parameters number of the function is 2, but the number of provided arguments is 3.
FunctionGraph ID : func_40
NodeInfo: In file t2.py:12~13, 4~28
   def func(x, y):
 16
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/ps/static_analysis/stack_frame.cc:104 DoJump
 21
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
# 0 In file t2.py:18, 23~41
       c = ops.mul(b, self.func(a, a, b))
                      ^~~~~~~~~~~~~~~~~~
(See file '/workspace/mindspore/rank_0/om/analyze_fail.ir' for more details. Get instructions about `analyze_fail.ir` at https://www.mindspore.cn/search?inputValue=analyze_fail.ir)

Above exception is "TypeError: The parameters number of the function is 2, but the number of provided arguments is 3…". And it tells us FunctionGraph ID : func_40 only needs two parameters, but actually gives 3. We can find the related code is self.func(a, a, b) from 'The function call stack … In file t2.py:18'. Easily, by checking the code, we know that we gave too much parameter to the calling function.

Sometimes when the exception information is not enough easy to understand, or we want to see the part of graph information that have evaluated, we use text editing software (e.g., vi) to open the file (in parentheses on line 28) that prompts in the error message: /workspace/mindspore/rank_0/om/analyze_fail.ir with the following additional content (Here is MindSpore 2.3, and the content may have some imperceptible changes with the version upgrade):

# ===============================================================================================
# The following shows the IR when the function graphs evaluation fails to help locate the problem.
# You can search the last ------------------------> to the node which is evaluated failure.
# Refer to https://www.mindspore.cn/search?inputValue=analyze_fail.ir to get more instructions.
# ===============================================================================================
  6
# IR entry: @__main___Net_construct_11
# Total subgraphs: 0
  9
# Total params: 2
# Params:
%para1_x: <null>
%para2_y: <null>
 14
subgraph attr:
subgraph instance: __main___Net_construct_11 : 0x12fb85c18
# In file t2.py:15~18, 4~42/    def construct(self, x, y):/
subgraph @__main___Net_construct_11() {
 %0(CNode_3) = resolve(NameSpace[Entry: '__main__.Net.construct'], __main__.Net.construct)
     : (<External, NoShape>, <External, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
 22
#------------------------> 0
 %1(CNode_2) = %0(%para1_x, %para2_y)
     : (<Tensor[Float32], ()>, <Tensor[Float32], ()>) -> (<null>)
     #scope: (Default)
 Return(%1)
     : (<null>)
     #scope: (Default)
     # In file t2.py:15~18, 4~42/    def construct(self, x, y):/
}
# Order:
#   1: @__main___Net_construct_11:CNode_3{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> Entry: '__main__.Net.construct', [2]: ValueNode<Symbol> __main__.Net.construct}
#   2: @__main___Net_construct_11:CNode_2{[0]: CNode_3, [1]: param_x, [2]: param_y}
#   3: @__main___Net_construct_11:CNode_44{[0]: ValueNode<Primitive> Return, [1]: CNode_2}
 36
 37
subgraph attr:
subgraph instance: __main___Net_construct_11 : 0x12fac0218
# In file t2.py:15~18, 4~42/    def construct(self, x, y):/
subgraph @__main___Net_construct_11(%para0_x, %para0_y) {
 %0(CNode_12) = resolve(NameSpace[SymbolStr: 'Namespace:__main__'], ops)
     : (<External, NoShape>, <External, NoShape>) -> (<External, NoShape>)
     #scope: (Default)
     # In file t2.py:16, 12~15/        a = ops.sub(x, 1)/
 %1(CNode_17) = getattr(%0, "mul")
     : (<External, NoShape>, <String, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file t2.py:18, 12~19/        c = ops.mul(b, self.func(a, a, b))/
 %2(CNode_15) = getattr(%0, "add")
     : (<External, NoShape>, <String, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file t2.py:17, 12~19/        b = ops.add(a, y)/
 %3(CNode_13) = getattr(%0, "sub")
     : (<External, NoShape>, <String, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file t2.py:16, 12~19/        a = ops.sub(x, 1)/
 %4(a) = %3(%para0_x, I64(1))
     : (<Tensor[Float32], ()>, <Int64, NoShape>) -> (<Tensor[Float32], ()>)
     #scope: (Default)
     # In file t2.py:16, 12~25/        a = ops.sub(x, 1)/
 %5(b) = %2(%4, %para0_y)
     : (<Tensor[Float32], ()>, <Tensor[Float32], ()>) -> (<Tensor[Float32], ()>)
     #scope: (Default)
     # In file t2.py:17, 12~25/        b = ops.add(a, y)/
 %6(CNode_18) = resolve(NameSpace[ClassMember: 'Namespace:__main__..<Net::5265583376>'], func)
     : (<External, NoShape>, <External, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file t2.py:18, 23~32/        c = ops.mul(b, self.func(a, a, b))/
 70
#------------------------> 1
 %7(CNode_19) = %6(%4, %4, %5)
     : (<Tensor[Float32], ()>, <Tensor[Float32], ()>, <Tensor[Float32], ()>) -> (<null>)
     #scope: (Default)
     # In file t2.py:18, 23~41/        c = ops.mul(b, self.func(a, a, b))/
 %8(c) = %1(%5, %7)
     : (<Tensor[Float32], ()>, <null>) -> (<null>)
     #scope: (Default)
     # In file t2.py:18, 12~42/        c = ops.mul(b, self.func(a, a, b))/
 %9(CNode_22) = StopGradient(%8)
     : (<null>) -> (<null>)
     #scope: (Default)
     # In file t2.py:15~18, 4~42/    def construct(self, x, y):/
 %10(CNode_21) = Depend(None, %9) primitive_attrs: {side_effect_propagate: I64(1)} cnode_attrs: {topo_sort_rhs_first: Bool(1)}
     : (<null>, <null>) -> (<null>)
     #scope: (Default)
     # In file t2.py:15~18, 4~42/    def construct(self, x, y):/
 Return(%10)
     : (<null>)
     #scope: (Default)
     # In file t2.py:15~18, 4~42/    def construct(self, x, y):/
}
# Order:
#   1: @__main___Net_construct_11:CNode_12{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> SymbolStr: 'Namespace:__main__', [2]: ValueNode<Symbol> ops}
#   2: @__main___Net_construct_11:CNode_13{[0]: ValueNode<Primitive> getattr, [1]: CNode_12, [2]: ValueNode<StringImm> sub}
#   3: @__main___Net_construct_11:CNode_45{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> CommonOPS: 'Namespace:mindspore._extends.parse.trope', [2]: ValueNode<Symbol> MakeTuple}
#   5: @__main___Net_construct_11:a{[0]: CNode_13, [1]: param_x, [2]: ValueNode<Int64Imm> 1}
#   6: @__main___Net_construct_11:CNode_15{[0]: ValueNode<Primitive> getattr, [1]: CNode_12, [2]: ValueNode<StringImm> add}
#   7: @__main___Net_construct_11:CNode_46{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> CommonOPS: 'Namespace:mindspore._extends.parse.trope', [2]: ValueNode<Symbol> MakeTuple}
#   9: @__main___Net_construct_11:b{[0]: CNode_15, [1]: a, [2]: param_y}
#  10: @__main___Net_construct_11:CNode_17{[0]: ValueNode<Primitive> getattr, [1]: CNode_12, [2]: ValueNode<StringImm> mul}
#  11: @__main___Net_construct_11:CNode_18{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> ClassMember: 'Namespace:__main__..<Net::5265583376>', [2]: ValueNode<Symbol> func}
#  12: @__main___Net_construct_11:CNode_47{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> CommonOPS: 'Namespace:mindspore._extends.parse.trope', [2]: ValueNode<Symbol> MakeTuple}
#  14: @__main___Net_construct_11:CNode_19{[0]: CNode_18, [1]: a, [2]: a, [3]: b}
#  15: @__main___Net_construct_11:CNode_48{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> CommonOPS: 'Namespace:mindspore._extends.parse.trope', [2]: ValueNode<Symbol> MakeTuple}
#  17: @__main___Net_construct_11:c{[0]: CNode_17, [1]: b, [2]: CNode_19}
#  19: @__main___Net_construct_11:CNode_44{[0]: ValueNode<Primitive> Return, [1]: CNode_21}
108
109
# ===============================================================================================
# The total of function graphs in evaluation stack: 2
# ===============================================================================================
113
114
# ===============================================================================================
# The rest function graphs are the following:
# ===============================================================================================
No more function graphs.

The file analyze_fail.ir has the same information format with ir file. The only difference is analyze_fail.ir will locate the node which inferring failed. Searching the point by the text of ------------------------>, we reach ------------------------> 1 at line 71. This points to the node that derives the error, which is %7(CNode_19) = %6(%4, %4, %5) ..... We can know the node have 3 parameters from (%4, %4, %5). From the source parsing call stack, it can be known that the function is actually self.func, which is defined in the script as def func(x, y):.... In the function definition, only two parameters are needed, so there will be a deduction failure error, and we need to modify the number of parameters passed in the script to solve the problem.

Example 2: BiasAdd inputs shape mismatch

import numpy as np
import mindspore as ms
from mindspore import nn, ops, set_context, Tensor, Parameter
from mindspore.common.initializer import initializer
  5
ms.set_context(mode=ms.GRAPH_MODE)
  7
class Net(nn.Cell):
   def __init__(self):
       super(Net, self).__init__()
       self.weight = Parameter(initializer('normal', [32, 8]), name="weight")
       self.bias = Parameter(initializer('zeros', [4]), name="bias")
 13
   def construct(self, x1):
       x = ops.matmul(x1, self.weight)
       x = ops.bias_add(x, self.bias)
       return x
 18
net = Net()
x = Tensor(np.arange(3*32).reshape(3, 32), ms.float32)
out = net(x)
print('out', out.shape)

An error happens.

Traceback (most recent call last):
 File "t2.py", line 21, in <module>
   out = net(x)
 File "/workspace/mindspore/build/package/mindspore/nn/cell.py", line 701, in __call__
   out = self.compile_and_run(*args, **kwargs)
 File "/workspace/mindspore/build/package/mindspore/nn/cell.py", line 1051, in compile_and_run
   self.compile(*args, **kwargs)
 File "/workspace/mindspore/build/package/mindspore/nn/cell.py", line 1034, in compile
   _cell_graph_executor.compile(self, *self._compile_args, phase=self.phase,
 File "/workspace/mindspore/build/package/mindspore/common/api.py", line 1815, in compile
   result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
ValueError: For 'BiasAdd', bias[0] shape should be equal to input_x[1] shape when data_format is 0, but got bias shape: .[const vector]{4}, input_shape: [const vector]{3, 8}.
 13
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/core/ops/ops_func_impl/bias_add.cc:71 CheckShapeValid
 18
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
# 0 In file t2.py:16, 11~37
      x = ops.bias_add(x, self.bias)
          ^~~~~~~~~~~~~~~~~~~~~~~~~~
# 1 In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6280, 11~37
   return bias_add_op(input_x, bias)
          ^~~~~~~~~~~~~~~~~~~~~~~~~~
(See file '/workspace/mindspore/rank_0/om/analyze_fail.ir' for more details. Get instructions about `analyze_fail.ir` at https://www.mindspore.cn/search?inputValue=analyze_fail.ir)

The above reports that the errors is caused by the mismatching of the shape of the first input and the second input of the operator BiasAdd. To further understand what changes have taken place in the shape of the operator, we use text editing software (e.g., vi) to open the file that prompts in the error message: /workspace/mindspore/rank_0/om/analyze_fail.ir with the following additional content (Here is MindSpore 2.3, and the content may have some imperceptible changes with the version upgrade):

# ===============================================================================================
# The following shows the IR when the function graphs evaluation fails to help locate the problem.
# You can search the last ------------------------> to the node which is evaluated failure.
# Refer to https://www.mindspore.cn/search?inputValue=analyze_fail.ir to get more instructions.
# ===============================================================================================
  6
# IR entry: @__main___Net_construct_6
# Total subgraphs: 0
  9
# Total params: 3
# Params:
%para1_x1: <null>
%para2_weight: <Ref[Tensor[Float32]], (32, 8), ref_key=weight>  :  has_default
%para3_bias: <Ref[Tensor[Float32]], (4), ref_key=bias>  :  has_default
 15
subgraph attr:
subgraph instance: __main___Net_construct_6 : 0x128910818
# In file t2.py:14~17, 4~15/    def construct(self, x1):/
subgraph @__main___Net_construct_6() {
 %0(CNode_3) = resolve(NameSpace[Entry: '__main__.Net.construct'], __main__.Net.construct)
     : (<External, NoShape>, <External, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
 23
#------------------------> 0
 %1(CNode_2) = %0(%para1_x1)
     : (<Tensor[Float32], (3, 32)>) -> (<null>)
     #scope: (Default)
 Return(%1)
     : (<null>)
     #scope: (Default)
     # In file t2.py:17, 7~15/       return x/
}
# Order:
#   1: @__main___Net_construct_6:CNode_3{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> Entry: '__main__.Net.construct', [2]: ValueNode<Symbol> __main__.Net.construct}
#   2: @__main___Net_construct_6:CNode_2{[0]: CNode_3, [1]: param_x1}
#   3: @__main___Net_construct_6:CNode_162{[0]: ValueNode<Primitive> Return, [1]: CNode_2}
 37
 38
subgraph attr:
subgraph instance: __main___Net_construct_6 : 0x14bb64c18
# In file t2.py:14~17, 4~15/    def construct(self, x1):/
subgraph @__main___Net_construct_6(%para0_x1) {
 %0(CNode_7) = resolve(NameSpace[SymbolStr: 'Namespace:__main__'], ops)
     : (<External, NoShape>, <External, NoShape>) -> (<External, NoShape>)
     #scope: (Default)
     # In file t2.py:15, 11~14/       x = ops.matmul(x1, self.weight)/
 %1(CNode_11) = getattr(%0, "bias_add")
     : (<External, NoShape>, <String, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file t2.py:16, 11~23/       x = ops.bias_add(x, self.bias)/
 %2(CNode_8) = getattr(%0, "matmul")
     : (<External, NoShape>, <String, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file t2.py:15, 11~21/       x = ops.matmul(x1, self.weight)/
 %3(CNode_9) = resolve(NameSpace[ClassMember: 'Namespace:__main__..<Net::4300844784>'], weight)
     : (<External, NoShape>, <External, NoShape>) -> (<Ref[Tensor[Float32]], (32, 8)>)
     #scope: (Default)
     # In file t2.py:15, 26~37/       x = ops.matmul(x1, self.weight)/
 %4(x) = %2(%para0_x1, %3)
     : (<Tensor[Float32], (3, 32)>, <Ref[Tensor[Float32]], (32, 8)>) -> (<Tensor[Float32], (3, 8)>)
     #scope: (Default)
     # In file t2.py:15, 11~38/       x = ops.matmul(x1, self.weight)/
 %5(CNode_12) = resolve(NameSpace[ClassMember: 'Namespace:__main__..<Net::4300844784>'], bias)
     : (<External, NoShape>, <External, NoShape>) -> (<Ref[Tensor[Float32]], (4)>)
     #scope: (Default)
     # In file t2.py:16, 27~36/       x = ops.bias_add(x, self.bias)/
 67
#------------------------> 1
 %6(x) = %1(%4, %5)
     : (<Tensor[Float32], (3, 8)>, <Ref[Tensor[Float32]], (4)>) -> (<null>)
     #scope: (Default)
     # In file t2.py:16, 11~37/       x = ops.bias_add(x, self.bias)/
 Return(%6)
     : (<null>)
     #scope: (Default)
     # In file t2.py:17, 7~15/       return x/
}
# Order:
#   1: @__main___Net_construct_6:CNode_7{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> SymbolStr: 'Namespace:__main__', [2]: ValueNode<Symbol> ops}
#   2: @__main___Net_construct_6:CNode_8{[0]: ValueNode<Primitive> getattr, [1]: CNode_7, [2]: ValueNode<StringImm> matmul}
#   3: @__main___Net_construct_6:CNode_9{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> ClassMember: 'Namespace:__main__..<Net::4300844784>', [2]: ValueNode<Symbol> weight}
#   4: @__main___Net_construct_6:CNode_163{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> CommonOPS: 'Namespace:mindspore._extends.parse.trope', [2]: ValueNode<Symbol> MakeTuple}
#   6: @__main___Net_construct_6:x{[0]: CNode_8, [1]: param_x1, [2]: CNode_9}
#   7: @__main___Net_construct_6:CNode_11{[0]: ValueNode<Primitive> getattr, [1]: CNode_7, [2]: ValueNode<StringImm> bias_add}
#   8: @__main___Net_construct_6:CNode_12{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> ClassMember: 'Namespace:__main__..<Net::4300844784>', [2]: ValueNode<Symbol> bias}
#   9: @__main___Net_construct_6:CNode_164{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> CommonOPS: 'Namespace:mindspore._extends.parse.trope', [2]: ValueNode<Symbol> MakeTuple}
#  11: @__main___Net_construct_6:x{[0]: CNode_11, [1]: x, [2]: CNode_12}
#  12: @__main___Net_construct_6:CNode_162{[0]: ValueNode<Primitive> Return, [1]: x}
 89
 90
subgraph attr:
subgraph instance: bias_add_14 : 0x128910e18
# In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6244~6280, 0~37/def bias_add(input_x, bias):/
subgraph @bias_add_14(%para0_input_x, %para0_bias) {
 %0(CNode_15) = resolve(NameSpace[SymbolStr: 'Namespace:mindspore.ops.function.nn_func'], _get_cache_prim)
     : (<External, NoShape>, <External, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6279, 18~33/    bias_add_op = _get_cache_prim(P.BiasAdd)(data_format="NCHW")/
 %1(CNode_16) = resolve(NameSpace[SymbolStr: 'Namespace:mindspore.ops.function.nn_func'], P)
     : (<External, NoShape>, <External, NoShape>) -> (<External, NoShape>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6279, 34~35/    bias_add_op = _get_cache_prim(P.BiasAdd)(data_format="NCHW")/
 %2(CNode_17) = getattr(%1, "BiasAdd")
     : (<External, NoShape>, <String, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6279, 34~43/    bias_add_op = _get_cache_prim(P.BiasAdd)(data_format="NCHW")/
 %3(CNode_18) = %0(%2)
     : (<Func, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6279, 18~44/    bias_add_op = _get_cache_prim(P.BiasAdd)(data_format="NCHW")/
 %4(CNode_131) = resolve(NameSpace[CommonOPS: 'Namespace:mindspore._extends.parse.trope'], make_dict)
     : (<External, NoShape>, <External, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6279, 18~64/    bias_add_op = _get_cache_prim(P.BiasAdd)(data_format="NCHW")/
 %5(CNode_132) = resolve(NameSpace[CommonOPS: 'Namespace:mindspore._extends.parse.trope'], MakeTuple)
     : (<External, NoShape>, <External, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6279, 18~64/    bias_add_op = _get_cache_prim(P.BiasAdd)(data_format="NCHW")/
 %6(CNode_133) = %5("data_format")
     : (<String, NoShape>) -> (<Tuple[String], TupleShape(NoShape)>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6279, 18~64/    bias_add_op = _get_cache_prim(P.BiasAdd)(data_format="NCHW")/
 %7(CNode_135) = resolve(NameSpace[CommonOPS: 'Namespace:mindspore._extends.parse.trope'], MakeTuple)
     : (<External, NoShape>, <External, NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6279, 18~64/    bias_add_op = _get_cache_prim(P.BiasAdd)(data_format="NCHW")/
 %8(CNode_136) = %7("NCHW")
     : (<String, NoShape>) -> (<Tuple[String], TupleShape(NoShape)>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6279, 18~64/    bias_add_op = _get_cache_prim(P.BiasAdd)(data_format="NCHW")/
 %9(CNode_21) = %4(%6, %8)
     : (<Tuple[String], TupleShape(NoShape)>, <Tuple[String], TupleShape(NoShape)>) -> (<Dictionary[[data_format,],[String]], NoShape>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6279, 18~64/    bias_add_op = _get_cache_prim(P.BiasAdd)(data_format="NCHW")/
 %10(bias_add_op) = UnpackCall_unpack_call(%3, %9)
     : (<Func, NoShape>, <Dictionary[[data_format,],[String]], NoShape>) -> (<Func, NoShape>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6279, 18~64/    bias_add_op = _get_cache_prim(P.BiasAdd)(data_format="NCHW")/
139
#------------------------> 2
 %11(CNode_22) = %10(%para0_input_x, %para0_bias)
     : (<Tensor[Float32], (3, 8)>, <Ref[Tensor[Float32]], (4)>) -> (<null>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6280, 11~37/    return bias_add_op(input_x, bias)/
 Return(%11)
     : (<null>)
     #scope: (Default)
     # In file /workspace/mindspore/build/package/mindspore/ops/function/nn_func.py:6280, 4~37/    return bias_add_op(input_x, bias)/
}
# Order:
#   1: @bias_add_14:CNode_15{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> SymbolStr: 'Namespace:mindspore.ops.function.nn_func', [2]: ValueNode<Symbol> _get_cache_prim}
#   2: @bias_add_14:CNode_16{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> SymbolStr: 'Namespace:mindspore.ops.function.nn_func', [2]: ValueNode<Symbol> P}
#   3: @bias_add_14:CNode_17{[0]: ValueNode<Primitive> getattr, [1]: CNode_16, [2]: ValueNode<StringImm> BiasAdd}
#   4: @bias_add_14:CNode_166{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> CommonOPS: 'Namespace:mindspore._extends.parse.trope', [2]: ValueNode<Symbol> MakeTuple}
#   6: @bias_add_14:CNode_18{[0]: CNode_15, [1]: CNode_17}
#   7: @bias_add_14:CNode_132{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> CommonOPS: 'Namespace:mindspore._extends.parse.trope', [2]: ValueNode<Symbol> MakeTuple}
#   8: @bias_add_14:CNode_133{[0]: CNode_132, [1]: ValueNode<StringImm> data_format}
#   9: @bias_add_14:CNode_135{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> CommonOPS: 'Namespace:mindspore._extends.parse.trope', [2]: ValueNode<Symbol> MakeTuple}
#  10: @bias_add_14:CNode_136{[0]: CNode_135, [1]: ValueNode<StringImm> NCHW}
#  11: @bias_add_14:CNode_131{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> CommonOPS: 'Namespace:mindspore._extends.parse.trope', [2]: ValueNode<Symbol> make_dict}
#  12: @bias_add_14:CNode_21{[0]: CNode_131, [1]: CNode_133, [2]: CNode_136}
#  13: @bias_add_14:bias_add_op{[0]: ValueNode<UnpackCall> MetaFuncGraph-unpack_call.20, [1]: CNode_18, [2]: CNode_21}
#  14: @bias_add_14:CNode_167{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> CommonOPS: 'Namespace:mindspore._extends.parse.trope', [2]: ValueNode<Symbol> MakeTuple}
#  16: @bias_add_14:CNode_22{[0]: bias_add_op, [1]: param_input_x, [2]: param_bias}
#  17: @bias_add_14:CNode_165{[0]: ValueNode<Primitive> Return, [1]: CNode_22}
166
167
# ===============================================================================================
# The total of function graphs in evaluation stack: 3/5 (Ignored 2 internal frames).
# ===============================================================================================
171
172
# ===============================================================================================
# The rest function graphs are the following:
# ===============================================================================================
No more function graphs.

Search ------------------------> to the position where inferring failed at line 68. According to ...(%4, %5) : (<Tensor[Float32], (3, 8)>, <Ref[Tensor[Float32]], (4)>) -> (), BiasAdd's inputs are %4 and %5. That %4' with shape [3, 8] and %5 with shape [4] doesn't meet the requirement about bias (Tensor) - The bias tensor, with shape (C). C must be the same as channel dimension C of input_x... for BiasAdd API. Thus, an error happens.

To solve this problem, we need modify the shape of %4 or %5 (namely self.bias).

For %5 (namely self.bias), we modify the shape of self.bias by self.bias = Parameter(initializer('zeros', [8]), name="bias").
For %4, we need know what %4 is. According to line 59, %4 is a MatMul with output shape [3, 8]. Its inputs are (%para0_x1, %3). The shape of the first input (namely given arg x) is [3, 32] and the shape of the second input (namely self.weight) is [32, 8]. To meet the requirement of BiasAdd with the data shape [4], the shape of %4 output needs to be [3, 4]. Therefore, we modify self.weight by self.weight = Parameter(initializer('normal', [32, 4]), name="weight").