mindarmour.fuzz_testing

This module provides a neuron coverage-gain based fuzz method to evaluate the robustness of given model.

class mindarmour.fuzz_testing.CoverageMetrics(model, incremental=False, batch_size=32)[source]

The abstract base class for Neuron coverage classes calculating coverage metrics.

As we all known, each neuron output of a network will have a output range after training (we call it original range), and test dataset is used to estimate the accuracy of the trained network. However, neurons’ output distribution would be different with different test datasets. Therefore, similar to function fuzz, model fuzz means testing those neurons’ outputs and estimating the proportion of original range that has emerged with test datasets.

Reference: DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems

Parameters

  • model (Model) – The pre-trained model which waiting for testing.

  • incremental (bool) – Metrics will be calculate in incremental way or not. Default: False.

  • batch_size (int) – The number of samples in a fuzz test batch. Default: 32.

abstract get_metrics(dataset)[source]

Calculate coverage metrics of given dataset.

Parameters

dataset (numpy.ndarray) – Dataset used to calculate coverage metrics.

Raises

NotImplementedError – It is an abstract method.

class mindarmour.fuzz_testing.Fuzzer(target_model)[source]

Fuzzing test framework for deep neural networks.

Reference: DeepHunter: A Coverage-Guided Fuzz Testing Framework for Deep Neural Networks

Parameters

target_model (Model) – Target fuzz model.

Examples

>>> from mindspore.common.initializer import TruncatedNormal
>>> from mindspore.ops import operations as P
>>> from mindspore.train import Model
>>> from mindspore.ops import TensorSummary
>>> from mindarmour.fuzz_testing import Fuzzer
>>> from mindarmour.fuzz_testing import KMultisectionNeuronCoverage
>>> class Net(nn.Cell):
...     def __init__(self):
...         super(Net, self).__init__()
...         self.conv1 = nn.Conv2d(1, 6, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.conv2 = nn.Conv2d(6, 16, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.fc1 = nn.Dense(16 * 5 * 5, 120, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc2 = nn.Dense(120, 84, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc3 = nn.Dense(84, 10, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.relu = nn.ReLU()
...         self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)
...         self.reshape = P.Reshape()
...         self.summary = TensorSummary()
...
...     def construct(self, x):
...         x = self.conv1(x)
...         x = self.relu(x)
...         self.summary('conv1', x)
...         x = self.max_pool2d(x)
...         x = self.conv2(x)
...         x = self.relu(x)
...         self.summary('conv2', x)
...         x = self.max_pool2d(x)
...         x = self.reshape(x, (-1, 16 * 5 * 5))
...         x = self.fc1(x)
...         x = self.relu(x)
...         self.summary('fc1', x)
...         x = self.fc2(x)
...         x = self.relu(x)
...         self.summary('fc2', x)
...         x = self.fc3(x)
...         self.summary('fc3', x)
...         return x
>>> net = Net()
>>> model = Model(net)
>>> mutate_config = [{'method': 'GaussianBlur',
...                   'params': {'ksize': [1, 2, 3, 5], 'auto_param': [True, False]}},
...                  {'method': 'MotionBlur',
...                   'params': {'degree': [1, 2, 5], 'angle': [45, 10, 100, 140, 210, 270, 300],
...                   'auto_param': [True]}},
...                  {'method': 'UniformNoise',
...                   'params': {'factor': [0.1, 0.2, 0.3], 'auto_param': [False, True]}},
...                  {'method': 'GaussianNoise',
...                   'params': {'factor': [0.1, 0.2, 0.3], 'auto_param': [False, True]}},
...                  {'method': 'Contrast',
...                   'params': {'alpha': [0.5, 1, 1.5], 'beta': [-10, 0, 10], 'auto_param': [False, True]}},
...                  {'method': 'Rotate',
...                   'params': {'angle': [20, 90], 'auto_param': [False, True]}},
...                  {'method': 'FGSM',
...                   'params': {'eps': [0.3, 0.2, 0.4], 'alpha': [0.1], 'bounds': [(0, 1)]}}]
>>> batch_size = 8
>>> num_classe = 10
>>> train_images = np.random.rand(32, 1, 32, 32).astype(np.float32)
>>> test_images = np.random.rand(batch_size, 1, 32, 32).astype(np.float32)
>>> test_labels = np.random.randint(num_classe, size=batch_size).astype(np.int32)
>>> test_labels = (np.eye(num_classe)[test_labels]).astype(np.float32)
>>> initial_seeds = []
>>> # make initial seeds
>>> for img, label in zip(test_images, test_labels):
...     initial_seeds.append([img, label])
>>> initial_seeds = initial_seeds[:10]
>>> nc = KMultisectionNeuronCoverage(model, train_images, segmented_num=100, incremental=True)
>>> model_fuzz_test = Fuzzer(model)
>>> samples, gt_labels, preds, strategies, metrics = model_fuzz_test.fuzzing(mutate_config, initial_seeds,
...                                                                          nc, max_iters=100)
fuzzing(mutate_config, initial_seeds, coverage, evaluate=True, max_iters=10000, mutate_num_per_seed=20)[source]

Fuzzing tests for deep neural networks.

Parameters

  • mutate_config (list) – Mutate configs. The format is [{‘method’: ‘GaussianBlur’, ‘params’: {‘ksize’: [1, 2, 3, 5], ‘auto_param’: [True, False]}}, {‘method’: ‘UniformNoise’, ‘params’: {‘factor’: [0.1, 0.2, 0.3], ‘auto_param’: [False, True]}}, {‘method’: ‘GaussianNoise’, ‘params’: {‘factor’: [0.1, 0.2, 0.3], ‘auto_param’: [False, True]}}, {‘method’: ‘Contrast’, ‘params’: {‘alpha’: [0.5, 1, 1.5], ‘beta’: [-10, 0, 10], ‘auto_param’: [False, True]}}, {‘method’: ‘Rotate’, ‘params’: {‘angle’: [20, 90], ‘auto_param’: [False, True]}}, {‘method’: ‘FGSM’, ‘params’: {‘eps’: [0.3, 0.2, 0.4], ‘alpha’: [0.1], ‘bounds’: [(0, 1)]}}] …]. The supported methods list is in self._strategies, and the params of each method must within the range of optional parameters. Supported methods are grouped in two types: Firstly, natural robustness methods include: ‘Translate’, ‘Scale’, ‘Shear’, ‘Rotate’, ‘Perspective’, ‘Curve’, ‘GaussianBlur’, ‘MotionBlur’, ‘GradientBlur’, ‘Contrast’, ‘GradientLuminance’, ‘UniformNoise’, ‘GaussianNoise’, ‘SaltAndPepperNoise’, ‘NaturalNoise’. Secondly, attack methods include: ‘FGSM’, ‘PGD’ and ‘MDIIM’. ‘FGSM’, ‘PGD’ and ‘MDIIM’. are abbreviations of FastGradientSignMethod, ProjectedGradientDescent and MomentumDiverseInputIterativeMethod. mutate_config must have method in [‘Contrast’, ‘GradientLuminance’, ‘GaussianBlur’, ‘MotionBlur’, ‘GradientBlur’, ‘UniformNoise’, ‘GaussianNoise’, ‘SaltAndPepperNoise’, ‘NaturalNoise’]. The way of setting parameters for first and second type methods can be seen in ‘mindarmour/natural_robustness/transform/image’. For third type methods, the optional parameters refer to self._attack_param_checklists.

  • initial_seeds (list[list]) – Initial seeds used to generate mutated samples. The format of initial seeds is [[image_data, label], […], …] and the label must be one-hot.

  • coverage (CoverageMetrics) – Class of neuron coverage metrics.

  • evaluate (bool) – return evaluate report or not. Default: True.

  • max_iters (int) – Max number of select a seed to mutate. Default: 10000.

  • mutate_num_per_seed (int) – The number of mutate times for a seed. Default: 20.

Returns

  • list, mutated samples in fuzz_testing.

  • list, ground truth labels of mutated samples.

  • list, preds of mutated samples.

  • list, strategies of mutated samples.

  • dict, metrics report of fuzzer.

Raises

  • ValueError – Coverage must be subclass of CoverageMetrics.

  • ValueError – If initial seeds is empty.

  • ValueError – If element of seed is not two in initial seeds.

class mindarmour.fuzz_testing.KMultisectionNeuronCoverage(model, train_dataset, segmented_num=100, incremental=False, batch_size=32)[source]

Get the metric of ‘k-multisection neuron coverage’. KMNC measures how thoroughly the given set of test inputs covers the range of neurons output values derived from training dataset.

Parameters

  • model (Model) – The pre-trained model which waiting for testing.

  • train_dataset (numpy.ndarray) – Training dataset used for determine the neurons’ output boundaries.

  • segmented_num (int) – The number of segmented sections of neurons’ output intervals. Default: 100.

  • incremental (bool) – Metrics will be calculate in incremental way or not. Default: False.

  • batch_size (int) – The number of samples in a fuzz test batch. Default: 32.

get_metrics(dataset)[source]

Get the metric of ‘k-multisection neuron coverage’.

Parameters

dataset (numpy.ndarray) – Dataset used to calculate coverage metrics.

Returns

float, the metric of ‘k-multisection neuron coverage’.

Examples

>>> from mindspore.common.initializer import TruncatedNormal
>>> from mindspore.ops import operations as P
>>> from mindspore.train import Model
>>> from mindspore.ops import TensorSummary
>>> from mindarmour.fuzz_testing import KMultisectionNeuronCoverage
>>> class Net(nn.Cell):
...     def __init__(self):
...         super(Net, self).__init__()
...         self.conv1 = nn.Conv2d(1, 6, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.conv2 = nn.Conv2d(6, 16, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.fc1 = nn.Dense(16 * 5 * 5, 120, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc2 = nn.Dense(120, 84, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc3 = nn.Dense(84, 10, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.relu = nn.ReLU()
...         self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)
...         self.reshape = P.Reshape()
...         self.summary = TensorSummary()
...     def construct(self, x):
...         x = self.conv1(x)
...         x = self.relu(x)
...         self.summary('conv1', x)
...         x = self.max_pool2d(x)
...         x = self.conv2(x)
...         x = self.relu(x)
...         self.summary('conv2', x)
...         x = self.max_pool2d(x)
...         x = self.reshape(x, (-1, 16 * 5 * 5))
...         x = self.fc1(x)
...         x = self.relu(x)
...         self.summary('fc1', x)
...         x = self.fc2(x)
...         x = self.relu(x)
...         self.summary('fc2', x)
...         x = self.fc3(x)
...         self.summary('fc3', x)
...         return x
>>> net = Net()
>>> model = Model(net)
>>> batch_size = 8
>>> num_classe = 10
>>> train_images = np.random.rand(32, 1, 32, 32).astype(np.float32)
>>> test_images = np.random.rand(batch_size, 1, 32, 32).astype(np.float32)
>>> kmnc = KMultisectionNeuronCoverage(model, train_images, segmented_num=100)
>>> metrics = kmnc.get_metrics(test_images)
class mindarmour.fuzz_testing.NeuronBoundsCoverage(model, train_dataset, incremental=False, batch_size=32)[source]

Get the metric of ‘neuron boundary coverage’ \(NBC = (|UpperCornerNeuron| + |LowerCornerNeuron|)/(2*|N|)\), where \(|N|\) is the number of neurons, NBC refers to the proportion of neurons whose neurons output value in the test dataset exceeds the upper and lower bounds of the corresponding neurons output value in the training dataset.

Parameters

  • model (Model) – The pre-trained model which waiting for testing.

  • train_dataset (numpy.ndarray) – Training dataset used for determine the neurons’ output boundaries.

  • incremental (bool) – Metrics will be calculate in incremental way or not. Default: False.

  • batch_size (int) – The number of samples in a fuzz test batch. Default: 32.

get_metrics(dataset)[source]

Get the metric of ‘neuron boundary coverage’.

Parameters

dataset (numpy.ndarray) – Dataset used to calculate coverage metrics.

Returns

float, the metric of ‘neuron boundary coverage’.

Examples

>>> from mindspore.common.initializer import TruncatedNormal
>>> from mindspore.ops import operations as P
>>> from mindspore.train import Model
>>> from mindspore.ops import TensorSummary
>>> from mindarmour.fuzz_testing import NeuronBoundsCoverage
>>> class Net(nn.Cell):
...     def __init__(self):
...         super(Net, self).__init__()
...         self.conv1 = nn.Conv2d(1, 6, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.conv2 = nn.Conv2d(6, 16, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.fc1 = nn.Dense(16 * 5 * 5, 120, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc2 = nn.Dense(120, 84, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc3 = nn.Dense(84, 10, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.relu = nn.ReLU()
...         self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)
...         self.reshape = P.Reshape()
...         self.summary = TensorSummary()
...     def construct(self, x):
...         x = self.conv1(x)
...         x = self.relu(x)
...         self.summary('conv1', x)
...         x = self.max_pool2d(x)
...         x = self.conv2(x)
...         x = self.relu(x)
...         self.summary('conv2', x)
...         x = self.max_pool2d(x)
...         x = self.reshape(x, (-1, 16 * 5 * 5))
...         x = self.fc1(x)
...         x = self.relu(x)
...         self.summary('fc1', x)
...         x = self.fc2(x)
...         x = self.relu(x)
...         self.summary('fc2', x)
...         x = self.fc3(x)
...         self.summary('fc3', x)
...         return x
>>> net = Net()
>>> model = Model(net)
>>> batch_size = 8
>>> num_classe = 10
>>> train_images = np.random.rand(32, 1, 32, 32).astype(np.float32)
>>> test_images = np.random.rand(batch_size, 1, 32, 32).astype(np.float32)
>>> nbc = NeuronBoundsCoverage(model, train_images)
>>> metrics = nbc.get_metrics(test_images)
class mindarmour.fuzz_testing.NeuronCoverage(model, threshold=0.1, incremental=False, batch_size=32)[source]

Calculate the neurons activated coverage. Neuron is activated when its output is greater than the threshold. Neuron coverage equals the proportion of activated neurons to total neurons in the network.

Parameters

  • model (Model) – The pre-trained model which waiting for testing.

  • threshold (float) – Threshold used to determined neurons is activated or not. Default: 0.1.

  • incremental (bool) – Metrics will be calculate in incremental way or not. Default: False.

  • batch_size (int) – The number of samples in a fuzz test batch. Default: 32.

get_metrics(dataset)[source]

Get the metric of neuron coverage: the proportion of activated neurons to total neurons in the network.

Parameters

dataset (numpy.ndarray) – Dataset used to calculate coverage metrics.

Returns

float, the metric of ‘neuron coverage’.

Examples

>>> from mindspore.common.initializer import TruncatedNormal
>>> from mindspore.ops import operations as P
>>> from mindspore.train import Model
>>> from mindspore.ops import TensorSummary
>>> from mindarmour.fuzz_testing import NeuronCoverage
>>> class Net(nn.Cell):
...     def __init__(self):
...         super(Net, self).__init__()
...         self.conv1 = nn.Conv2d(1, 6, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.conv2 = nn.Conv2d(6, 16, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.fc1 = nn.Dense(16 * 5 * 5, 120, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc2 = nn.Dense(120, 84, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc3 = nn.Dense(84, 10, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.relu = nn.ReLU()
...         self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)
...         self.reshape = P.Reshape()
...         self.summary = TensorSummary()
...     def construct(self, x):
...         x = self.conv1(x)
...         x = self.relu(x)
...         self.summary('conv1', x)
...         x = self.max_pool2d(x)
...         x = self.conv2(x)
...         x = self.relu(x)
...         self.summary('conv2', x)
...         x = self.max_pool2d(x)
...         x = self.reshape(x, (-1, 16 * 5 * 5))
...         x = self.fc1(x)
...         x = self.relu(x)
...         self.summary('fc1', x)
...         x = self.fc2(x)
...         x = self.relu(x)
...         self.summary('fc2', x)
...         x = self.fc3(x)
...         self.summary('fc3', x)
...         return x
>>> net = Net()
>>> model = Model(net)
>>> batch_size = 8
>>> num_classe = 10
>>> train_images = np.random.rand(32, 1, 32, 32).astype(np.float32)
>>> test_images = np.random.rand(batch_size, 1, 32, 32).astype(np.float32)
>>> nc = NeuronCoverage(model, threshold=0.1)
>>> nc_metrics = nc.get_metrics(test_images)
class mindarmour.fuzz_testing.SuperNeuronActivateCoverage(model, train_dataset, incremental=False, batch_size=32)[source]

Get the metric of ‘super neuron activation coverage’. \(SNAC = |UpperCornerNeuron|/|N|\). SNAC refers to the proportion of neurons whose neurons output value in the test set exceeds the upper bounds of the corresponding neurons output value in the training set.

Parameters

  • model (Model) – The pre-trained model which waiting for testing.

  • train_dataset (numpy.ndarray) – Training dataset used for determine the neurons’ output boundaries.

  • incremental (bool) – Metrics will be calculate in incremental way or not. Default: False.

  • batch_size (int) – The number of samples in a fuzz test batch. Default: 32.

get_metrics(dataset)[source]

Get the metric of ‘strong neuron activation coverage’.

Parameters

dataset (numpy.ndarray) – Dataset used to calculate coverage metrics.

Returns

float, the metric of ‘strong neuron activation coverage’.

Examples

>>> from mindspore.common.initializer import TruncatedNormal
>>> from mindspore.ops import operations as P
>>> from mindspore.train import Model
>>> from mindspore.ops import TensorSummary
>>> from mindarmour.fuzz_testing import SuperNeuronActivateCoverage
>>> class Net(nn.Cell):
...     def __init__(self):
...         super(Net, self).__init__()
...         self.conv1 = nn.Conv2d(1, 6, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.conv2 = nn.Conv2d(6, 16, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.fc1 = nn.Dense(16 * 5 * 5, 120, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc2 = nn.Dense(120, 84, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc3 = nn.Dense(84, 10, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.relu = nn.ReLU()
...         self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)
...         self.reshape = P.Reshape()
...         self.summary = TensorSummary()
...     def construct(self, x):
...         x = self.conv1(x)
...         x = self.relu(x)
...         self.summary('conv1', x)
...         x = self.max_pool2d(x)
...         x = self.conv2(x)
...         x = self.relu(x)
...         self.summary('conv2', x)
...         x = self.max_pool2d(x)
...         x = self.reshape(x, (-1, 16 * 5 * 5))
...         x = self.fc1(x)
...         x = self.relu(x)
...         self.summary('fc1', x)
...         x = self.fc2(x)
...         x = self.relu(x)
...         self.summary('fc2', x)
...         x = self.fc3(x)
...         self.summary('fc3', x)
...         return x
>>> net = Net()
>>> model = Model(net)
>>> batch_size = 8
>>> num_classe = 10
>>> train_images = np.random.rand(32, 1, 32, 32).astype(np.float32)
>>> test_images = np.random.rand(batch_size, 1, 32, 32).astype(np.float32)
>>> snac = SuperNeuronActivateCoverage(model, train_images)
>>> metrics = snac.get_metrics(test_images)
class mindarmour.fuzz_testing.TopKNeuronCoverage(model, top_k=3, incremental=False, batch_size=32)[source]

Calculate the top k activated neurons coverage. Neuron is activated when its output has the top k largest value in that hidden layers. Top k neurons coverage equals the proportion of activated neurons to total neurons in the network.

Parameters

  • model (Model) – The pre-trained model which waiting for testing.

  • top_k (int) – Neuron is activated when its output has the top k largest value in that hidden layers. Default: 3.

  • incremental (bool) – Metrics will be calculate in incremental way or not. Default: False.

  • batch_size (int) – The number of samples in a fuzz test batch. Default: 32.

get_metrics(dataset)[source]

Get the metric of Top K activated neuron coverage.

Parameters

dataset (numpy.ndarray) – Dataset used to calculate coverage metrics.

Returns

float, the metrics of ‘top k neuron coverage’.

Examples

>>> from mindspore.common.initializer import TruncatedNormal
>>> from mindspore.ops import operations as P
>>> from mindspore.train import Model
>>> from mindspore.ops import TensorSummary
>>> from mindarmour.fuzz_testing import TopKNeuronCoverage
>>> class Net(nn.Cell):
...     def __init__(self):
...         super(Net, self).__init__()
...         self.conv1 = nn.Conv2d(1, 6, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.conv2 = nn.Conv2d(6, 16, 5, padding=0, weight_init=TruncatedNormal(0.02), pad_mode="valid")
...         self.fc1 = nn.Dense(16 * 5 * 5, 120, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc2 = nn.Dense(120, 84, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.fc3 = nn.Dense(84, 10, TruncatedNormal(0.02), TruncatedNormal(0.02))
...         self.relu = nn.ReLU()
...         self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)
...         self.reshape = P.Reshape()
...         self.summary = TensorSummary()
...     def construct(self, x):
...         x = self.conv1(x)
...         x = self.relu(x)
...         self.summary('conv1', x)
...         x = self.max_pool2d(x)
...         x = self.conv2(x)
...         x = self.relu(x)
...         self.summary('conv2', x)
...         x = self.max_pool2d(x)
...         x = self.reshape(x, (-1, 16 * 5 * 5))
...         x = self.fc1(x)
...         x = self.relu(x)
...         self.summary('fc1', x)
...         x = self.fc2(x)
...         x = self.relu(x)
...         self.summary('fc2', x)
...         x = self.fc3(x)
...         self.summary('fc3', x)
...         return x
>>> net = Net()
>>> model = Model(net)
>>> batch_size = 8
>>> num_classe = 10
>>> train_images = np.random.rand(32, 1, 32, 32).astype(np.float32)
>>> test_images = np.random.rand(batch_size, 1, 32, 32).astype(np.float32)
>>> tknc = TopKNeuronCoverage(model, top_k=3)
>>> metrics = tknc.get_metrics(test_images)