gRPC-based MindSpore Serving Access
Overview
Before running the sample network, ensure that MindSpore Serving has been properly installed and the environment variables are configured. To install and configure MindSpore Serving on your PC, go to the MindSpore Serving installation page.
The gRPC API is provided to access the MindSpore Serving. In the Python environment, the mindspore_serving.client module is provided to fill in requests and parse responses. The Serving server depends on the specific inference hardware environment, and the client running does not depend on a specific hardware environment. The following uses add
and ResNet-50
as examples to describe how to use the gRPC Python API on clients.
add
This example comes from add example. The add
Servable provides the add_common
method to add up two 2x2 tensors. The code of the gRPC Python client is as follows. One gRPC request includes three pairs of independent 2x2 tensors.
from mindspore_serving.client import Client
import numpy as np
def run_add_common():
"""invoke Servable add method add_common"""
client = Client("127.0.0.1:5500", "add", "add_common")
instances = []
# instance 1
x1 = np.asarray([[1, 1], [1, 1]]).astype(np.float32)
x2 = np.asarray([[1, 1], [1, 1]]).astype(np.float32)
instances.append({"x1": x1, "x2": x2})
# instance 2
x1 = np.asarray([[2, 2], [2, 2]]).astype(np.float32)
x2 = np.asarray([[2, 2], [2, 2]]).astype(np.float32)
instances.append({"x1": x1, "x2": x2})
# instance 3
x1 = np.asarray([[3, 3], [3, 3]]).astype(np.float32)
x2 = np.asarray([[3, 3], [3, 3]]).astype(np.float32)
instances.append({"x1": x1, "x2": x2})
result = client.infer(instances)
print(result)
if __name__ == '__main__':
run_add_common()
Export the model, start the Serving server, and run the preceding client code by referring to the MindSpore Serving-based Inference Service Deployment. After the code runs properly, the following information is displayed. To facilitate display, the format is adjusted.
[{'y': array([[2., 2.], [2., 2.]], dtype=float32)},
{'y': array([[4., 4.], [4., 4.]], dtype=float32)},
{'y': array([[6., 6.], [6., 6.]], dtype=float32)}]
Details are described as follows:
Build
Client
.When
Client
is built, the network address of Serving are indicated, and the Servable’s name and method it provides are given. Servable indicates a single model or a combination of multiple models (not supported yet) and provides different services in various methods.In the preceding
add
example, Serving runs on thelocalhost
, the gRPC port number is set to5500
, andadd
Servable is run to provide theadd_common
method.Add instances.
Each gRPC request includes one or more independent instances which do not affect each other’s result.
For example, the
add_common
method provided byadd
Servable provides the function of adding two 2x2 tensors. That is, an instance contains two 2x2 tensor inputs and one 2x2 tensor output. A request may include one or more such instances, and one result is returned for each instance. The precedingadd
example provides three instances, so it is expected that three results will be returned.The input parameters of the
Client.infer
API can be a list or tuple consisting of one or more instance inputs, or a single instance input. Each instance input consists of the input name and value. The value can be in the following format:numpy array
: represents a tensor. For example, np.ones((3,224), np.float32).numpy number
: represents a scalar. For example, np.int8(5).python bool int float
: represents a scalar. Currently, int is regarded as int64, and float is regarded as float64. For example, 32.0.python str
: represents a character string. For example, “this is a text”.python bytes
: represents binary data. For example, image data.
In the preceding example,
x1
andx2
are the input parameters of theadd_common
method provided byadd
Servable. Each input value is specified when an instance is added.Obtain the inference result.
Use
Client.infer
to enter one or more instances. The return results may be in the following format:If all instances are correctly inferred, the following result is returned:
[{'y': array([[2., 2.], [2., 2.]], dtype=float32)}, {'y': array([[4., 4.], [4., 4.]], dtype=float32)}, {'y': array([[6., 6.], [6., 6.]], dtype=float32)}]
If certain errors occur in all instances , a dict containing
error
is returned. In the example,add_common
is changed toadd_common2
, and the returned result is as follows:{'error', 'Request Servable(add) method(add_common2), method is not available'}
If inference errors occur in certain instances, the error instances return a dict containing
error
. In the example, an inputdtype
of instance2 is changed tonp.int32
, and the returned result is as follows:[{'y': array([[2., 2.], [2., 2.]], dtype=float32)}, {'error': 'Given model input 1 data type kMSI_Int32 not match ...'}, {'y': array([[6., 6.], [6., 6.]], dtype=float32)}]
Each instance returns a dict. The key value comes from the Servable method definition. In this example, the
add_common
method provided byadd
Servable has only one output, which isy
. The value is in the following format:Serving Output Type
Client Return Type
Description
Example
Tensor
numpy array
Tensor array
np.ones((3,224), np.float32)
Scalar:
int8, int16, int32, int64,
uint8, uint16, uint32, uint64,
bool, float16, float32, float64numpy scalar
Converts data format from scalar to numpy scalar.
np.int8(5)
String
python str
Converts output format from character string to python str.
“news_car”
Bytes
python bytes
Converts output format from binary to python bytes.
Image data
ResNet-50
This example comes from ResNet-50 example. ResNet-50
Servable provides the classify_top1
method to recognize images. In the classify_top1
method, input the image data to obtain the output character string, perform operations such as decoding and resizing on images, and then perform inference. The classification label with the highest score is returned through post-processing.
import os
from mindspore_serving.client import Client
def run_classify_top1():
client = Client("localhost:5500", "resnet50", "classify_top1")
instances = []
for path, _, file_list in os.walk("./test_image/"):
for file_name in file_list:
image_file = os.path.join(path, file_name)
print(image_file)
with open(image_file, "rb") as fp:
instances.append({"image": fp.read()})
result = client.infer(instances)
print(result)
if __name__ == '__main__':
run_classify_top1()
The preceding classify_top1
method requires users to input image
(the image binary data) in each instance.
If the execution is properly completed, the following information is displayed:
[{'label': 'tabby, tabby cat'}, {'label': 'ox'}]
If the ResNet-50 model is not trained, there may be other unknown classification results.
Accessing Serving Server through Unix Domain Socket
MindSpore Serving server and client can communicate through TCP/IP. When they are inside one machine, they can also communicate through Unix domain socket to improve the communication performance.
When the serving server starts the grpc service, the address
parameter of mindspore_serving.server.start_grpc_server
should be filled with 'unix:{some_file_path}'
as the access address of the gRPC service, where {some_file_path}
is a relative or absolute file path, and the directory where the file is located must already exist. After the interface is successfully called, the file will be overwrited. At the same time, the ‘address’ parameter of mindspore_serving.client.Client
should be filled with the above address. For example:
The Server:
import os
import sys
from mindspore_serving import server
def start():
servable_dir = os.path.dirname(os.path.realpath(sys.argv[0]))
servable_config = server.ServableStartConfig(servable_directory=servable_dir, servable_name="resnet50",
device_ids=(0, 1))
server.start_servables(servable_configs=servable_config)
server.start_grpc_server(address="unix:/tmp/resnet50_test_temp_file")
if __name__ == "__main__":
start()
The Client:
import os
from mindspore_serving.client import Client
def run_classify_top1():
client = Client("unix:/tmp/resnet50_test_temp_file", "resnet50", "classify_top1")
instances = []
for path, _, file_list in os.walk("./test_image/"):
for file_name in file_list:
image_file = os.path.join(path, file_name)
print(image_file)
with open(image_file, "rb") as fp:
instances.append({"image": fp.read()})
result = client.infer(instances)
print(result)
if __name__ == '__main__':
run_classify_top1()
Accessing SSL or TLS enabled Serving Service
Mindspore Serving supports server and client communicating based on SSL/TLS
.
SSL/TLS
is a secure communication protocol that can be used to verify the identity of a client or server, encrypt all data, and secure communication.
Digital certificates are used to identify the server or client, and private keys are used to decrypt data and sign information digests.
We can use openssl to generate the private keys and certificates related to server and client.
Here’s an example of how to generate a certificate and perform single-bidirectional authentication:
One-way authentication
Only the client verifies the identity of the server, so we need the server’s certificate and private key. You can execute the following openssl command to generate the relevant certificate.
# Generate the root certificate used to issue the certificate of server or client
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ca.key -out ca.crt -subj "/C=CN/ST=xx/L=xx/OU=gRPC/CN=Root"
# Generate server's private key
openssl genrsa -out server.key 2048
# Generate server's certificate sign request
# You can customize the server name on the certificate by setting CN (Common Name). In this case we can set CN to localhost.
# When the gRPC client accesses the server with this certificate, address needs to be localhost.
openssl req -new -key server.key -out server.csr -subj "/C=XX/ST=MyST/L=XX/O=HW/OU=gRPC/CN=localhost"
# Use the root certificate to issue a server certificate
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256
We got server.key
, server.crt
and ca.crt
files. Pass them to the corresponding SSLConfig
.
Server:
import os import sys from mindspore_serving import server def start(): servable_dir = os.path.dirname(os.path.realpath(sys.argv[0])) servable_config = server.ServableStartConfig(servable_directory=servable_dir, servable_name="add", device_ids=(0, 1)) server.start_servables(servable_configs=servable_config) ssl_config = server.SSLConfig(certificate="server.crt", private_key="server.key", custom_ca=None, verify_client=False) server.start_grpc_server(address="127.0.0.1:5500", ssl_config=ssl_config) if __name__ == "__main__": start()
ssl_config
represents the server’sSSL
configuration. This parameter defaults toNone
, which meansSSL/TLS
is not enabled. EnablingSSL/TLS
requiresmindspore serving.server.SSLConfig
object passed to this parameter.certificate
is the path to the server’s certificate file.private_key
is the path to the server’s private key file.custom_ca
is the path to the server’s root certificate file which is for verifying client certificate. Whenverify_client
isTrue
, the client’s certificate needs to be verified, so this parameter can’t beNone
, the corresponding path must be passed in.verify_client
indicates whether to verify the identity of the client.
Setting
verify_client
toFalse
represents one-way authentication. We pass in the certificateserver.crt
and the private keyserver.key
, respectively. Due to the server does not need to verify the client socustom_ca
is ignored.Client:
from mindspore_serving.client import Client from mindspore_serving.client import SSLConfig import numpy as np def run_add_common(): """invoke Servable add method add_common""" ssl_config = SSLConfig(custom_ca="ca.crt") client = Client("localhost:5500", "add", "add_common", ssl_config=ssl_config) instances = [] # instance 1 x1 = np.asarray([[1, 1], [1, 1]]).astype(np.float32) x2 = np.asarray([[1, 1], [1, 1]]).astype(np.float32) instances.append({"x1": x1, "x2": x2}) result = client.infer(instances) print(result) if __name__ == '__main__': run_add_common()
ssl_config
represents the client’sSSL
configuration. This parameter defaults toNone
, which meansSSL/TLS
is not enabled. EnablingSSL/TLS
requires passingmindspore_serving.client.SSLConfig
object tossl_config
.certificate
is the path to the client’s certificate file.private_key
is the path to the client’s private key file.custom_ca
is the path to the client’s root certificate file, which is used to verify the identity of the server. This parameter can beNone
, at which point gRPC finds the corresponding root certificate through thegrpc/_cython/_credentials/roots.pem
file under the gRPC installation path or theGRPC_DEFAULT_SSL_ROOTS_FILE_PATH
environment variable.
Because only the client verifies the server certificate, you only need to set
custom_ca
toca.crt
which issues the server’s certificate.
Mutual authentication
Both the client and the server need to verify each other’s identity, so in addition to the server’s certificate, we need to execute the following command to generate the client’s certificate.
# Generate client's private key
openssl genrsa -out client.key 2048
# Generate client's certificate sign request
openssl req -new -key client.key -out client.csr -subj "/C=XX/ST=MyST/L=XX/O=HW/OU=gRPC/CN=client"
# Use root certificate to issue client's certificate
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 365 -sha256
We got client.key
andcleint.crt
.
Server:
import os import sys from mindspore_serving import server def start(): servable_dir = os.path.dirname(os.path.realpath(sys.argv[0])) servable_config = server.ServableStartConfig(servable_directory=servable_dir, servable_name="add", device_ids=(0, 1)) server.start_servables(servable_configs=servable_config) ssl_config = server.SSLConfig(certificate="server.crt", private_key="server.key", custom_ca="ca.crt", verify_client=True) server.start_grpc_server(address="127.0.0.1:5500", ssl_config=ssl_config) if __name__ == "__main__": start()
Setting
verify_client
toTrue
represents two-way authentication. Also setcustom_ca
toca.crt
to verify the client certificate.Client:
from mindspore_serving.client import Client from mindspore_serving.client import SSLConfig import numpy as np def run_add_common(): """invoke Servable add method add_common""" ssl_config = SSLConfig(certificate="client.crt", private_key="client.key", custom_ca="ca.crt") client = Client("localhost:5500", "add", "add_common", ssl_config=ssl_config) instances = [] # instance 1 x1 = np.asarray([[1, 1], [1, 1]]).astype(np.float32) x2 = np.asarray([[1, 1], [1, 1]]).astype(np.float32) instances.append({"x1": x1, "x2": x2}) result = client.infer(instances) print(result) if __name__ == '__main__': run_add_common()
The client needs providing its own certificate to the server for authentication, and we pass in the client’s certificate
client.crt
and the private keyclient.key
, respectively.
When the gRPC server and client are not enabling SSL/TLS
at the same time, the server side or client side will get ssl3_get_record:wrong version number
error,
and you will need to confirm that both the server and the client have enabled SSL/TLS
.