The offical website give the client image. However, in some situations, we want to build the client environment locally because it will reduce the storage and communication.
A Triton backend is the implementation that executes a model. A backend can be a wrapper around a deep-learning framework, like PyTorch, TensorFlow, TensorRT or ONNX Runtime. Or a backend can be custom C/C++ logic performing any operation (for example, image pre-processing).
What is TensorRT
The TensorRT backend is used to execute TensorRT models. The server repo contains the source for the backend.
The OpenVINO backend is used to execute OpenVINO models. The openvino_backend repo contains the documentation and source for the backend.
Inter
常用深度学习框架
TensorFlow
The TensorFlow backend is used to execute TensorFlow models in both GraphDef and SavedModel formats. The same backend is used to execute both TensorFlow 1 and TensorFlow 2 models. The tensorflow_backend repo contains the documentation and source for the backend.
PyTorch
The PyTorch backend is used to execute TorchScript models. The pytorch_backend repo contains the documentation and source for the backend.
TorchScript类似于ONNX,存储了模型的参数和结构等,但仅支持tensor的操作
Python
The Python backend allows you to write your model logic in Python. For example, you can use this backend to execute pre/post processing code written in Python, or to execute a PyTorch Python script directly (instead of first converting it to TorchScript and then using the PyTorch backend). The python_backend repo contains the documentation and source for the backend.
加速预处理框架
DALI
DALI is a collection of highly optimized building blocks and an execution engine that accelerates the pre-processing of the input data for deep learning applications. The DALI backend allows you to execute your DALI pipeline within Triton. The dali_backend repo contains the documentation and source for the backend.
A Triton backend must implement the C interface defined in tritonbackend.h
what is in tritonbackend.h
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
// vi triton/core/tritonbackend.h
#include"triton/core/tritonserver.h" structTRITONBACKEND_MemoryManager; structTRITONBACKEND_Input; structTRITONBACKEND_Output; structTRITONBACKEND_State; structTRITONBACKEND_Request; structTRITONBACKEND_ResponseFactory; /// A response factory allows a request /// to be released before all responses have been sent. Releasing a /// request as early as possible releases all input tensor data and /// therefore may be desirable in some cases. structTRITONBACKEND_Response; structTRITONBACKEND_Backend; structTRITONBACKEND_Model; structTRITONBACKEND_ModelInstance;
project(tutorialminimalbackend LANGUAGES C CXX) # 项目名字和指定的语言
# # Options # # Must include options required for this project as well as any # projects included in this one by FetchContent. # # GPU support is disabled by default because minimal backend doesn't # use GPUs. # option(TRITON_ENABLE_GPU "Enable GPU support in backend" OFF) # 是否使用某些库 option(TRITON_ENABLE_STATS "Include statistics collections in backend" ON) # 定义一些变量,CACHE指的是作用于整个cmake的全局变量 set(TRITON_COMMON_REPO_TAG "main" CACHE STRING "Tag for triton-inference-server/common repo") set(TRITON_CORE_REPO_TAG "main" CACHE STRING "Tag for triton-inference-server/core repo") set(TRITON_BACKEND_REPO_TAG "main" CACHE STRING "Tag for triton-inference-server/backend repo")
# # Dependencies # # FetchContent requires us to include the transitive closure of all # repos that we depend on so that we can override the tags. # # include 指令用来载入并运行来自于文件或模块的 CMake 代码 include(FetchContent)
# # The backend must be built into a shared library. Use an ldscript to # hide all symbols except for the TRITONBACKEND API. # configure_file(src/libtriton_minimal.ldscript libtriton_minimal.ldscript COPYONLY)
target_link_libraries( triton-minimal-backend PRIVATE triton-core-serverapi # from repo-core triton-core-backendapi # from repo-core triton-core-serverstub # from repo-core triton-backend-utils # from repo-backend )
# # Export from build tree # export( EXPORT triton-minimal-backend-targets FILE ${CMAKE_CURRENT_BINARY_DIR}/TutorialMinimalBackendTargets.cmake NAMESPACE TutorialMinimalBackend:: )
export(PACKAGE TutorialMinimalBackend)
Concept
TRITONBACKEND_Backend
A TRITONBACKEND_Backend object represents the backend itself. The same backend object is shared across all models that use the backend. The associated API, like TRITONBACKEND_BackendName, is used to get information about the backend and to associate a user-defined state with the backend.
A backend can optionally implement TRITONBACKEND_Initialize and TRITONBACKEND_Finalize to get notification of when the backend object is created and destroyed (for more information see backend lifecycles).
TRITONBACKEND_Model
A TRITONBACKEND_Model object represents a model. Each model loaded by Triton is associated with a TRITONBACKEND_Model. Each model can use the TRITONBACKEND_ModelBackend API to get the backend object representing the backend that is used by the model.
The same model object is shared across all instances of that model. The associated API, like TRITONBACKEND_ModelName, is used to get information about the model and to associate a user-defined state with the model.
Most backends will implement TRITONBACKEND_ModelInitialize and TRITONBACKEND_ModelFinalize to initialize the backend for a given model and to manage the user-defined state associated with the model (for more information see backend lifecycles).
The backend must take into account threading concerns when implementing TRITONBACKEND_ModelInitialize and TRITONBACKEND_ModelFinalize. Triton will not perform multiple simultaneous calls to these functions for a given model; however, if a backend is used by multiple models Triton may simultaneously call the functions with a different thread for each model. As a result, the backend must be able to handle multiple simultaneous calls to the functions. Best practice for backend implementations is to use only function-local and model-specific user-defined state in these functions, as is shown in the tutorial.
TRITONBACKEND_ModelInstance
A TRITONBACKEND_ModelInstance object represents a model instance. Triton creates one or more instances of the model based on the instance_group settings specified in the model configuration. Each of these instances is associated with a TRITONBACKEND_ModelInstance object.
The only function that the backend must implement is TRITONBACKEND_ModelInstanceExecute. The TRITONBACKEND_ModelInstanceExecute function is called by Triton to perform inference/computation on a batch of inference requests. Most backends will also implement TRITONBACKEND_ModelInstanceInitialize and TRITONBACKEND_ModelInstanceFinalize to initialize the backend for a given model instance and to manage the user-defined state associated with the model (for more information see backend lifecycles).
The backend must take into account threading concerns when implementing TRITONBACKEND_ModelInstanceInitialize, TRITONBACKEND_ModelInstanceFinalize and TRITONBACKEND_ModelInstanceExecute. Triton will not perform multiple simultaneous calls to these functions for a given model instance; however, if a backend is used by a model with multiple instances or by multiple models Triton may simultaneously call the functions with a different thread for each model instance. As a result, the backend must be able to handle multiple simultaneous calls to the functions. Best practice for backend implementations is to use only function-local and model-specific user-defined state in these functions, as is shown in the tutorial.
TRITONBACKEND_Request
A TRITONBACKEND_Request object represents an inference request made to the model. The backend takes ownership of the request object(s) in TRITONBACKEND_ModelInstanceExecute and must release each request by calling TRITONBACKEND_RequestRelease. However, the ownership of request object is returned back to Triton in case TRITONBACKEND_ModelInstanceExecute returns an error. See Inference Requests and Responses for more information about request lifecycle.
The Triton Backend API allows the backend to get information about the request as well as the input and request output tensors of the request. Each request input is represented by a TRITONBACKEND_Input object.
TRITONBACKEND_Response
A TRITONBACKEND_Response object represents a response sent by the backend for a specific request. The backend uses the response API to set the name, shape, datatype and tensor values for each output tensor included in the response. The response can indicate either a failed or a successful request. See Inference Requests and Responses for more information about request-response lifecycle.
// Copyright 2021, NVIDIA CORPORATION & AFFILIATES. All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions // are met: // * Redistributions of source code must retain the above copyright // notice, this list of conditions and the following disclaimer. // * Redistributions in binary form must reproduce the above copyright // notice, this list of conditions and the following disclaimer in the // documentation and/or other materials provided with the distribution. // * Neither the name of NVIDIA CORPORATION nor the names of its // contributors may be used to endorse or promote products derived // from this software without specific prior written permission. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY // EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE // IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR // PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR // CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, // EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, // PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR // PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY // OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
// // Minimal backend that demonstrates the TRITONBACKEND API. This // backend works for any model that has 1 input called "IN0" with // INT32 datatype and shape [ 4 ] and 1 output called "OUT0" with // INT32 datatype and shape [ 4 ]. The backend supports both batching // and non-batching models. // // For each batch of requests, the backend returns the input tensor // value in the output tensor. //
/////////////
// // ModelState // // State associated with a model that is using this backend. An object // of this class is created and associated with each // TRITONBACKEND_Model. ModelState is derived from BackendModel class // provided in the backend utilities that provides many common // functions. // classModelState : public BackendModel { public: static TRITONSERVER_Error* Create( TRITONBACKEND_Model* triton_model, ModelState** state); virtual ~ModelState() = default;
// Triton calls TRITONBACKEND_ModelInitialize when a model is loaded // to allow the backend to create any state associated with the model, // and to also examine the model configuration to determine if the // configuration is suitable for the backend. Any errors reported by // this function will prevent the model from loading. // TRITONSERVER_Error* TRITONBACKEND_ModelInitialize(TRITONBACKEND_Model* model) { // Create a ModelState object and associate it with the // TRITONBACKEND_Model. If anything goes wrong with initialization // of the model state then an error is returned and Triton will fail // to load the model. ModelState* model_state; RETURN_IF_ERROR(ModelState::Create(model, &model_state)); RETURN_IF_ERROR( TRITONBACKEND_ModelSetState(model, reinterpret_cast<void*>(model_state))); // 创建模型 returnnullptr; // success }
// Triton calls TRITONBACKEND_ModelFinalize when a model is no longer // needed. The backend should cleanup any state associated with the // model. This function will not be called until all model instances // of the model have been finalized. // TRITONSERVER_Error* TRITONBACKEND_ModelFinalize(TRITONBACKEND_Model* model) { void* vstate; RETURN_IF_ERROR(TRITONBACKEND_ModelState(model, &vstate)); ModelState* model_state = reinterpret_cast<ModelState*>(vstate); delete model_state; // 销毁模型 returnnullptr; // success }
} // extern "C"
/////////////
// // ModelInstanceState // // State associated with a model instance. An object of this class is // created and associated with each // TRITONBACKEND_ModelInstance. ModelInstanceState is derived from // BackendModelInstance class provided in the backend utilities that // provides many common functions. // classModelInstanceState : public BackendModelInstance { public: static TRITONSERVER_Error* Create( ModelState* model_state, TRITONBACKEND_ModelInstance* triton_model_instance, ModelInstanceState** state); virtual ~ModelInstanceState() = default;
// Get the state of the model that corresponds to this instance. ModelState* StateForModel()const{ return model_state_; }
// Triton calls TRITONBACKEND_ModelInstanceInitialize when a model // instance is created to allow the backend to initialize any state // associated with the instance. // TRITONSERVER_Error* TRITONBACKEND_ModelInstanceInitialize(TRITONBACKEND_ModelInstance* instance) { // Get the model state associated with this instance's model. TRITONBACKEND_Model* model; RETURN_IF_ERROR(TRITONBACKEND_ModelInstanceModel(instance, &model));
// Create a ModelInstanceState object and associate it with the // TRITONBACKEND_ModelInstance. ModelInstanceState* instance_state; RETURN_IF_ERROR( ModelInstanceState::Create(model_state, instance, &instance_state)); RETURN_IF_ERROR(TRITONBACKEND_ModelInstanceSetState( instance, reinterpret_cast<void*>(instance_state)));
returnnullptr; // success }
// Triton calls TRITONBACKEND_ModelInstanceFinalize when a model // instance is no longer needed. The backend should cleanup any state // associated with the model instance. // TRITONSERVER_Error* TRITONBACKEND_ModelInstanceFinalize(TRITONBACKEND_ModelInstance* instance) { void* vstate; RETURN_IF_ERROR(TRITONBACKEND_ModelInstanceState(instance, &vstate)); ModelInstanceState* instance_state = reinterpret_cast<ModelInstanceState*>(vstate); delete instance_state;
returnnullptr; // success }
} // extern "C"
/////////////
extern"C" {
// When Triton calls TRITONBACKEND_ModelInstanceExecute it is required // that a backend create a response for each request in the batch. A // response may be the output tensors required for that request or may // be an error that is returned in the response. // TRITONSERVER_Error* TRITONBACKEND_ModelInstanceExecute( TRITONBACKEND_ModelInstance* instance, TRITONBACKEND_Request** requests, constuint32_t request_count) { // Triton will not call this function simultaneously for the same // 'instance'. But since this backend could be used by multiple // instances from multiple models the implementation needs to handle // multiple calls to this function at the same time (with different // 'instance' objects). Best practice for a high-performance // implementation is to avoid introducing mutex/lock and instead use // only function-local and model-instance-specific state. ModelInstanceState* instance_state; RETURN_IF_ERROR(TRITONBACKEND_ModelInstanceState( instance, reinterpret_cast<void**>(&instance_state))); ModelState* model_state = instance_state->StateForModel();
// 'responses' is initialized as a parallel array to 'requests', // with one TRITONBACKEND_Response object for each // TRITONBACKEND_Request object. If something goes wrong while // creating these response objects, the backend simply returns an // error from TRITONBACKEND_ModelInstanceExecute, indicating to // Triton that this backend did not create or send any responses and // so it is up to Triton to create and send an appropriate error // response for each request. RETURN_IF_ERROR is one of several // useful macros for error handling that can be found in // backend_common.h.
std::vector<TRITONBACKEND_Response*> responses; responses.reserve(request_count); for (uint32_t r = 0; r < request_count; ++r) { TRITONBACKEND_Request* request = requests[r]; TRITONBACKEND_Response* response; RETURN_IF_ERROR(TRITONBACKEND_ResponseNew(&response, request)); responses.push_back(response); }
// At this point, the backend takes ownership of 'requests', which // means that it is responsible for sending a response for every // request. From here, even if something goes wrong in processing, // the backend must return 'nullptr' from this function to indicate // success. Any errors and failures must be communicated via the // response objects. // // To simplify error handling, the backend utilities manage // 'responses' in a specific way and it is recommended that backends // follow this same pattern. When an error is detected in the // processing of a request, an appropriate error response is sent // and the corresponding TRITONBACKEND_Response object within // 'responses' is set to nullptr to indicate that the // request/response has already been handled an no futher processing // should be performed for that request. Even if all responses fail, // the backend still allows execution to flow to the end of the // function. RESPOND_AND_SET_NULL_IF_ERROR, and // RESPOND_ALL_AND_SET_NULL_IF_ERROR are macros from // backend_common.h that assist in this management of response // objects.
// The backend could iterate over the 'requests' and process each // one separately. But for performance reasons it is usually // preferred to create batched input tensors that are processed // simultaneously. This is especially true for devices like GPUs // that are capable of exploiting the large amount parallelism // exposed by larger data sets. // // The backend utilities provide a "collector" to facilitate this // batching process. The 'collector's ProcessTensor function will // combine a tensor's value from each request in the batch into a // single contiguous buffer. The buffer can be provided by the // backend or 'collector' can create and manage it. In this backend, // there is not a specific buffer into which the batch should be // created, so use ProcessTensor arguments that cause collector to // manage it.
// To instruct ProcessTensor to "gather" the entire batch of IN0 // input tensors into a single contiguous buffer in CPU memory, set // the "allowed input types" to be the CPU ones (see tritonserver.h // in the triton-inference-server/core repo for allowed memory // types). std::vector<std::pair<TRITONSERVER_MemoryType, int64_t>> allowed_input_types = {{TRITONSERVER_MEMORY_CPU_PINNED, 0}, {TRITONSERVER_MEMORY_CPU, 0}};
// Finalize the collector. If 'true' is returned, 'input_buffer' // will not be valid until the backend synchronizes the CUDA // stream or event that was used when creating the collector. For // this backend, GPU is not supported and so no CUDA sync should // be needed; so if 'true' is returned simply log an error. constbool need_cuda_input_sync = collector.Finalize(); if (need_cuda_input_sync) { LOG_MESSAGE( TRITONSERVER_LOG_ERROR, "'minimal' backend: unexpected CUDA sync required by collector"); }
// 'input_buffer' contains the batched "IN0" tensor. The backend can // implement whatever logic is necesary to produce "OUT0". This // backend simply returns the IN0 value in OUT0 so no actual // computation is needed.
// This backend supports models that batch along the first dimension // and those that don't batch. For non-batch models the output shape // will be [ 4 ]. For batch models the output shape will be [ -1, 4 // ] and the backend "responder" utility below will set the // appropriate batch dimension value for each response. std::vector<int64_t> output_batch_shape; bool supports_first_dim_batching; RESPOND_ALL_AND_SET_NULL_IF_ERROR( responses, request_count, model_state->SupportsFirstDimBatching(&supports_first_dim_batching)); if (supports_first_dim_batching) { output_batch_shape.push_back(-1); } output_batch_shape.push_back(4);
// Because the OUT0 values are concatenated into a single contiguous // 'output_buffer', the backend must "scatter" them out to the // individual response OUT0 tensors. The backend utilities provide // a "responder" to facilitate this scattering process.
// The 'responders's ProcessTensor function will copy the portion of // 'output_buffer' corresonding to each request's output into the // response for that request.
// Finalize the responder. If 'true' is returned, the OUT0 // tensors' data will not be valid until the backend synchronizes // the CUDA stream or event that was used when creating the // responder. For this backend, GPU is not supported and so no // CUDA sync should be needed; so if 'true' is returned simply log // an error. constbool need_cuda_output_sync = responder.Finalize(); if (need_cuda_output_sync) { LOG_MESSAGE( TRITONSERVER_LOG_ERROR, "'minimal' backend: unexpected CUDA sync required by responder"); }
// Send all the responses that haven't already been sent because of // an earlier error. for (auto& response : responses) { if (response != nullptr) { LOG_IF_ERROR( TRITONBACKEND_ResponseSend( response, TRITONSERVER_RESPONSE_COMPLETE_FINAL, nullptr), "failed to send response"); } }
// Done with the request objects so release them. for (uint32_t r = 0; r < request_count; ++r) { auto& request = requests[r]; LOG_IF_ERROR( TRITONBACKEND_RequestRelease(request, TRITONSERVER_REQUEST_RELEASE_ALL), "failed releasing request"); }
If you like this blog or find it useful for you, you are welcome to comment on it. You are also welcome to share this blog, so that more people can participate in it. If the images used in the blog infringe your copyright, please contact the author to delete them. Thank you !