|
As you progress your machine studying (ML) workloads into manufacturing, you must repeatedly monitor your deployed fashions and iterate while you observe a deviation in your mannequin efficiency. When you construct a brand new mannequin, you sometimes begin validating the mannequin offline utilizing historic inference request knowledge. But this knowledge typically fails to account for present, real-world situations. For instance, new merchandise may develop into trending that your product advice mannequin hasn’t seen but. Or, you expertise a sudden spike within the quantity of inference requests in manufacturing that you simply by no means uncovered your mannequin to earlier than.
Today, I’m excited to announce Amazon SageMaker assist for shadow testing!
Deploying a mannequin in shadow mode permits you to conduct a extra holistic take a look at by routing a duplicate of the stay inference requests for a manufacturing mannequin to the brand new (shadow) mannequin. Yet, solely the responses from the manufacturing mannequin are returned to the calling software. Shadow testing helps you construct additional confidence in your mannequin and catch potential configuration errors and efficiency points earlier than they impression finish customers. Once you full a shadow take a look at, you need to use the deployment guardrails for SageMaker inference endpoints to soundly replace your mannequin in manufacturing.
Get Started with Amazon SageMaker Shadow Testing
You can create shadow exams utilizing the brand new SageMaker Inference Console and APIs. Shadow testing provides you a completely managed expertise for setup, monitoring, viewing, and performing on the outcomes of shadow exams. If you’ve got current workflows constructed round SageMaker endpoints, you may as well deploy a mannequin in shadow mode utilizing the present SageMaker Inference APIs.
On the SageMaker console, choose Inference and Shadow exams to create, monitor, and deploy shadow exams.
To create a shadow take a look at, choose an current (or create a brand new) SageMaker endpoint and manufacturing variant you wish to take a look at in opposition to.
Next, configure the proportion of site visitors to ship to the shadow variant, the comparability metrics you wish to consider, and the period of the take a look at. You may allow knowledge seize to your manufacturing and shadow variant.
That’s it. SageMaker now routinely deploys the brand new variant in shadow mode and routes a duplicate of the inference requests to it in actual time, all throughout the identical endpoint. The following diagram illustrates this workflow.
Note that solely the responses of the manufacturing variant are returned to the calling software. You can select to both discard or log the responses of the shadow variant for offline comparability.
You may use shadow testing to validate modifications you made to any part in your manufacturing variant, together with the serving container or ML occasion. This will be helpful while you’re upgrading to a brand new framework model of your serving container, making use of patches, or if you wish to be sure that there isn’t any impression to latency or error price as a consequence of this modification. Similarly, for those who think about transferring to a different ML occasion sort, for instance, Amazon EC2 C7g situations based mostly on AWS Graviton processors, or EC2 G5 situations powered by NVIDIA A10G Tensor Core GPUs, you need to use shadow testing to judge the efficiency on manufacturing site visitors previous to rollout.
You can monitor the progress of the shadow take a look at and efficiency metrics equivalent to latency and error price by way of a stay dashboard. On the SageMaker console, choose Inference and Shadow exams, then choose the shadow take a look at you wish to monitor.
If you resolve to advertise the shadow mannequin to manufacturing, choose Deploy shadow variant and outline the infrastructure configuration to deploy the shadow variant.
You may use the SageMaker deployment guardrails if you wish to add linear or canary site visitors shifting modes and auto rollbacks to your replace.
Availability and Pricing
SageMaker assist for shadow testing is out there immediately in all AWS Regions the place SageMaker internet hosting is out there apart from the AWS GovCloud (US) Regions and AWS China Regions.
There isn’t any extra cost for SageMaker shadow testing apart from utilization fees for the ML situations and ML storage provisioned to host the shadow variant. The pricing for ML situations and ML storage dimensions is similar because the real-time inference possibility. There isn’t any extra cost for knowledge processed out and in of shadow deployments. The SageMaker pricing web page has all the main points.
To study extra, go to Amazon SageMaker shadow testing.
Start validating your new ML fashions with SageMaker shadow exams immediately!
— Antje