Overview of getting inferences on Agent Platform

An inference is the output of a trained machine learning model. This page provides an overview of the workflow for getting inferences from your models on Gemini Enterprise Agent Platform.

Agent Platform supports online inferences:

  • Online inferences are synchronous requests made to a model that is deployed to an Endpoint. Therefore, before sending a request, you must first deploy the Model resource to an endpoint. This associates compute resources with the model so that the model can serve online inferences with low latency. Use online inferences when you are making requests in response to application input or in situations that require timely inference.

Test your model locally

Before getting inferences, it's useful to deploy your model to a local endpoint during the development and testing phase. This lets you both iterate more quickly and test your model without deploying it to an online endpoint or incurring inference costs. Local deployment is intended for local development and testing, not for production deployments.

To deploy a model locally, use the Agent Platform SDK for Python and deploy a LocalModel to a LocalEndpoint.

Even if your client is not written in Python, you can still use the Agent Platform SDK for Python to launch the container and server so that you can test requests from your client.

Get inferences from custom trained models

To get inferences, you must first import your model. After it's imported, it becomes a Model resource that is visible in Model Registry.

Then, read the following documentation to learn how to get inferences:

Deploy a model to an endpoint and get online inferences.

What's next