Supported models and schemas

When you configure object enrichments, you choose which machine learning models analyze your stored objects. Each model evaluates eligible images or PDF documents and attaches structured metadata directly to your Cloud Storage objects.

This page describes each supported model, its eligibility requirements, and its output schema to help you parse and query enrichment metadata in BigQuery or through an API.

Relationship between models and object enrichments

You activate models by creating a feature configuration (FeatureConfig) that specifies which buckets to scan and which models to run. During asynchronous processing, Cloud Storage evaluates eligible objects by using your chosen models and generates structured JSON metadata called object enrichments.

Each model produces a dedicated JSON payload that Cloud Storage attaches directly to the object as a Google-managed object context under the contexts.custom property. A unique context key prefixed with google:annotation: identifies each payload.

Depending on your workflow, you can consume this metadata in two ways:

• Inspect individual objects: Retrieve enrichments for a specific object by using object APIs, such as the viewFullContext method (Preview).
• Analyze at scale: Query enriched metadata across all objects in a bucket by using Storage Insights datasets and standard SQL in BigQuery.

Face detection

Use the face-detector model to detect human faces in stored images. The model detects bounding polygons for entire heads and individual faces, and locates 36 facial landmarks, such as eyes, nose, and mouth. It also calculates head orientation angles, such as roll, pan, and tilt, and assesses emotional expression likelihoods.

Model details

• Model name: face-detector
• Context key: google:annotation:face-detector-v1
• Eligible objects: Image objects 20 MiB or smaller with a supported Content-Type of BMP, GIF, ICO, JPEG, PNG, TIFF, or WEBP.

Output schema

The model outputs a JSON object containing modelName, modelVersion, and a faceAnnotationGroup with an array of faceAnnotations. If model execution fails, the JSON object contains an error object instead of faceAnnotationGroup.

BoundingPoly

The BoundingPoly object represents a bounding polygon enclosing a detected image feature.

Vertex

The Vertex object represents a 2D coordinate point scaled to the original image dimensions.

NormalizedVertex

The NormalizedVertex object represents a 2D coordinate point expressed as a ratio relative to original image dimensions.

FaceLandmark

The FaceLandmark object represents a specific facial landmark detected within a face.

Landmark type

The model recognizes 36 specific facial landmarks:

• TYPE_UNSPECIFIED
• LEFT_EYE
• RIGHT_EYE
• LEFT_OF_LEFT_EYEBROW
• RIGHT_OF_LEFT_EYEBROW
• LEFT_OF_RIGHT_EYEBROW
• RIGHT_OF_RIGHT_EYEBROW
• MIDPOINT_BETWEEN_EYES
• NOSE_TIP
• UPPER_LIP
• LOWER_LIP
• MOUTH_LEFT
• MOUTH_RIGHT
• MOUTH_CENTER
• NOSE_BOTTOM_RIGHT
• NOSE_BOTTOM_LEFT
• NOSE_BOTTOM_CENTER
• LEFT_EYE_TOP_BOUNDARY
• LEFT_EYE_RIGHT_CORNER
• LEFT_EYE_BOTTOM_BOUNDARY
• LEFT_EYE_LEFT_CORNER
• RIGHT_EYE_TOP_BOUNDARY
• RIGHT_EYE_RIGHT_CORNER
• RIGHT_EYE_BOTTOM_BOUNDARY
• RIGHT_EYE_LEFT_CORNER
• LEFT_EYEBROW_UPPER_MIDPOINT
• RIGHT_EYEBROW_UPPER_MIDPOINT
• LEFT_EAR_TRAGION
• RIGHT_EAR_TRAGION
• LEFT_EYE_PUPIL
• RIGHT_EYE_PUPIL
• FOREHEAD_GLABELLA
• CHIN_GNATHION
• CHIN_LEFT_GONION
• CHIN_RIGHT_GONION
• LEFT_CHEEK_CENTER
• RIGHT_CHEEK_CENTER

Position

The Position object represents the 3D coordinates of a facial landmark.

Sample output

{
  "modelName": "face-detector",
  "modelVersion": "v1",
  "faceAnnotationGroup": {
    "faceAnnotations": [
      {
        "boundingPoly": {
          "vertices": [
            { "x": 1079, "y": 148 },
            { "x": 2152, "y": 148 },
            { "x": 2152, "y": 1396 },
            { "x": 1079, "y": 1396 }
          ],
          "normalizedVertices": [
            { "x": 0.35, "y": 0.12 },
            { "x": 0.70, "y": 0.12 },
            { "x": 0.70, "y": 0.95 },
            { "x": 0.35, "y": 0.95 }
          ]
        },
        "faceDetectionBoundingPoly": {
          "vertices": [
            { "x": 1190, "y": 427 },
            { "x": 2037, "y": 427 },
            { "x": 2037, "y": 1285 },
            { "x": 1190, "y": 1285 }
          ]
        },
        "landmarks": [
          {
            "type": "LEFT_EYE",
            "position": { "x": 1368.8, "y": 737.6, "z": 0.0016 }
          },
          {
            "type": "RIGHT_EYE",
            "position": { "x": 1659.2, "y": 755.1, "z": -116.4 }
          }
        ],
        "rollAngle": 1.959,
        "panAngle": -21.954,
        "tiltAngle": -2.267,
        "detectionConfidence": 0.98,
        "landmarkingConfidence": 0.61,
        "joyProbability": 0.18,
        "headwearProbability": 0.80
      }
    ]
  }
}

Object detection

Use the object-detector model to detect and track multiple distinct objects in an image. For each identified object, the model provides an entity name, a Knowledge Graph entity ID (knowledgeGraphId), a confidence score, and bounding coordinates.

Model details

• Model name: object-detector
• Context key: google:annotation:object-detector-v1
• Eligible objects: Image objects 20 MiB or smaller with a supported Content-Type of BMP, GIF, ICO, JPEG, PNG, TIFF, or WEBP.

Output schema

The model outputs a JSON object containing modelName, modelVersion, and an objectAnnotationGroup with an array of objectAnnotations. If model execution fails, the JSON object contains an error object instead of objectAnnotationGroup.

Sample output

{
  "modelName": "object-detector",
  "modelVersion": "v1",
  "objectAnnotationGroup": {
    "objectAnnotations": [
      {
        "knowledgeGraphId": "/m/01g317",
        "name": "Person",
        "score": 0.89,
        "boundingPoly": {
          "normalizedVertices": [
            { "x": 0.1, "y": 0.15 },
            { "x": 0.2, "y": 0.15 },
            { "x": 0.2, "y": 0.3 },
            { "x": 0.1, "y": 0.3 }
          ]
        }
      }
    ]
  }
}

General image labeling

Use the label-detector model to generate descriptive labels, keywords, and broad semantic categories for images. The model identifies general entities, scenery, activities, and locations, such as Landscape, Urban, or Night, making your visual assets searchable and catalogable.

Model details

• Model name: label-detector
• Context key: google:annotation:label-detector-v1
• Eligible objects: Image objects 20 MiB or smaller with a supported Content-Type of BMP, GIF, ICO, JPEG, PNG, TIFF, or WEBP.

Output schema

The model outputs a JSON object containing modelName, modelVersion, and a labelAnnotationGroup with an array of labelAnnotations. If model execution fails, the JSON object contains an error object instead of labelAnnotationGroup.

Sample output

{
  "modelName": "label-detector",
  "modelVersion": "v1",
  "labelAnnotationGroup": {
    "labelAnnotations": [
      {
        "knowledgeGraphId": "/m/01d74z",
        "description": "Night",
        "score": 0.92,
        "topicality": 0.19
      }
    ]
  }
}

Explicit content detection

Use the content-safety model to screen uploaded images for sensitive, explicit, or inappropriate content. The model evaluates images against Google safety policies. It assesses the likelihood of adult, violent, and other sensitive material to help you automate content moderation workflows.

Model details

• Model name: content-safety
• Context key: google:annotation:content-safety-v1
• Eligible objects: Image objects 20 MiB or smaller with a supported Content-Type of BMP, GIF, ICO, JPEG, PNG, TIFF, or WEBP.

Output schema

The model outputs a JSON object containing modelName, modelVersion, and a contentSafetyAnnotation object with likelihood ratings. If model execution fails, the JSON object contains an error object instead of contentSafetyAnnotation.

Sample output

{
  "modelName": "content-safety",
  "modelVersion": "v1",
  "contentSafetyAnnotation": {
    "adultContentLikelihood": "VERY_UNLIKELY",
    "violentContentLikelihood": "UNLIKELY"
  }
}

PDF classification

Use the pdf-classification model to automatically categorize PDF documents based on their textual and visual structure. The model assigns standardized hierarchical categories for business, industry, and topics. These categories help you organize large document collections, ground generative AI retrieval, and route documents to downstream workflows.

Model details

• Model name: pdf-classification
• Context key: google:annotation:pdf-classification-v1
• Eligible objects: PDF document objects 20 MiB or smaller with a Content-Type of application/pdf.

Output schema

The model outputs a JSON object containing modelName, modelVersion, and a pdfClassificationCategoryList with an array of classified categories. If model execution fails, the JSON object contains an error object instead of pdfClassificationCategoryList.

Supported categories

The pdf-classification model assigns standardized hierarchical content categories to PDF documents.

Categories are structured hierarchically with forward slashes (/) separating category depth. They begin with a broad top-level vertical and narrow to specific subcategories:

• Top-level verticals: Broad subject domains, such as /Finance, /Law & Government, or /Computers & Electronics.
• Subcategories: Specific subjects within a vertical, such as /Finance/Accounting & Auditing/Tax Preparation & Planning.
• /Other subcategories: Categories for content that aligns with the parent category but doesn't fit into any specific child subcategory.

If a document covers multiple distinct topics, the model can assign multiple high-confidence categories to a single PDF object.

Top-level categories

The model classifies PDF documents across 27 top-level categories:

• /Adult
• /Arts & Entertainment
• /Autos & Vehicles
• /Beauty & Fitness
• /Books & Literature
• /Business & Industrial
• /Computers & Electronics
• /Finance
• /Food & Drink
• /Games
• /Health
• /Hobbies & Leisure
• /Home & Garden
• /Internet & Telecom
• /Jobs & Education
• /Law & Government
• /News
• /Online Communities
• /People & Society
• /Pets & Animals
• /Real Estate
• /Reference
• /Science
• /Sensitive Subjects
• /Shopping
• /Sports
• /Travel & Transportation

Subcategories follow the hierarchical path /Top-Level-Vertical/Subcategory, with optional deeper levels, under these top-level categories. For the complete list of all supported category paths, see PDF classification categories.

Sample output

{
  "modelName": "pdf-classification",
  "modelVersion": "v1",
  "pdfClassificationCategoryList": {
    "categories": [
      {
        "label": "/Finance/Accounting & Auditing/Tax Preparation & Planning",
        "score": 0.85
      },
      {
        "label": "/Business & Industrial/Printing & Publishing/Document & Printing Services",
        "score": 0.25
      }
    ]
  }
}

What's next

• Read about object enrichments.
• Configure object enrichments.
• Manage object enrichments.
• Analyze object enrichments in BigQuery.
• Review the featureConfigs JSON API resource.
• Troubleshoot object enrichments issues.