gcloud alpha dataplex dbt metadata-jobs create

gcloud alpha dataplex dbt metadata-jobs create - transform dbt-core artifacts and import them into Dataplex Catalog

gcloud alpha dataplex dbt metadata-jobs create [[METADATA_JOB] --location=LOCATION] --storage-uri=STORAGE_URI [--artifacts-path=ARTIFACTS_PATH; default="."] [--aspects-only] [--async] [--entry-group=ENTRY_GROUP; default="dbt-metadata-ingestion"] [--no-include-entry-links] [--skip-bigquery-link] [--validate-only] [GCLOUD_WIDE_FLAG …]

Only the entry group that receives the dbt entries must exist in the caller's project beforehand. The caller must also be able to USE the dbt connector types (dataplex.aspectTypes.use / the dbt-connector-types alternate-use permission).

Unless --aspects-only is passed, the import is a FULL sync of the dbt contents of the entry group: any dbt entry in the entry group that this run's artifacts do not describe is DELETED. Give each dbt project its own entry group. Two dbt projects importing their own artifacts into one shared entry group will each delete the other's entries on every run.

The Metadata Job ID identifies the import run and, if provided, must:

• Contain only lowercase letters, numbers, and hyphens.
• Start with a letter and end with a number or a letter.
• Be 1-63 characters and unique within the project / location.

gcloud alpha dataplex dbt metadata-jobs create my-dbt-import --project=my-project --location=us-central1 --artifacts-path=. --entry-group=dbt-metadata-ingestion --storage-uri=gs://my-bucket/dbt-imports/

The artifacts may also be read from Cloud Storage, e.g. when they are published there by a dbt CI job:

gcloud alpha dataplex dbt metadata-jobs create my-dbt-import --project=my-project --location=us-central1 --artifacts-path=gs://my-bucket/dbt-artifacts/ --entry-group=dbt-metadata-ingestion --storage-uri=gs://my-bucket/dbt-imports/

To only build and upload the JSONL and validate the job without ingesting, add --validate-only.

Metadata job resource - Arguments and flags that define the Dataplex metdata job you want to create. The arguments in this group can be used to specify the attributes of this resource. (NOTE) Some attributes are not given arguments in this group but can be set in other ways.

To set the project attribute:

• provide the argument metadata_job on the command line with a fully specified name;
• job ID is optional and will be generated if not specified with a fully specified name;
• provide the argument --project on the command line;
• set the property core/project.

[METADATA_JOB]

ID of the metadata job or fully qualified identifier for the metadata job.

To set the metadata_job attribute:

• provide the argument metadata_job on the command line;
• job ID is optional and will be generated if not specified.

--location=LOCATION

The location of the Dataplex resource.

To set the location attribute:

• provide the argument metadata_job on the command line with a fully specified name;
• job ID is optional and will be generated if not specified with a fully specified name;
• provide the argument --location on the command line;
• set the property dataplex/location.

--storage-uri=STORAGE_URI

Cloud Storage URI prefix (gs://bucket/path/) the transformed JSONL is uploaded to and the import job reads from. The caller must have write access and the Dataplex service agent must have read access.

--artifacts-path=ARTIFACTS_PATH; default="."

Path to the dbt-core artifacts: a local directory, or a Cloud Storage folder (gs://bucket/folder/) they were published to. May point at the dbt project root (the target/ subdirectory is detected automatically) or directly at the directory containing manifest.json. manifest.json is required; catalog.json, run_results.json and sources.json are read if present. Defaults to the current working directory.

--aspects-only

Update only the metadata this dbt run observed, and leave the rest of the entry group untouched. No entry is created, deleted or re-parented, no entry link is emitted, and an aspect whose dbt artifact was absent from this run keeps the value a previous run gave it.

Use this for routine ingestion, after whichever dbt command your pipeline already runs: dbt build, dbt test, dbt source freshness, or a --select-narrowed rebuild. It is safe to run repeatedly and from several jobs.

Omit it when the set of dbt resources itself changed (a model added, renamed or deleted), since only a full run creates and prunes entries. A full run also refreshes display names, descriptions, labels, entry links and the entry hierarchy, which this flag leaves alone; and because a full run must write every entry's required aspects, run it from as complete an artifact set as your pipeline can produce.

The first ingestion into an entry group must be a full run: there are no entries to attach aspects to yet.

--async

Return immediately, without waiting for the operation in progress to complete.

--entry-group=ENTRY_GROUP; default="dbt-metadata-ingestion"

Short ID of the entry group that receives the dbt entries. Must already exist in the project / location. Use a separate entry group per dbt project: without --aspects-only, a run deletes the dbt entries in this entry group that its own artifacts do not describe.

--include-entry-links

Also emit EntryLink records capturing dbt lineage and semantic relationships (depends-on-lineage-imported, represents, depends-on-imported, etc.).

Enabled by default, use --no-include-entry-links to disable.

--skip-bigquery-link

Skip represents links (dbt node -> physical BigQuery table entry). Otherwise a represents link is emitted for each materialized dbt node (model/seed/snapshot) whose BigQuery dataset lives in the import location (--location); links can only reference @bigquery entries in that same region, so datasets in another region are skipped automatically. Use this flag when the BigQuery tables are not cataloged in Dataplex.

--validate-only

Build and upload the JSONL and validate the metadata job, but don't actually ingest.

Run $ gcloud help for details.