Um die Umstellung von Vector Search 1.0 zu vereinfachen, wurde eine neue Funktion in die ImportDataObjects API eingeführt.
Der Migrationsprozess umfasst drei wichtige Schritte:
Sammlung mit einem passenden Schema erstellen: Vor dem Importieren müssen Sie eine Sammlung erstellen. Das zugehörige Datenschema muss so strukturiert sein, dass die transformierten Vector Search 1.0-Daten aufgenommen werden können.
Importvorgang starten Rufen Sie die
ImportDataObjectsAPI auf und geben Sie den Cloud Storage-Speicherort Ihrer Vector Search 1.0-Daten an. Aktivieren Sie außerdem das Konvertierungs-Flagdetect_and_convert_vs1_json.Datentransformation verstehen: Machen Sie sich damit vertraut, wie Ihre Vector Search 1.0-Datenfelder der neuen Datenobjektstruktur zugeordnet werden.
Sammlung erstellen
Erstellen Sie zuerst eine Sammlung mit einem Datenschema, das die Struktur Ihrer Vector Search 1.0-Daten widerspiegelt.
REST
curl -X POST \
'https://vectorsearch.googleapis.com/v1beta/projects/PROJECT_ID/locations/LOCATION/collections?collection_id=COLLECTION_ID' \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H 'Content-Type: application/json' \
-d '{
"data_schema": {
"type": "object",
"properties": {
"restricts": {
"type": "object",
"properties": {
"genres": {
"type": "array",
"items": {
"type": "string"
}
},
"director": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
"restricts_deny": {
"type": "object",
"properties": {
"genres": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
"numeric_restricts": {
"type": "object",
"properties": {
"year": {
"type": "integer"
},
"imdb_rating": {
"type": "number",
"format": "float"
}
}
},
"embedding_metadata": {
"type": "object",
"properties": {
"plot": {
"type": "string"
},
"customers_review_summary": {
"type": "string"
},
"critics_review_summary": {
"type": "string"
}
},
}
}
},
"vector_schema": {
"embedding": {
"dense_vector": {
"dimensions": 768
}
},
"sparse_embedding": {
"sparse_vector": {}
}
}
}'
Python
from google.cloud import vectorsearch_v1beta
# Create the client
vector_search_service_client = vectorsearch_v1beta.VectorSearchServiceClient()
# The JSON schema for the data
data_schema = {
"type": "object",
"properties": {
"restricts": {
"type": "object",
"properties": {
"genres": {"type": "array", "items": {"type": "string"}},
"director": {"type": "array", "items": {"type": "string"}},
},
},
"restricts_deny": {
"type": "object",
"properties": {
"genres": {"type": "array", "items": {"type": "string"}}
},
},
"numeric_restricts": {
"type": "object",
"properties": {
"year": {"type": "integer"},
"imdb_rating": {"type": "number", "format": "float"},
},
},
"embedding_metadata": {
"type": "object",
"properties": {
"plot": {"type": "string"},
"customers_review_summary": {"type": "string"},
"critics_review_summary": {"type": "string"},
},
},
},
}
# The JSON schema for the vector
vector_schema = {
"embedding": {"dense_vector": {"dimensions": 768}},
"sparse_embedding": {"sparse_vector": {}},
}
collection = vectorsearch_v1beta.Collection(
data_schema=data_schema,
vector_schema=vector_schema,
)
request = vectorsearch_v1beta.CreateCollectionRequest(
parent="projects/PROJECT_ID/locations/LOCATION",
collection_id="COLLECTION_ID",
collection=collection,
)
# Create the collection
operation = vector_search_service_client.create_collection(request=request)
operation.result()
Daten aus der Vektorsuche 1.0 importieren
Verwenden Sie dann die ImportDataObjects API für die neu erstellte Sammlung.
Verweisen Sie darauf, dass der Cloud Storage-Bucket Ihre Vector Search 1.0-Daten enthält.
REST
curl -X POST \
"https://vectorsearch.googleapis.com/v1beta/projects/PROJECT_ID/locations/LOCATION/collections/COLLECTION_ID:importDataObjects" \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{
"gcs_import": {
"contents_uri": "gs://your-bucket/path/to/your-data.jsonl",
"error_uri": "gs://your-bucket/path/to/import-errors/"
}
}'
Python
from google.cloud import vectorsearch_v1beta
# Create the client
vector_search_service_client = vectorsearch_v1beta.VectorSearchServiceClient()
# Initialize request
request = vectorsearch_v1beta.ImportDataObjectsRequest(
name="projects/PROJECT_ID/locations/LOCATION/collections/COLLECTION_ID",
gcs_import={
"contents_uri": "gs://your-bucket/path/to/your-data.jsonl",
"error_uri": "gs://your-bucket/path/to/import-errors/",
},
)
# Make the request
operation = vector_search_service_client.import_data_objects(request=request)
# Wait for the result (note this may take up to several minutes)
operation.result()
Datentransformation
Während des Importvorgangs werden Ihre Vector Search 1.0-Daten in Agent Retrieval Data Objects umgewandelt. Die folgenden Beispiele veranschaulichen, wie die Felder zugeordnet werden.
Cloud Storage-Dateiformat für die Vektorsuche 1.0
{
"id": "movie-789",
"embedding": [-0.23, 0.88, 0.11, ...],
"sparse_embedding": {"values": [0.1, 0.2], "dimensions": [1, 4]},
"restricts": [
{"namespace": "genres", "allow": ["science-fiction", "action"], "deny": ["horror"]},
{"namespace": "director", "allow": ["Christopher Nolan"]}
],
"numeric_restricts": [
{"namespace": "year", "value_int": 2010},
{"namespace": "imdb_rating", "value_float": 8.8}
],
"embedding_metadata": {
"plot": "...",
"customers_review_summary": "...",
"critics_review_summary": "..."
}
}
Transformiertes Datenobjekt zum Abrufen von Agenten
DataObject(
name="/.../movie-789",
data={
"restricts": {
"genres": ["science-fiction", "action"],
"director": ["Christopher Nolan"],
},
"restricts_deny": {
"genres": ["horror"]
},
"numeric_restricts": {
"year": 2010,
"imdb_rating": 8.8,
},
"embedding_metadata": {
"plot": "...",
"customers_review_summary": "...",
"critics_review_summary": "...",
}
},
vectors={
"embedding": {"dense_vector": {"values": [-0.23, 0.88, 0.11, ...]}},
"sparse_embedding": {"sparse_vector": {"values": [0.1, 0.2], "indices": [1, 4]}},
}
)