Pour simplifier la transition depuis Vector Search 1.0, une nouvelle fonctionnalité a été ajoutée à l'API ImportDataObjects.
Le processus de migration comporte trois étapes clés :
Créez une collection avec un schéma correspondant. Avant d'importer des données, vous devez créer une collection. Son schéma de données doit être structuré pour s'adapter aux données transformées de Vector Search 1.0.
Lancez le processus d'importation. Appelez l'API
ImportDataObjects, en spécifiant l'emplacement Cloud Storage de vos données Vector Search 1.0 et en activant l'indicateur de conversiondetect_and_convert_vs1_json.Comprenez la transformation des données. Familiarisez-vous avec la façon dont les champs de données Vector Search 1.0 sont mappés à la nouvelle structure d'objet de données.
Créer une collection
Commencez par créer une collection avec un schéma de données qui reflète la structure de vos données Vector Search 1.0.
REST
curl -X POST \
'https://vectorsearch.googleapis.com/v1beta/projects/PROJECT_ID/locations/LOCATION/collections?collection_id=COLLECTION_ID' \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H 'Content-Type: application/json' \
-d '{
"data_schema": {
"type": "object",
"properties": {
"restricts": {
"type": "object",
"properties": {
"genres": {
"type": "array",
"items": {
"type": "string"
}
},
"director": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
"restricts_deny": {
"type": "object",
"properties": {
"genres": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
"numeric_restricts": {
"type": "object",
"properties": {
"year": {
"type": "integer"
},
"imdb_rating": {
"type": "number",
"format": "float"
}
}
},
"embedding_metadata": {
"type": "object",
"properties": {
"plot": {
"type": "string"
},
"customers_review_summary": {
"type": "string"
},
"critics_review_summary": {
"type": "string"
}
},
}
}
},
"vector_schema": {
"embedding": {
"dense_vector": {
"dimensions": 768
}
},
"sparse_embedding": {
"sparse_vector": {}
}
}
}'
Python
from google.cloud import vectorsearch_v1beta
# Create the client
vector_search_service_client = vectorsearch_v1beta.VectorSearchServiceClient()
# The JSON schema for the data
data_schema = {
"type": "object",
"properties": {
"restricts": {
"type": "object",
"properties": {
"genres": {"type": "array", "items": {"type": "string"}},
"director": {"type": "array", "items": {"type": "string"}},
},
},
"restricts_deny": {
"type": "object",
"properties": {
"genres": {"type": "array", "items": {"type": "string"}}
},
},
"numeric_restricts": {
"type": "object",
"properties": {
"year": {"type": "integer"},
"imdb_rating": {"type": "number", "format": "float"},
},
},
"embedding_metadata": {
"type": "object",
"properties": {
"plot": {"type": "string"},
"customers_review_summary": {"type": "string"},
"critics_review_summary": {"type": "string"},
},
},
},
}
# The JSON schema for the vector
vector_schema = {
"embedding": {"dense_vector": {"dimensions": 768}},
"sparse_embedding": {"sparse_vector": {}},
}
collection = vectorsearch_v1beta.Collection(
data_schema=data_schema,
vector_schema=vector_schema,
)
request = vectorsearch_v1beta.CreateCollectionRequest(
parent="projects/PROJECT_ID/locations/LOCATION",
collection_id="COLLECTION_ID",
collection=collection,
)
# Create the collection
operation = vector_search_service_client.create_collection(request=request)
operation.result()
Importer vos données Vector Search 1.0
Ensuite, utilisez l'API ImportDataObjects sur la collection que vous venez de créer.
Pointez-la vers le bucket Cloud Storage contenant vos données Vector Search 1.0.
REST
curl -X POST \
"https://vectorsearch.googleapis.com/v1beta/projects/PROJECT_ID/locations/LOCATION/collections/COLLECTION_ID:importDataObjects" \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{
"gcs_import": {
"contents_uri": "gs://your-bucket/path/to/your-data.jsonl",
"error_uri": "gs://your-bucket/path/to/import-errors/"
}
}'
Python
from google.cloud import vectorsearch_v1beta
# Create the client
vector_search_service_client = vectorsearch_v1beta.VectorSearchServiceClient()
# Initialize request
request = vectorsearch_v1beta.ImportDataObjectsRequest(
name="projects/PROJECT_ID/locations/LOCATION/collections/COLLECTION_ID",
gcs_import={
"contents_uri": "gs://your-bucket/path/to/your-data.jsonl",
"error_uri": "gs://your-bucket/path/to/import-errors/",
},
)
# Make the request
operation = vector_search_service_client.import_data_objects(request=request)
# Wait for the result (note this may take up to several minutes)
operation.result()
Transformation des données
Lors du processus d'importation, vos données Vector Search 1.0 seront transformées en objets de données de récupération d'agent. Les exemples suivants illustrent le mappage des champs.
Format de fichier Cloud Storage Vector Search 1.0
{
"id": "movie-789",
"embedding": [-0.23, 0.88, 0.11, ...],
"sparse_embedding": {"values": [0.1, 0.2], "dimensions": [1, 4]},
"restricts": [
{"namespace": "genres", "allow": ["science-fiction", "action"], "deny": ["horror"]},
{"namespace": "director", "allow": ["Christopher Nolan"]}
],
"numeric_restricts": [
{"namespace": "year", "value_int": 2010},
{"namespace": "imdb_rating", "value_float": 8.8}
],
"embedding_metadata": {
"plot": "...",
"customers_review_summary": "...",
"critics_review_summary": "..."
}
}
Objet de données de récupération d'agent transformé
DataObject(
name="/.../movie-789",
data={
"restricts": {
"genres": ["science-fiction", "action"],
"director": ["Christopher Nolan"],
},
"restricts_deny": {
"genres": ["horror"]
},
"numeric_restricts": {
"year": 2010,
"imdb_rating": 8.8,
},
"embedding_metadata": {
"plot": "...",
"customers_review_summary": "...",
"critics_review_summary": "...",
}
},
vectors={
"embedding": {"dense_vector": {"values": [-0.23, 0.88, 0.11, ...]}},
"sparse_embedding": {"sparse_vector": {"values": [0.1, 0.2], "indices": [1, 4]}},
}
)