Try BigQuery DataFrames

BigQuery DataFrames brings scalable Python analytics and machine learning (ML) to BigQuery. Computations execute in BigQuery with server-side processing, which lets you analyze and model large datasets without being constrained by local or notebook memory. You can use syntax similar to pandas (bigframes.pandas) and BigQuery ML (bigframes.bigquery) without writing SQL.

Use this quickstart to perform the following analysis and ML tasks by using the BigQuery DataFrames API in a BigQuery notebook:

  • Create a DataFrame over the bigquery-public-data.ml_datasets.penguins public dataset.
  • Calculate the average body mass of a penguin.
  • Clean and prepare a subset of the penguin data for training.
  • Train a linear regression model by using bigframes.bigquery.ml.create_model.
  • Evaluate the model by using bigframes.bigquery.ml.evaluate.

Before you begin

  1. In the Cloud de Confiance console, on the project selector page, select or create a Cloud de Confiance project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  2. Verify that billing is enabled for your Cloud de Confiance project.

  3. Verify that the BigQuery API is enabled.

    Enable the API

    If you created a new project, the BigQuery API is automatically enabled.

Required permissions

To create and run notebooks, you need the following Identity and Access Management (IAM) roles:

Create a notebook

Follow the instructions in Create a notebook from the BigQuery editor to create a new notebook.

Try BigQuery DataFrames

Try BigQuery DataFrames by following these steps:

  1. Create a new code cell in the notebook.

  2. Add the following code to the code cell:

    import bigframes.pandas as bpd

# Set BigQuery DataFrames options
# Note: The project option is not required in all environments.
# On BigQuery Studio, the project ID is automatically detected.
bpd.options.bigquery.project = your_gcp_project_id

# Use "partial" ordering mode to generate more efficient queries, but the
# order of the rows in DataFrames may not be deterministic if you have not
# explictly sorted it. Some operations that depend on the order, such as
# head() will not function until you explictly order the DataFrame. Set the
# ordering mode to "strict" (default) for more pandas compatibility.
bpd.options.bigquery.ordering_mode = "partial"

# Create a DataFrame from a BigQuery table
query_or_table = "bigquery-public-data.ml_datasets.penguins"
df = bpd.read_gbq(query_or_table)

# Efficiently preview the results using the .peek() method.
df.peek()

  3. Modify the bpd.options.bigquery.project = your_gcp_project_id line to specify your Cloud de Confiance project ID. For example, bpd.options.bigquery.project = "myProjectID".

  4. Run the code cell.

    The code returns a DataFrame object with data about penguins.

  5. Create a new code cell in the notebook and add the following code:

    # Use the DataFrame just as you would a pandas DataFrame, but calculations
# happen in the BigQuery query engine instead of the local system.
average_body_mass = df["body_mass_g"].mean()
print(f"average_body_mass: {average_body_mass}")

  6. Run the code cell.

    The code calculates the average body mass of the penguins and prints it to the Cloud de Confiance console.

  7. Create a new code cell in the notebook and add the following code:

    import bigframes.bigquery as bbq
from google.cloud import bigquery

# Ensure a dataset exists to store the model
client = bigquery.Client(project=bpd.options.bigquery.project)
client.create_dataset("bq_quickstart", exists_ok=True)

# Filter down to the Adelie Penguin species
adelie_data = df[df.species == "Adelie Penguin (Pygoscelis adeliae)"]

# Drop the columns that are not needed
adelie_data = adelie_data.drop(columns=["species"])

# Drop rows with nulls to get the training data
training_data = adelie_data.dropna()

# Train a linear regression model
model_name = f"{bpd.options.bigquery.project}.bq_quickstart.penguin_weight"
model_metadata = bbq.ml.create_model(
    model_name,
    replace=True,
    options={"model_type": "LINEAR_REG"},
    training_data=training_data.rename(columns={"body_mass_g": "label"}),
)

# Evaluate the model
evaluation = bbq.ml.evaluate(model_name)
print(evaluation)

  8. Run the code cell.

    The code trains the linear regression model directly in BigQuery and returns the model's evaluation metrics.

Clean up

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

  1. In the Cloud de Confiance console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next