This guide will walk you through setting up BigQuery as your store of records in Cargo. This setup ensures Cargo has the necessary permissions in BigQuery to read and write data efficiently.

​
Permissions

What Cargo can do
  • Read data from datasets and tables, even if they are spread across multiple projects
  • Write data into new datasets and tables
What Cargo will never do
  • Overwrite existing datasets and tables (Cargo always creates its own datasets and tables when needed)

​
Before you begin

To start, you need an existing Google Cloud Project with a payment method and billing enabled. Follow the official Google guide. Once you have created the project, you can continue with this guide which will cover enabling and creating the necessary elements in your new GCP project:
  • BigQuery API & Cloud Resource Manager API
  • BigQuery dataset (dedicated to Cargo)
  • Object Storage Bucket (dedicated to Cargo)
  • Service Account
If you have an existing BigQuery project and technical knowledge, you may skip any of these steps. Enable the necessary APIs Cargo uses two Google APIs that must be enabled. To do so: 
<ul>
  <li>Go to the Google Cloud Console.</li>
  <li>Select `APIs & Services`.</li>
  <li>Select `Enabled APIs & Services`.</li>
  <li>Search for and enable the following APIs: `BigQuery API`, `Cloud Resource Manager API`</li>
</ul>
Create a storage bucket
To enable Cargo to load and unload data from BigQuery, we need a dedicated storage bucket for this purpose. To create a new bucket:
  • Go to the Google Cloud Console.
  • Search Object storage in the search bar.
  • Create a new bucket and follow the steps.

​
Step 1: Create a dedicated dataset

Create a dedicated dataset for Cargo where all data managed by Cargo will be stored. Create a dataset called “cargo_dataset” You can create this through the BigQuery console or using the following command: 
bq mk --dataset your-project-id:cargo_dataset
If through the console, follow these steps:
  • Go to the Google Cloud Console
  • Search BigQuery in the search bar
  • In BigQuery Studio click three dots next to the project name
  • Click Create dataset

​
Step 2: Create a service account for Cargo

Grant the necessary permissions for Cargo to run commands as an authenticated service account on the dataset you just created. Create a service account 
gcloud iam service-accounts create cargo-service-account \
    --display-name="Cargo Service Account"
Grant BigQuery permissions to the service account 
gcloud projects add-iam-policy-binding your-project-id \
    --member="serviceAccount:cargo-service-account@your-project-id.iam.gserviceaccount.com" \
    --role="roles/bigquery.dataEditor"

gcloud projects add-iam-policy-binding your-project-id \
    --member="serviceAccount:cargo-service-account@your-project-id.iam.gserviceaccount.com" \
    --role="roles/bigquery.jobUser"
Create and download a key file 
gcloud iam service-accounts keys create cargo-key.json \
    --iam-account=cargo-service-account@your-project-id.iam.gserviceaccount.com
If in the console, follow the steps below:
  • Go to the Google Cloud Console
  • Click on IAM & Admin
  • Click on IAM
  • Click on Service Accounts
  • Click on Create service account
  • Give the service account a name
  • Grant the following roles: BigQuery Data Editor, BigQuery Job User, Storage Object User
  • Click on Done

​
Step 3: Grant additional permissions

If you need Cargo to access data outside the dedicated dataset, grant additional permissions: Grant read access to other datasets 
bq add-iam-policy-binding \
    --member="serviceAccount:cargo-service-account@your-project-id.iam.gserviceaccount.com" \
    --role="roles/bigquery.dataViewer" \
    your-project-id:other_dataset

​
Step 4: Verify permissions

Make sure the Cargo service account has access to the following permissions:
  • bigquery.datasets.create
  • bigquery.tables.create
  • bigquery.tables.getData
  • bigquery.tables.updateData
  • bigquery.jobs.create
Check service account permissions 
gcloud projects get-iam-policy your-project-id \
    --flatten="bindings[].members" \
    --format='table(bindings.role)' \
    --filter="bindings.members:cargo-service-account@your-project-id.iam.gserviceaccount.com"

​
Step 5: Setup system of records

Now that we have all required elements, navigate to workspace settings and select “System of records”. Fill in the settings form with the data we gathered in previous steps:
  • Copy and paste the content of the service account key file into the field labeled Service Account
  • Select the location that was chosen during step 3
  • Fill in the name of the bucket created in step 2
  • Select Dataset as scope
  • Fill in the name of the BigQuery dataset created in step 3
  • Click Setup

​
Next steps

  • Create your first data model using BigQuery tables
  • Set up data connectors to import data from other sources
  • Configure model relationships to connect different datasets
  • Set up filters and segments for targeted plays