If you run into the following error during an airflow run:

Error
psycopg2.errors.ConfigurationLimitExceeded: temporary file size exceeds temp_file_limit (#######kB)


You will need to increase the temp_file_limit flag on your batch-ingest Cloud SQL instance.

We recommend setting this value to at least 10% of the total disk size of your instance. Get the size of your disk in Gb by running the following command:

Get disk size
gcloud sql instances describe batch-ingest --format="value(settings.dataDiskSizeGb)" --project "${GCP_PROJECT_ID}"

Example

If your disk size was 50Gb then you would want to set temp_file_limit to at least 5000000 which is 5Gb


To set temp_file_limit run the following command:

Set temp file limit
gcloud sql instances patch batch-ingest --database-flags temp_file_limit=${SIZE_IN_KB} --project "${GCP_PROJECT_ID}"

Or in the console by navigating to https://console.cloud.google.com/sql/instances/batch-ingest/edit?project=${GCP_PROJECT_ID} and selecting Flags > temp_file_limit. Make sure to save your changes.

  • No labels