Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

If you run into the following error during an airflow run:

Code Block
languagebash
titleError
psycopg2.errors.ConfigurationLimitExceeded: temporary file size exceeds temp_file_limit (#######kB)


You will need to increase the temp_file_limit flag on your batch-ingest Cloud SQL instance.

We recommend setting this value to at least 10% of the total disk size of your instance. Can the size of your disk in Gb by running the following command:

Code Block
languagebash
titleDisk size
gcloud sql instances describe batch-ingest --format="value(settings.dataDiskSizeGb)" --project "${GCP_PROJECT_ID}"


Info
titleExample

Ex: if your disk size was 50Gb then you would want to set temp_file_limit to at least 5000000 which is 5Gb


To set temp_file_limit run the following command:

Code Block
languagebash
titlePatch
gcloud sql instances patch batch-ingest --database-flags temp_file_limit=${SIZE_IN_KB} --project "${GCP_PROJECT_ID}"


Or in the console by navigating to https://console.cloud.google.com/sql/instances/batch-ingest/edit?project=${GCP_PROJECT_ID} and selecting Flags > temp_file_limit. Make sure to save your changes.