When Airflow executes a task, it needs a log file to which the tasks' output is written. If Airflow is unable to find or connect to the log file, it fails the task. While the batch-ingest Airflow application always provides Airflow's scheduler with a log file, the Airflow scheduler may still fail to find the log file.
Occasionally, an Airflow task will fail and Airflow will claim that the log file for that task does not exist:
The problem is a known bug in Airflow. We are waiting for the Airflow team to fix this issue.
At present, the only way to resolve this problem is to clear the task and its downstream dependencies. This tells the Airflow scheduler to execute the tasks again.