We’re encountering a bug where the volume mounted at /scratch is not being cleared properly between tasks, causing the system to eventually run out of space.
The error arises in a Celery task, where the system is trying to create a directory under /scratch using the Python os.makedirs(), but the volume has run out of space due to accumulated data from previous jobs. As a result, you might see your job fail with the following error:
OSError: [Errno 28] No space left on device: '/scratch/<job-id>'
We appreciate your patience as we address this issue.