400 “Bad Request” error when uploading multiple files to CE

Calculation Engine Support
,
1

Hi everyone,

I am encountering a ‘Bad request‘ error with status code 400 when trying to upload multiple files to CE (see attached code). Uploading a single file works without any issues, and the error only occurs when I attempt batch uploads.

Has anyone run into this before, or knows what might be causing it? Any pointers would be appreciated.

debug = True
models = ['crust_dft', 'cmf', 'ceft']

# container to store the uploaded file names and uuids 
upload_synthesis_with_hyperons = {}

# loop over the files to be uploaded
for key, uuid in filenames.items():
    print('Result:', key)
    value = 0
    upload_synthesis_with_hyperons[key] = {} 
    
    for model in models:
        file_path= (f'/home/ania/projects/muses/explore_parameter_space/output_tests/cmf_crust_outdir/v3/{key}/{uuid}/lepton-{model}/opt/output/beta_equilibrium_eos.csv')
        
        print('Check if file exists:', os.path.exists(file_path))

        upload_path = f'synthesis_hyperons_test/{key}/{model}/eos.csv'
        print('Upload path:', upload_path)
        
        if debug:
            print(f"Uploading {file_path} → {upload_path}")

        upload = api.upload_file(
            file_path=file_path,
            upload_path=upload_path,
            public=False
         )
        
        if upload:
            print(f"Uploaded EoS file with UUID: {upload['uuid']} and checksum: {upload['checksum']}")
        else:
            print(f"Failed to upload EoS file reason: {upload.reason}, status code: {upload.status_code}")

        # extract metadata
        print(upload)
        upload_uuid = upload.get('uuid')
        checksum   = upload.get('checksum')

        # # store in structured form
        upload_synthesis_with_hyperons[key][model] = {
            'upload_path': upload_path,
            'uuid': upload_uuid,
            'checksum': checksum
        }

# print(json.dumps(results, indent=2))

Output with the error message at the end:

Result: cmf_crust_with_hyperons_cmf_00060
Check if file exists: True
Upload path: synthesis_hyperons_test/cmf_crust_with_hyperons_cmf_00060/crust_dft/eos.csv
Uploading /home/ania/projects/muses/explore_parameter_space/output_tests/cmf_crust_outdir/v3/cmf_crust_with_hyperons_cmf_00060/7ab6b749-1c1b-4189-a042-0370b0c6d54a/lepton-crust_dft/opt/output/beta_equilibrium_eos.csv → synthesis_hyperons_test/cmf_crust_with_hyperons_cmf_00060/crust_dft/eos.csv
Failed to upload EoS file reason: Bad Request, status code: 400
<Response [400]>
2

Sorry you’re having trouble, @aliu. @mrpelicer asked me to look into this after troubleshooting with you, but I admit it is not obvious what is failing. The server logs do show the 400 errors, along with what look like be internal networking failures that might explain the behavior.

Does the failure occur on the same files repeatedly? Or is it random? Perhaps the upload loop is too fast… What happens if you introduce a 2 second delay after calling the upload API function at the end of each model iteration?

3

Hi @andrew.manning thank you for looking into this.

I tried adding a delay after calling the upload function, but the error still persists. I also tested removing the loop over models entirely and instead ran the function with a single model while looping only over the files, as well as having just one file in the `filenames` dictionary — this did not change the behaviour either.

I’ve tried several different files, and the error appears consistently.

As a workaround, I can try running the synthesis within the same workflow as the other jobs, so the files do not need to be uploaded separately and re-called. That said, it is intriguing why this behaviour occurs.

4

It is possible to save a job and reference it as input to future workflows precisely so you can avoid having to needlessly download and upload. Take a look at this documentation about workflow inputs.