CMF calculation never finishes

[melvin.storbacka]

Hi!

I ran a calculation with the CMF module, which failed during post-processing with the message:

 "Error: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))"

I have since tried rerunning the calculation, but it doesn’t seem to finish (even when run for > 5 times as long as the original one). Is there any way I can check how the process is doing by myself (access log files while it’s running, etc.), or understand why it’s not terminating?

Some questions that also arose: Is there a way to find out what the “saved” storage limit is? And what happens when this limit is reached (if it can be reached during a calculation), will my calculation just freeze?

But these questions shouldn’t really be an issue, unless that’s why it crashed before (both calculations were set to “saved” from the start).

[andrew.manning (T. Andrew Manning)]

@melvin.storbacka

Some questions that also arose: Is there a way to find out what the “saved” storage limit is?

This reminds me that we need to document this better. We have a related issue. If you open your usage limit page you will see some of the information you want.

You should delete jobs you no longer need to free storage space. The de facto Python API has a job_delete_all function that shows one way to do this.

@melvin.storbacka

And what happens when this limit is reached (if it can be reached during a calculation), will my calculation just freeze?

The job time limit is really long right now: 36 hours. The error message you are seeing is not what you should encounter if the job were aborted due to the time limit.

[andrew.manning (T. Andrew Manning)]

@melvin.storbacka

I have since tried rerunning the calculation, but it doesn’t seem to finish (even when run for > 5 times as long as the original one). Is there any way I can check how the process is doing by myself (access log files while it’s running, etc.), or understand why it’s not terminating?

Can you please share the log file (formatted as a code block) with the error, as well as the full workflow config?

[melvin.storbacka]

Great, thank you for your reply!

The entire log file is below, as well as the config file (modified from a version by Nikolas Cruz Camacho).

	
0	'Job "58eb4ebe-1152-4726-a7a6-3beb478fc0af": Celery task "0da236c5-a037-4445-af00-49974d14c924" triggered for task "cmf" running module "cmf_solver".'
1	"/usr/local/bin/python3 yaml_preprocess.py && ./cmf && /usr/local/bin/python3 clean_output.py && /usr/local/bin/python3 postprocess.py;"
2	"2025-08-22 20:47:26,052 - INFO - Starting execution of /opt/src/yaml_preprocess.py..."
3	"2025-08-22 20:47:26,239 - INFO - Simplified YAML configuration saved to /opt/src/../input/validated_config.yaml"
4	"2025-08-22 20:47:26,239 - INFO - Execution end of /opt/src/yaml_preprocess.py."
5	""
6	"\u001b[1;32mCMF++ execution start !!!\u001b[0m"
7	""
8	"\u001b[1;34m+---------------------------------------------------------------------------------------------------------------------+"
9	"|                                                                                                                     |"
10	"|        CCCCCCCCCCCCC  MMMMMMMM               MMMMMMMM  FFFFFFFFFFFFFFFFFFFFFF                                       |"
11	"|     CCC::::::::::::C  M:::::::M             M:::::::M  F::::::::::::::::::::F                                       |"
12	"|   CC:::::::::::::::C  M::::::::M           M::::::::M  F::::::::::::::::::::F                                       |"
13	"|  C:::::CCCCCCCC::::C  M:::::::::M         M:::::::::M  F::::::FFFFFFFFF:::::F       ++++++              ++++++      |"
14	"| C:::::C       CCCCCC  M::::::::::M       M::::::::::M    F:::::F       FFFFFF       +::::+              +::::+      |"
15	"|C:::::C                M:::::::::::M     M:::::::::::M    F:::::F                    +::::+              +::::+      |"
16	"|C:::::C                M:::::::M::::M   M::::M:::::::M    F::::::FFFFFFFFFF    +++++++::::+++++++  +++++++::::+++++++|"
17	"|C:::::C                M::::::M M::::M M::::M M::::::M    F:::::::::::::::F    +::::::::::::::::+  +::::::::::::::::+|"
18	"|C:::::C                M::::::M  M::::M::::M  M::::::M    F:::::::::::::::F    +::::::::::::::::+  +::::::::::::::::+|"
19	"|C:::::C                M::::::M   M:::::::M   M::::::M    F::::::FFFFFFFFFF    +++++++::::+++++++  +++++++::::+++++++|"
20	"|C:::::C                M::::::M    M:::::M    M::::::M    F:::::F                    +::::+              +::::+      |"
21	"| C:::::C       CCCCCC  M::::::M     MMMMM     M::::::M    F:::::F                    +::::+              +::::+      |"
22	"|  C:::::CCCCCCCC::::C  M::::::M               M::::::M  FF:::::::FF                  ++++++              ++++++      |"
23	"|   CC:::::::::::::::C  M::::::M               M::::::M  F::::::::FF                                                  |"
24	"|     CCC::::::::::::C  M::::::M               M::::::M  F::::::::FF                                                  |"
25	"|        CCCCCCCCCCCCC  MMMMMMMM               MMMMMMMM  FFFFFFFFFFF                                                  |"
26	"|                                                                                                                     |"
27	"|      ______     __  __     __     ______     ______     __                   ██    ██   ██     ██████     ██████    |"
28	"|     /\\  ___\\   /\\ \\_\\ \\   /\\ \\   /\\  == \\   /\\  __ \\   /\\ \\                  ██    ██  ███    ██  ████         ██   |"
29	"|     \\ \\ \\____  \\ \\  __ \\  \\ \\ \\  \\ \\  __<   \\ \\  __ \\  \\ \\ \\____             ██    ██   ██    ██ ██ ██     █████    |"
30	"|      \\ \\_____\\  \\ \\_\\ \\_\\  \\ \\_\\  \\ \\_\\ \\_\\  \\ \\_\\ \\_\\  \\ \\_____\\             ██  ██    ██    ████  ██         ██   |"
31	"|       \\/_____/   \\/_/\\/_/   \\/_/   \\/_/ /_/   \\/_/\\/_/   \\/_____/              ████     ██ ██  ██████  ██ ██████    |"
32	"|                                                                                                                     |"
33	"|            __    __     ______     ______     __   __                                                               |"
34	'|           /\\ "-./  \\   /\\  ___\\   /\\  __ \\   /\\ "-.\\ \\                                                              |'
35	"|           \\ \\ \\-./\\ \\  \\ \\  __\\   \\ \\  __ \\  \\ \\ \\-.  \\                                                             |"
36	'|            \\ \\_\\ \\ \\_\\  \\ \\_____\\  \\ \\_\\ \\_\\  \\ \\_\\\\"\\_\\                         __  __ _   _ ___ ___ ___           |'
37	"|             \\/_/  \\/_/   \\/_____/   \\/_/\\/_/   \\/_/ \\/_/                        |  \\/  | | | / __| __/ __|          |"
38	"|                                                                                 | |\\/| | |_| \\__ \\ _|\\__ \\          |"
39	"|                 ______   __     ______     __         _____                     |_|  |_|\\___/|___/___|___/          |"
40	"|                /\\  ___\\ /\\ \\   /\\  ___\\   /\\ \\       /\\  __-.                                                       |"
41	"|                \\ \\  __\\ \\ \\ \\  \\ \\  __\\   \\ \\ \\____  \\ \\ \\/\\ \\                                                      |"
42	"|                 \\ \\_\\    \\ \\_\\  \\ \\_____\\  \\ \\_____\\  \\ \\____-                                                      |"
43	"|                  \\/_/     \\/_/   \\/_____/   \\/_____/   \\/____/                                                      |"
44	"|                                                                                                                     |"
45	"+---------------------------------------------------------------------------------------------------------------------+"
46	"\u001b[0m"
47	"\u001b[1;32mUsing ../input/validated_config.yaml as input file\u001b[0m"
48	""
49	"\u001b[1;32mUsing ../output as output directory\u001b[0m"
50	""
51	"\u001b[1;32mUsing ./PDG/PDG2021Plus_massorder.dat as PDG table\u001b[0m"
52	""
53	"\u001b[1;32mUsing ./PDG/PDG2021Plus_quarks.dat as PDG quark table\u001b[0m"
54	""
55	"Input parameters for this run:"
56	"run_name:\tC4_default_muB_muS_muQ_test_coarse_30"
57	"use_ideal_gas:\t0"
58	"use_quarks:\t0"
59	"use_octet:\t1"
60	"use_decuplet:\t1"
61	"use_pure_glue:\t0"
62	"use_hyperons:\t1"
63	"use_constant_sigma_mean_field:\t0"
64	"use_constant_zeta_mean_field:\t0"
65	"use_constant_delta_mean_field:\t0"
66	"use_constant_omega_mean_field:\t0"
67	"use_constant_phi_mean_field:\t0"
68	"use_constant_rho_mean_field:\t0"
69	"use_Phi_order :\t1"
70	"use_constant_Phi_order_field :\t0"
71	"output_debug:\t0"
72	"use_default_vector_couplings:\t1"
73	"output_flavor_equilibration:\t1"
74	"output_particle_properties:\t1"
75	"use_fine_grid:\t0"
76	"vector_potential:\t4"
77	"baryon_mass_coupling:\t1"
78	"V_Delta:\t1.2"
79	"solution_resolution:\t1e-15"
80	"maximum_for_residues:\t0.0001"
81	"muB_begin:\t600"
82	"muB_end:\t2000"
83	"muB_step:\t10"
84	"muQ_begin:\t-750"
85	"muQ_end:\t1000"
86	"muQ_step:\t10"
87	"muS_begin:\t-650"
88	"muS_end:\t1000"
89	"muS_step:\t10"
90	"muB1_begin:\t900"
91	"muB1_end:\t1000"
92	"muB1_step:\t2"
93	"muQ1_begin:\t0"
94	"muQ1_end:\t10"
95	"muQ1_step:\t100"
96	"muS1_begin:\t0"
97	"muS1_end:\t10"
98	"muS1_step:\t100"
99	"sigma0_begin:\t-100"
100	"sigma0_end:\t-9"
101	"sigma0_step:\t45"
102	"zeta0_begin:\t-110"
103	"zeta0_end:\t-9"
104	"zeta0_step:\t50"
105	"delta0_begin:\t-30"
106	"delta0_end:\t31"
107	"delta0_step:\t30"
108	"omega0_begin:\t0"
109	"omega0_end:\t101"
110	"omega0_step:\t100"
111	"phi0_begin:\t-100"
112	"phi0_end:\t1"
113	"phi0_step:\t100"
114	"rho0_begin:\t-50"
115	"rho0_end:\t51"
116	"rho0_step:\t50"
117	"Phi_order0_begin:\t0"
118	"Phi_order0_end:\t1"
119	"Phi_order0_step:\t10"
120	"d_betaQCD:\t0.0606061"
121	"f_pi:\t93.3"
122	"f_K:\t122"
123	"gqu_sigma:\t-3"
124	"gqu_zeta:\t0"
125	"gqu_delta:\t0"
126	"gqu_omega:\t0"
127	"gqu_phi:\t0"
128	"gqu_rho:\t0"
129	"gqd_sigma:\t-3"
130	"gqd_zeta:\t0"
131	"gqd_delta:\t0"
132	"gqd_omega:\t0"
133	"gqd_phi:\t0"
134	"gqd_rho:\t0"
135	"gqs_sigma:\t0"
136	"gqs_zeta:\t-3"
137	"gqs_delta:\t0"
138	"gqs_omega:\t0"
139	"gqs_phi:\t0"
140	"gqs_rho:\t0"
141	"gQ_Phi_order :\t500"
142	"gB_Phi_order :\t1500"
143	"pion_vacuum_mass:\t139"
144	"kaon_vacuum_mass:\t498"
145	"nucleon_vacuum_mass:\t937.243"
146	"Lambda_vacuum_mass:\t1115"
147	"Sigma_vacuum_mass:\t1202"
148	"Delta_vacuum_mass:\t1232"
149	"Sigma_star_vacuum_mass:\t1385"
150	"Omega_vacuum_mass:\t1691"
151	"up_quark_bare_mass:\t5"
152	"down_quark_bare_mass:\t5"
153	"strange_quark_bare_mass:\t150"
154	"m_1:\t0"
155	"m_2:\t0"
156	"m_3H:\t0.859146"
157	"m_3D:\t1.25"
158	"k_0:\t2.37322"
159	"k_1:\t1.4"
160	"k_2:\t-5.54911"
161	"k_3:\t-2.65242"
162	"a_1:\t-0.001443"
163	"a_3:\t-0.396"
164	"T0:\t200"
165	"g_X1:\t-7.32378"
166	"g_X8:\t-2.33897"
167	"alpha_X:\t1.44834"
168	"alpha_DX:\t0.350848"
169	"gN_sigma:\t-10.5668"
170	"gN_zeta:\t0.467039"
171	"g_4:\t38.9"
172	"gN_omega:\t11.9"
173	"gN_phi:\t0"
174	"gN_rho:\t4.03"
175	"mass0:\t150"
176	"sigma_mean_field_vacuum_mass:\t-93.3"
177	"zeta_mean_field_vacuum_mass:\t-106.561"
178	"omega_mean_field_vacuum_mass:\t780.563"
179	"phi_mean_field_vacuum_mass:\t1019"
180	"rho_mean_field_vacuum_mass:\t761.063"
181	"chi_mean_field_vacuum_value:\t401.934"
182	"hbarc3:\t7.68351e+06"
183	""
184	"\u001b[1;32mOutput directory ../output/C4_default_muB_muS_muQ_test_coarse_30/ created successfully\u001b[0m"
185	""
186	"\u001b[1;32mFile ../input/validated_config.yaml copied to ../output/C4_default_muB_muS_muQ_test_coarse_30/\u001b[0m"
187	""
188	"Points for initial guessess:"
189	""
190	"Baryon chemical potential \t141"
191	"Strangeness chemical potential \t166"
192	"Charge chemical potential \t176"
193	""
194	"Points per field:"
195	""
196	"sigma \t3"
197	"zeta \t3"
198	"delta \t3"
199	"omega \t2"
200	"phi \t2"
201	"rho \t3"
202	"Phi order \t1"
203	""
204	"Total initial points: \t1334703744"
205	""
206	"Computing solutions, please wait..."
207	""
208	"Creating coupling constants"
209	""
210	"Creating bare masses"
211	""
212	"Initializing Baryon octet"
213	""
214	"Initializing Baryon decuplet"
215	"[..................................................]   0%  Progress:        0 / 1334703744            \r[=.................................................]   1%  Progress: 13347037 / 1334703744 

[Skipped because of character limit]

          \r[=================================================.]  98%  Progress: 1308009626 / 1334703744            \r[=================================================.]  99%  Progress: 1321356663 / 1334703744            \r[==================================================] 100%  Progress: 1334703700 / 1334703744"
216	"\u001b[1;32mCMF++ successful execution complete !!!\u001b[0m"
217	"2025-08-23 19:14:23,153 - INFO - Starting execution of /opt/src/clean_output.py..."
218	"2025-08-23 19:14:23,163 - INFO - Loaded configuration from /opt/src/../input/validated_config.yaml"
219	"2025-08-23 19:14:23,163 - INFO - Running for folder: /opt/src/../output/C4_default_muB_muS_muQ_test_coarse_30"
220	"2025-08-23 19:14:23,163 - INFO - Starting cleaning of /opt/src/../output/C4_default_muB_muS_muQ_test_coarse_30/CMF_intermediate_output.csv..."
221	"2025-08-23 19:15:37,080 - INFO - Dimensions: ['B', 'S', 'Q']"
222	"2025-08-23 19:15:37,115 - INFO - Number of unique muB: 141"
223	"2025-08-23 19:15:37,150 - INFO - Number of unique muS: 166"
224	"2025-08-23 19:15:37,186 - INFO - Number of unique muQ: 176"
225	"2025-08-23 19:15:37,190 - INFO - Filtering stable points in 1D..."
226	"2025-08-23 19:15:37,190 - INFO - Processing along muB..."
227	"/opt/src/clean_output.py:1156: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation."
228	"combined_unstable_muB = pd.concat(combined_unstable)"
229	"2025-08-23 21:51:33,385 - INFO - Processing along muS..."
230	"2025-08-23 22:14:07,870 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
231	"2025-08-23 22:14:13,773 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
232	"2025-08-23 22:14:37,427 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
233	"2025-08-23 22:14:43,234 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
234	"2025-08-23 22:14:49,124 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
235	"2025-08-23 22:15:23,350 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
236	"2025-08-23 22:15:40,228 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
237	"2025-08-23 22:15:58,536 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
238	"2025-08-23 22:16:37,716 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
239	"2025-08-23 22:16:49,156 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
240	"2025-08-23 22:17:06,217 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
241	"2025-08-23 22:17:17,328 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
242	"2025-08-23 22:17:22,817 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
243	"/opt/src/clean_output.py:1199: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation."
244	"combined_stable_muS = pd.concat(combined_stable)"
245	"2025-08-23 22:17:48,568 - INFO - Processing along muQ..."
246	"2025-08-23 22:29:40,052 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
247	"2025-08-23 22:29:40,085 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
248	"2025-08-23 22:29:40,118 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
249	"2025-08-23 22:29:40,153 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
250	"2025-08-23 22:29:40,186 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
251	"2025-08-23 22:29:40,221 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
252	"2025-08-23 22:29:40,253 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
253	"2025-08-23 22:29:40,286 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
254	"2025-08-23 22:29:40,320 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
255	"2025-08-23 22:29:40,353 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
256	"2025-08-23 22:29:40,386 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
257	"2025-08-23 22:29:40,419 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
258	"2025-08-23 22:29:40,452 - WARNING - Error computing derivative: index 0 is out of bounds for axis 0 with size 0"
259	"/opt/src/clean_output.py:1242: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation."
260	"combined_stable_muQ = pd.concat(combined_stable)"
261	"/opt/src/clean_output.py:1248: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation."
262	"combined_unstable_muQ = pd.concat(combined_unstable)"
263	"2025-08-23 22:30:00,322 - INFO - Filtering stable points in 2D..."
264	"2025-08-23 22:30:00,322 - INFO - computing chi2_B"
265	"/opt/src/clean_output.py:388: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
266	".apply(lambda group: compute_chi2_for_X(group, X=X))"
267	"/opt/src/clean_output.py:388: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
268	".apply(lambda group: compute_chi2_for_X(group, X=X))"
269	"2025-08-23 22:30:28,330 - INFO - computing chi2_S"
270	"/opt/src/clean_output.py:388: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
271	".apply(lambda group: compute_chi2_for_X(group, X=X))"
272	"/opt/src/clean_output.py:388: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
273	".apply(lambda group: compute_chi2_for_X(group, X=X))"
274	"2025-08-23 22:30:50,515 - INFO - computing chi2_Q"
275	"/opt/src/clean_output.py:388: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
276	".apply(lambda group: compute_chi2_for_X(group, X=X))"
277	"/opt/src/clean_output.py:388: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
278	".apply(lambda group: compute_chi2_for_X(group, X=X))"
279	"2025-08-23 22:31:11,591 - INFO - computing chi11_BS"
280	"/opt/src/clean_output.py:432: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
281	".apply(lambda group: compute_chi11_for_XY(group, X=X, Y=Y))"
282	"/opt/src/clean_output.py:432: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
283	".apply(lambda group: compute_chi11_for_XY(group, X=X, Y=Y))"
284	"2025-08-23 22:31:35,016 - INFO - computing chi11_SQ"
285	"/opt/src/clean_output.py:432: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
286	".apply(lambda group: compute_chi11_for_XY(group, X=X, Y=Y))"
287	"/opt/src/clean_output.py:432: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
288	".apply(lambda group: compute_chi11_for_XY(group, X=X, Y=Y))"
289	"2025-08-23 22:31:57,248 - INFO - computing chi11_QB"
290	"/opt/src/clean_output.py:432: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
291	".apply(lambda group: compute_chi11_for_XY(group, X=X, Y=Y))"
292	"/opt/src/clean_output.py:432: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning."
293	".apply(lambda group: compute_chi11_for_XY(group, X=X, Y=Y))"
294	"2025-08-23 22:32:21,954 - INFO - Filtering stable points in 3D..."
295	"2025-08-23 22:34:41,731 - INFO - Saved /opt/src/../output/C4_default_muB_muS_muQ_test_coarse_30/CMF_output_stable.csv"
296	"2025-08-23 22:34:41,777 - INFO - Saved /opt/src/../output/C4_default_muB_muS_muQ_test_coarse_30/CMF_output_metastable.csv"
297	"2025-08-23 22:36:55,285 - INFO - Saved /opt/src/../output/C4_default_muB_muS_muQ_test_coarse_30/CMF_output_unstable.csv"
298	"2025-08-23 22:36:55,521 - INFO - Saved /opt/src/../output/C4_default_muB_muS_muQ_test_coarse_30/CMF_output_quarks.csv"
299	"2025-08-23 22:39:04,738 - INFO - Saved /opt/src/../output/C4_default_muB_muS_muQ_test_coarse_30/CMF_output_baryons.csv"
300	"2025-08-23 22:39:14,202 - INFO - Saved /opt/src/../output/C4_default_muB_muS_muQ_test_coarse_30/CMF_output_vacuum.csv"
301	"2025-08-23 22:39:14,223 - INFO - Starting cleaning of /opt/src/../output/C4_default_muB_muS_muQ_test_coarse_30/CMF_intermediate_output_particle_properties.csv..."
302	"Error: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))"

Config:

processes:
- name: cmf
  module: cmf_solver
  config:
    computational_parameters:
      constant_fields:
        use_constant_Phi_order_field: false
        use_constant_delta_mean_field: false
        use_constant_omega_mean_field: false
        use_constant_phi_mean_field: false
        use_constant_rho_mean_field: false
        use_constant_sigma_mean_field: false
        use_constant_zeta_mean_field: false
      maximum_for_residues: 0.0001
      options:
        baryon_mass_coupling: 1
        use_Phi_order: true
        use_decuplet: true
        use_default_vector_couplings: true
        use_fine_grid: false
        use_hyperons: true
        use_ideal_gas: false
        use_octet: true
        use_pure_glue: false
        use_quarks: false
        vector_potential: 4
      output_files:
        output_Lepton: true
        output_debug: false
        output_flavor_equilibration: true
        output_format: CSV
        output_particle_properties: true
      production_run: false
      run_name: default_muB_muS_muQ_test_coarse_30
      solution_resolution: 1.0e-15
      variables:
        chemical_optical_potentials:
          muB_begin: 600.0
          muB_end: 2000.0
          muB_step: 10.0
          muQ_begin: -750.0
          muQ_end: 1000.0
          muQ_step: 10.0
          muS_begin: -650.0
          muS_end: 1000.0
          muS_step: 10.0
        mean_fields_and_Phi_field:
          Phi_order0_begin: 0.0
          Phi_order0_end: 0.9999
          Phi_order0_step: 0.333
          delta0_begin: -30.0
          delta0_end: 31.0
          delta0_step: 30.0
          omega0_begin: 0.0
          omega0_end: 101.0
          omega0_step: 100.0
          phi0_begin: -100.0
          phi0_end: 1.0
          phi0_step: 100.0
          rho0_begin: -50.0
          rho0_end: 51.0
          rho0_step: 50.0
          sigma0_begin: -100.0
          sigma0_end: -9.0
          sigma0_step: 45.0
          zeta0_begin: -110.0
          zeta0_end: -9.0
          zeta0_step: 50.0
    physical_parameters:
      Phi_order_optical_potential:
        T0: 200.0
        T0_gauge: 270.0
        a_1: -0.001443
        a_3: -0.396
      baryon_to_Phi_field_coupling:
        gB_Phi_order: 1500.0
      chi_mean_field_vacuum_value: 401.933763
      d_betaQCD: 0.0606060606
      explicit_symmetry_breaking:
        V_Delta: 0.0
        m_1: 0.0
        m_2: 0.0
        m_3D: 1.25
        m_3H: 0.0
      f_K: 122.0
      f_pi: 93.3000031
      hbarc: 197.3269804
      mean_field_vacuum_masses:
        omega_mean_field_vacuum_mass: 780.562988
        phi_mean_field_vacuum_mass: 1019.0
        rho_mean_field_vacuum_mass: 761.062988
      quark_bare_masses:
        down_quark_bare_mass: 5.0
        strange_quark_bare_mass: 150.0
        up_quark_bare_mass: 5.0
      quark_to_fields_couplings:
        gQ_Phi_order: 500.0
        gqd_delta: 0.0
        gqd_omega: 0.0
        gqd_phi: 0.0
        gqd_rho: 0.0
        gqd_sigma: -3.0
        gqd_zeta: 0.0
        gqs_delta: 0.0
        gqs_omega: 0.0
        gqs_phi: 0.0
        gqs_rho: 0.0
        gqs_sigma: 0.0
        gqs_zeta: -3.0
        gqu_delta: 0.0
        gqu_omega: 0.0
        gqu_phi: 0.0
        gqu_rho: 0.0
        gqu_sigma: -3.0
        gqu_zeta: 0.0
      scalar_mean_field_equation:
        k_0: 2.3732188
        k_1: 1.39999998
        k_2: -5.54911336
        k_3: -2.65241888
      scalar_nucleon_couplings:
        alpha_X: 1.44833948
        gN_sigma: -10.5668
        gN_zeta: 0.467039
      vacuum_masses:
        Delta_vacuum_mass: 1232.0
        Lambda_vacuum_mass: 1115.0
        Omega_vacuum_mass: 1691.0
        Sigma_star_vacuum_mass: 1385.0
        Sigma_vacuum_mass: 1202.0
        kaon_vacuum_mass: 498.0
        mass0: 150.0
        nucleon_vacuum_mass: 937.242981
        pion_vacuum_mass: 139.0
      vector_nucleon_couplings:
        gN_omega: 11.9
        gN_phi: 0.0
        gN_rho: 4.03
        g_4: 38.9
- name: lepton-baryons
  module: lepton
  config:
    global:
      run_name: ''
      use_beta_equilibrium: false
      use_charge_neutrality: true
      check_eos_stability: true
      verbose: 3
    output:
      output_derivatives: true
      output_hdf5: false
      output_compOSE: false
      output_particle_properties: true
    particles:
      use_electron: true
      use_muon: false
  pipes:
    input_eos:
      label: CMF_for_Lepton_baryons_only
      module: cmf_solver
      process: cmf
    input_particle_properties:
      label: CMF_particle_properties_baryons_only
      module: cmf_solver
      process: cmf
components:
- type: chain
  name: workflow
  sequence:
  - cmf
  - lepton-baryons

[andrew.manning (T. Andrew Manning)]

You are probably going to need help from someone like @cnc6 to debug this. When I run your workflow (with the benefit of having access to the worker logs), the CMF module rapidly reaches a point where it appears frozen with no other output after logging “Initializing Baryon decuplet”:

CMF++ execution start !!!
...
Output directory ../output/C4_default_muB_muS_muQ_test_coarse_30/ created successfully

File ../input/validated_config.yaml copied to ../output/C4_default_muB_muS_muQ_test_coarse_30/

Points for initial guessess:

Baryon chemical potential 	141
Strangeness chemical potential 	166
Charge chemical potential 	176

Points per field:

sigma 	3
zeta 	3
delta 	3
omega 	2
phi 	2
rho 	3
Phi order 	1

Total initial points: 	1334703744

Computing solutions, please wait...

Creating coupling constants

Creating bare masses

Initializing Baryon octet

Initializing Baryon decuplet

[melvin.storbacka]

Thank you, that’s interesting… Interestingly as well, when running a more coarse grid, with the same start / end points but steps of 30 in chemical potential, everything works and finished in a couple of hours.