HammerCloud | ATLAS rss

Administration
<< Back

PanDA incidents history

The following table summarizes the incidents happened in HammerCloud tests. You can filter the results using the form below and clicking in 'Refresh'.


Incidents for last 72 hours

TimestampSeverityCommentTest
Tue 12 May 2026 01:50:31warningThe following sites need more test jobs for the ARM template 1367: CERN-ARM 20331997 »
Tue 12 May 2026 01:50:31warningThe following sites need more test jobs for the AFT template 1368: ANALY_BNL_VP, LUMI, UNIGE-BAOBAB 20331987 »
Tue 12 May 2026 01:50:31warningThe following sites need more test jobs for the AFT template 1369: ANALY_BNL_VP, LUMI, UNIGE-BAOBAB 20331987 »
Tue 12 May 2026 01:50:31warningThe following sites need more test jobs for the PFT template 1370: CERN-P1, DE-TARDIS, LRZ-LMU_TEST, NCG-INGRID-PT, UKI-SCOTGRID-ECDF_CLOUD, UKI-SOUTHGRID-BHAM-HEP, UNIGE-BAOBAB 20331996 »
Tue 12 May 2026 01:50:31warningThe following sites need more test jobs for the PFT template 1371: DE-TARDIS, LRZ-LMU_TEST, NCG-INGRID-PT, SiGNET-NSC_MCORE, UKI-SCOTGRID-ECDF_CLOUD, UKI-SOUTHGRID-BHAM-HEP, UNIGE-BAOBAB 20331996 »
Tue 12 May 2026 01:50:31warningThe following sites need more test jobs for the GPU template 1366: FZK-LCG2_GPU, UKI-SOUTHGRID-RALPP_GPU 20331973 »
Tue 12 May 2026 01:50:31warningThe following sites need more test jobs for the PFT_MCORE template 1372: UKI-SOUTHGRID-SUSX_TEST 20331983 »
Tue 12 May 2026 01:50:01warningVega_GPU->Vega_GPU: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7136216755 ==> modificationHost: aipanda403.cern.ch pilot:::9000 Unknown error ddm:::200 expected output f55a5669-0d19-42d3-9e91-b4ae885a51a0_52972.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7136152908 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 61a71925-7807-4a52-8f5a-22716d1399a7_46371.job.log.tgz is missing in pilot JSON 20331973 »
Tue 12 May 2026 01:50:01warningUNIGE-BAOBAB->UNIGE-BAOBAB: Auto-inclusion policy Last-Two-From-All not passed. Site needs jobs for all templates (1369, 1368, 1371, 1370) 20331987 »
Tue 12 May 2026 01:50:01warningUM6P->UM6P: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7136771472 ==> modificationHost: slurm-compute-h22b5-u35-svn2 pilot:::9000 Failed in data staging: Transfer timed out;Failed in data staging: Transfer timed out ddm:::200 expected output 10b537c5-f7db-42a4-8c12-01893a97173e_66326.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7136736477 ==> modificationHost: slurm-compute-h21c8-u12-svn3 pilot:::9000 Failed in data staging: Transfer timed out;Failed in data staging: Transfer timed out ddm:::200 expected output b9212d2f-3f05-4b32-a9c4-5a3bf7de56ae_9419.1.job.log.tgz is missing in pilot JSON 20331983 »
Tue 12 May 2026 01:50:01warningUKI-NORTHGRID-LIV-HEP->UKI-NORTHGRID-LIV-HEP: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7136886956 ==> modificationHost: slot1_22@r26-n10.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.84881e5f-e16f-4cba-963b-fee98cd3974a_57592.pool.root to UKI-NORTHGRID-LIV-HEP_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136849674 ==> modificationHost: slot1_10@r26-n20.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:hc_20331979.UKI-NORTHGRID-LIV-HEP.35298d79-75cd-4666-ba57-d4436ddd9ba2_44194.hist-out.root to UKI-NORTHGRID-LIV-HEP_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136892898 ==> modificationHost: slot1_6@r25-n01.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.3999a512-6bef-400c-87ee-fb1fc98cb1ee_70761.pool.root to UKI-NORTHGRID-LIV-HEP_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136850590 ==> modificationHost: slot1_31@r26-n19.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:hc_20331987.UKI-NORTHGRID-LIV-HEP.baf5fd14-c6a8-4828-830b-fa0132dada74_29729.hist-out.root to UKI-NORTHGRID-LIV-HEP_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136864307 ==> modificationHost: slot1_23@r26-n20.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:output.hc_20331996.UKI-NORTHGRID-LIV-HEP.b0e808a1-37dc-4af7-8991-c52116f5bc4d_63698.EVNT.root to UKI-NORTHGRID-LIV-HEP_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136813081 ==> modificationHost: slot1_22@r25-n03.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:output.hc_20331996.UKI-NORTHGRID-LIV-HEP.8260f6fb-b2c1-4dd8-9c07-af11bc9860da_30951.EVNT.root to UKI-NORTHGRID-LIV-HEP_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136884678 ==> modificationHost: slot1_30@r11-n11.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.ee6b9259-0c06-4294-9c03-0e76ec41c4c6_12154.pool.root to UKI-NORTHGRID-LIV-HEP_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136850328 ==> modificationHost: slot1_5@r26-n13.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:Hits.hc_20331981.UKI-NORTHGRID-LIV-HEP.8318f8ef-e887-4d33-8857-d385aeee9b51_55122.HITS.pool.root to UKI-NORTHGRID-LIV-HEP_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] 20331987 »
Tue 12 May 2026 01:50:01warningPUHTI->PUHTI: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7136655149 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 5f6df664-5e6a-427a-a1dd-6e1de1ab1156_61532.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7136586865 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 043869e6-bdae-4a7a-bcca-2e2f3e544ef6_66429.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7136625420 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output bd7f274b-bfa9-4f35-b9f3-780479b853ba_36083.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7136654309 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 19cdc157-f252-408e-9f42-81706effd237_50235.1.job.log.tgz is missing in pilot JSON 20331996 »
Tue 12 May 2026 01:50:01warningAM-01-AANL->AM-01-AANL: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7136859399 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output b5ffeed6-7cdc-4d8e-bf8b-fb874618a276_16535.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7136863592 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 2899578f-2e55-41ed-8c25-3baacf214ba1_35551.1.job.log.tgz is missing in pilot JSON 20331983 »
Tue 12 May 2026 01:50:01blacklistingChecking these sites currently test because of HC: ['AM-01-AANL', 'PUHTI', 'UKI-NORTHGRID-LIV-HEP', 'UM6P', 'UNIGE-BAOBAB', 'Vega_GPU']  
Tue 12 May 2026 01:35:33warningThe following sites need more test jobs for the PFT template 1370: CERN-P1, DE-TARDIS, LRZ-LMU_TEST, NCG-INGRID-PT, UKI-SCOTGRID-ECDF_CLOUD, UKI-SOUTHGRID-BHAM-HEP, UNIGE-BAOBAB 20331996 »
Tue 12 May 2026 01:35:33warningThe following sites need more test jobs for the PFT template 1371: DE-TARDIS, LRZ-LMU_TEST, NCG-INGRID-PT, SiGNET-NSC_MCORE, UKI-SCOTGRID-ECDF_CLOUD, UKI-SOUTHGRID-BHAM-HEP, UNIGE-BAOBAB 20331996 »
Tue 12 May 2026 01:35:33warningThe following sites need more test jobs for the GPU template 1366: FZK-LCG2_GPU, UKI-SOUTHGRID-RALPP_GPU 20331973 »
Tue 12 May 2026 01:35:33warningThe following sites need more test jobs for the ARM template 1367: CERN-ARM 20331997 »
Tue 12 May 2026 01:35:33warningThe following sites need more test jobs for the PFT_MCORE template 1372: UKI-SOUTHGRID-SUSX_TEST 20331983 »
Tue 12 May 2026 01:35:33warningThe following sites need more test jobs for the AFT template 1368: ANALY_BNL_VP, LUMI, UNIGE-BAOBAB 20331987 »
Tue 12 May 2026 01:35:33warningThe following sites need more test jobs for the AFT template 1369: ANALY_BNL_VP, LUMI, UNIGE-BAOBAB 20331987 »
Tue 12 May 2026 01:35:01warningVega_GPU->Vega_GPU: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7136216755 ==> modificationHost: aipanda403.cern.ch pilot:::9000 Unknown error ddm:::200 expected output f55a5669-0d19-42d3-9e91-b4ae885a51a0_52972.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7136152908 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 61a71925-7807-4a52-8f5a-22716d1399a7_46371.job.log.tgz is missing in pilot JSON 20331973 »
Tue 12 May 2026 01:35:01warningUNIGE-BAOBAB->UNIGE-BAOBAB: Auto-inclusion policy Last-Two-From-All not passed. Site needs jobs for all templates (1369, 1368, 1371, 1370) 20331987 »
Tue 12 May 2026 01:35:01warningUM6P->UM6P: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7136771472 ==> modificationHost: slurm-compute-h22b5-u35-svn2 pilot:::9000 Failed in data staging: Transfer timed out;Failed in data staging: Transfer timed out ddm:::200 expected output 10b537c5-f7db-42a4-8c12-01893a97173e_66326.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7136736477 ==> modificationHost: slurm-compute-h21c8-u12-svn3 pilot:::9000 Failed in data staging: Transfer timed out;Failed in data staging: Transfer timed out ddm:::200 expected output b9212d2f-3f05-4b32-a9c4-5a3bf7de56ae_9419.1.job.log.tgz is missing in pilot JSON 20331983 »
Tue 12 May 2026 01:35:01warningUKI-NORTHGRID-LIV-HEP->UKI-NORTHGRID-LIV-HEP: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7136886956 ==> modificationHost: slot1_22@r26-n10.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.84881e5f-e16f-4cba-963b-fee98cd3974a_57592.pool.root to UKI-NORTHGRID-LIV-HEP_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136849674 ==> modificationHost: slot1_10@r26-n20.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:hc_20331979.UKI-NORTHGRID-LIV-HEP.35298d79-75cd-4666-ba57-d4436ddd9ba2_44194.hist-out.root to UKI-NORTHGRID-LIV-HEP_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136850590 ==> modificationHost: slot1_31@r26-n19.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:hc_20331987.UKI-NORTHGRID-LIV-HEP.baf5fd14-c6a8-4828-830b-fa0132dada74_29729.hist-out.root to UKI-NORTHGRID-LIV-HEP_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136848763 ==> modificationHost: slot1_3@r26-n15.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.15a9af25-6642-4714-a509-13308bc8980c_73128.pool.root to UKI-NORTHGRID-LIV-HEP_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136864307 ==> modificationHost: slot1_23@r26-n20.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:output.hc_20331996.UKI-NORTHGRID-LIV-HEP.b0e808a1-37dc-4af7-8991-c52116f5bc4d_63698.EVNT.root to UKI-NORTHGRID-LIV-HEP_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136813081 ==> modificationHost: slot1_22@r25-n03.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:output.hc_20331996.UKI-NORTHGRID-LIV-HEP.8260f6fb-b2c1-4dd8-9c07-af11bc9860da_30951.EVNT.root to UKI-NORTHGRID-LIV-HEP_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136850328 ==> modificationHost: slot1_5@r26-n13.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:Hits.hc_20331981.UKI-NORTHGRID-LIV-HEP.8318f8ef-e887-4d33-8857-d385aeee9b51_55122.HITS.pool.root to UKI-NORTHGRID-LIV-HEP_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7136822596 ==> modificationHost: slot1_10@r26-n14.ph.liv.ac.uk pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.87adf658-b6f7-49f4-8194-0134947886d4_9764.pool.root to UKI-NORTHGRID-LIV-HEP_DATADISK, copy command timed out: TimeoutException: Timeout reached, timeout=410 seconds')]:failed to transfer files using copytools=['rucio'] 20331987 »
«« first Page 1 of 163 next » last »»