HammerCloud | ATLAS rss

Administration
<< Back

PanDA incidents history

The following table summarizes the incidents happened in HammerCloud tests. You can filter the results using the form below and clicking in 'Refresh'.


Incidents for last 72 hours

TimestampSeverityCommentTest
Tue 08 Sep 2026 17:05:46warningThe following sites need more test jobs for the PFT template 1370: CYFRONET_EOS, DE-TARDIS, INFN-NAPOLI-ATLAS, LRZ-LMU_TEST, UKI-SCOTGRID-ECDF_CLOUD, UKI-SOUTHGRID-BHAM-HEP, UKI-SOUTHGRID-SUSX, UNIGE-BAOBAB, Vega 20337320 »
Tue 08 Sep 2026 17:05:46warningThe following sites need more test jobs for the PFT template 1371: CYFRONET_EOS, DE-TARDIS, LRZ-LMU_TEST, SiGNET-NSC_MCORE, UKI-SCOTGRID-ECDF_CLOUD, UKI-SOUTHGRID-BHAM-HEP, UKI-SOUTHGRID-SUSX, UNIGE-BAOBAB, Vega 20337320 »
Tue 08 Sep 2026 17:05:46warningThe following sites need more test jobs for the ARM template 1367: FZK-LCG2_ARM 20337316 »
Tue 08 Sep 2026 17:05:46warningThe following sites need more test jobs for the AFT template 1368: ANALY_BNL_VP, CYFRONET_EOS, DESY-HH_NAF, UNIGE-BAOBAB, wuppertal_VP 20337308 »
Tue 08 Sep 2026 17:05:46warningThe following sites need more test jobs for the AFT template 1369: ANALY_BNL_VP, CYFRONET_EOS, DESY-HH_NAF, UNIGE-BAOBAB, wuppertal_VP 20337308 »
Tue 08 Sep 2026 17:05:46warningThe following sites need more test jobs for the PFT_MCORE template 1372: SARA-MATRIX_HPC, UKI-SOUTHGRID-SUSX_TEST 20337306 »
Tue 08 Sep 2026 17:05:03warningVega_GPU->Vega_GPU: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7301530702 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Job submission to LRMS failed ddm:::200 expected output 52c422ef-2d59-42d0-9980-225f08450678_69781.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301516690 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Job submission to LRMS failed ddm:::200 expected output 8efc29ab-aece-4c67-ae53-71c792740287_64960.1.job.log.tgz is missing in pilot JSON 20337305 »
Tue 08 Sep 2026 17:05:03warningUNIGE-BAOBAB->UNIGE-BAOBAB: Auto-inclusion policy Last-Two-From-All not passed. Site needs jobs for all templates (1369, 1368, 1371, 1370) 20337308 »
Tue 08 Sep 2026 17:05:03warningUNIBE-LHEP-UBELIX->UNIBE-LHEP-UBELIX: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7301261409 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 36dd5faa-d7d2-467f-abc5-d039d2795e43_96390.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301256651 ==> modificationHost: aipanda403.cern.ch pilot:::9000 Unknown error ddm:::200 expected output faeeb2f1-1fce-4b1e-a2fc-72edac3d9cc8_74831.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301367860 ==> modificationHost: aipanda403.cern.ch pilot:::9000 Unknown error ddm:::200 expected output ef11c9f2-28b0-4fe0-87b1-8062440af3f5_55388.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301233964 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output e38ac660-45d8-4f9a-9fbb-07035928eaa1_4283.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301364956 ==> modificationHost: aipanda403.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 1e8cb731-a306-4127-bd73-854b8f5ea523_56516.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301358516 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output cebcc251-5df4-44e1-9891-3ead61a1e396_8765.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301331648 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 8642f785-f86f-4cab-acfb-bbc1dfb7434d_32927.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301337574 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 8622e38f-56f6-45cf-b8a9-d40668f255a5_46445.1.job.log.tgz is missing in pilot JSON 20337308 »
Tue 08 Sep 2026 17:05:03warningUM6P->UM6P: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7301374966 ==> modificationHost: aipanda403.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 22c441a4-c69e-4f4e-9527-06eb51159cdd_99446.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301267492 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 2228c2cf-d300-42e4-bda5-c47e5b047413_5145.1.job.log.tgz is missing in pilot JSON 20337306 »
Tue 08 Sep 2026 17:05:02warningUKI-NORTHGRID-LIV-HEP->UKI-NORTHGRID-LIV-HEP: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7301524747 ==> modificationHost: slot1_4@r25-n09.ph.liv.ac.uk pilot:::1368 Remote file open timed out: (ALRB: 1 s) | Beginning container setup Info: Migrating env variable SINGULARITY_CACHEDIR to APPTAINER_CACHEDIR (ALRB: 1 s) | Container runtime apptainer selected (ALRB: 1 s) | Construct commamd (ALRB: 1 20337308 »
Tue 08 Sep 2026 17:05:02warningPUHTI->PUHTI: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7301411371 ==> modificationHost: aipanda403.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 73309713-5725-4d5a-965d-b836327bfaaa_89296.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301335872 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 466e5a68-628a-4b5e-b68b-ceddf66946f3_22662.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301423802 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output dbfa8ca6-79d1-4fcc-9647-fc35cb0cb638_56827.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301349846 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output d7151dc3-0562-4424-b484-c2c17ec83216_31182.1.job.log.tgz is missing in pilot JSON 20337320 »
Tue 08 Sep 2026 17:05:02warningLRZ-LMU_TEST->LRZ-LMU_TEST: Auto-inclusion policy Last-Two-From-All not passed. Site needs jobs for all templates (1371, 1370) 20337320 »
Tue 08 Sep 2026 17:05:02warningLRZ-LMU->LRZ-LMU: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7301372829 ==> modificationHost: hpdar06c05s12 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://lcg-lrz-rootd.grid.lrz.de:1094/pnfs/lrz-muenchen.de/data/atlas/dq2/atlasdatadisk/rucio/mc20_13TeV/96/80/DAOD_PHYS.34870879._000009.pool.root.1'] ** http://bigpanda.cern.ch/job?pandaid=7301361725 ==> modificationHost: hpdar06c05s12 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://lcg-lrz-rootd.grid.lrz.de:1094/pnfs/lrz-muenchen.de/data/atlas/dq2/atlasdatadisk/rucio/mc20_13TeV/39/79/DAOD_PHYS.34870879._000008.pool.root.1'] ** http://bigpanda.cern.ch/job?pandaid=7301367459 ==> modificationHost: hpdar06c05s12 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://lcg-lrz-rootd.grid.lrz.de:1094/pnfs/lrz-muenchen.de/data/atlas/dq2/atlasdatadisk/rucio/mc20_13TeV/e8/e2/DAOD_PHYS.34870879._000011.pool.root.1'] ** http://bigpanda.cern.ch/job?pandaid=7301358890 ==> modificationHost: hpdar06c05s12 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://lcg-lrz-rootd.grid.lrz.de:1094/pnfs/lrz-muenchen.de/data/atlas/dq2/atlasdatadisk/rucio/mc20_13TeV/65/79/DAOD_PHYS.34870879._000010.pool.root.1'] ** http://bigpanda.cern.ch/job?pandaid=7301373229 ==> modificationHost: hpdar06c05s12 pilot:::1324 'mc23_13p6TeV:TXT.43313123._000011.tar.gz.1 from LRZ-LMU_DATADISK, Could not open source: Result Could not connect to server after 1 attempts')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7301361875 ==> modificationHost: hpdar06c05s12 pilot:::1324 'mc23_13p6TeV:TXT.43313123._000274.tar.gz.1 from LRZ-LMU_DATADISK, Could not open source: Result Could not connect to server after 1 attempts')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7301365093 ==> modificationHost: hpdar06c05s12 pilot:::1324 'mc21_13p6TeV:EVNT.29070483._000367.pool.root.1 from LRZ-LMU_DATADISK, Could not open source: Result Could not connect to server after 1 attempts')]:failed to transfer files using copytools=['rucio'] ** http://bigpanda.cern.ch/job?pandaid=7301356799 ==> modificationHost: hpdar06c05s12 pilot:::1324 'mc21_13p6TeV:EVNT.29070483._000407.pool.root.1 from LRZ-LMU_DATADISK, Could not open source: Result Could not connect to server after 1 attempts')]:failed to transfer files using copytools=['rucio'] 20337308 »
Tue 08 Sep 2026 17:05:02warningCYFRONET->CYFRONET: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7301526478 ==> modificationHost: x1001c7s6b0n2 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://eos01.grid.cyfronet.pl:1094//eos/atlas/atlasdatadisk/rucio/mc20_13TeV/39/79/DAOD_PHYS.34870879._000008.pool.root.1'] ** http://bigpanda.cern.ch/job?pandaid=7301511607 ==> modificationHost: x1001c7s6b0n2 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://eos01.grid.cyfronet.pl:1094//eos/atlas/atlasdatadisk/rucio/mc20_13TeV/f6/72/DAOD_PHYS.34870879._000015.pool.root.1'] ** http://bigpanda.cern.ch/job?pandaid=7301529027 ==> modificationHost: x1001c7s6b0n2 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://eos01.grid.cyfronet.pl:1094//eos/atlas/atlasdatadisk/rucio/mc20_13TeV/96/80/DAOD_PHYS.34870879._000009.pool.root.1'] ** http://bigpanda.cern.ch/job?pandaid=7301524728 ==> modificationHost: x1001c7s6b0n2 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://eos01.grid.cyfronet.pl:1094//eos/atlas/atlasdatadisk/rucio/mc20_13TeV/39/79/DAOD_PHYS.34870879._000008.pool.root.1'] ** http://bigpanda.cern.ch/job?pandaid=7301531915 ==> modificationHost: x1001c7s6b0n2 pilot:::1324 Service not available at the moment: mc23_13p6TeV:TXT.43313123._000117.tar.gz.1 from CYFRONET-LCG2-EOS_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Server responded with an error: [3010] Unable to open file /eos/atlas/atlasdatadisk/rucio/mc23_13p6TeV/89/6b/TXT.43313123._000117.tar.gz.1; Operation not permitted (source ** http://bigpanda.cern.ch/job?pandaid=7301529333 ==> modificationHost: x1001c7s6b0n2 pilot:::1324 Service not available at the moment: mc23_13p6TeV:TXT.43313123._000115.tar.gz.1 from CYFRONET-LCG2-EOS_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Server responded with an error: [3010] Unable to open file /eos/atlas/atlasdatadisk/rucio/mc23_13p6TeV/26/41/TXT.43313123._000115.tar.gz.1; Operation not permitted (source ** http://bigpanda.cern.ch/job?pandaid=7301516559 ==> modificationHost: x1001c0s4b0n0 pilot:::1324 Service not available at the moment: mc21_13p6TeV:EVNT.29070483._000132.pool.root.1 from CYFRONET-LCG2-EOS_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Server responded with an error: [3010] Unable to open file /eos/atlas/atlasdatadisk/rucio/mc21_13p6TeV/ef/df/EVNT.29070483._000132.pool.root.1; Operation not permitted ** http://bigpanda.cern.ch/job?pandaid=7301504765 ==> modificationHost: x1001c0s4b0n0 pilot:::1324 Service not available at the moment: mc21_13p6TeV:EVNT.29070483._000302.pool.root.1 from CYFRONET-LCG2-EOS_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Server responded with an error: [3010] Unable to open file /eos/atlas/atlasdatadisk/rucio/mc21_13p6TeV/0d/74/EVNT.29070483._000302.pool.root.1; Operation not permitted ** http://bigpanda.cern.ch/job?pandaid=7301515168 ==> modificationHost: x1001c0s4b0n0 pilot:::1324 Service not available at the moment: mc21_13p6TeV:EVNT.29070483._000269.pool.root.1 from CYFRONET-LCG2-EOS_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Server responded with an error: [3010] Unable to open file /eos/atlas/atlasdatadisk/rucio/mc21_13p6TeV/30/a1/EVNT.29070483._000269.pool.root.1; Operation not permitted ** http://bigpanda.cern.ch/job?pandaid=7301500591 ==> modificationHost: x1001c0s4b0n0 pilot:::1324 Service not available at the moment: mc21_13p6TeV:EVNT.29070483._000417.pool.root.1 from CYFRONET-LCG2-EOS_DATADISK, Error on XrdCl:CopyProcess:Run(): [ERROR] Server responded with an error: [3010] Unable to open file /eos/atlas/atlasdatadisk/rucio/mc21_13p6TeV/21/31/EVNT.29070483._000417.pool.root.1; Operation not permitted 20337308 »
Tue 08 Sep 2026 17:05:02warningANALY_LRZ_VP->ANALY_LRZ_VP: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7301164154 ==> modificationHost: hpdar06c05s12 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://grid-dc.t2.mpcdf.mpg.de:1094//atlasdatadisk/rucio/mc20_13TeV/f6/72/DAOD_PHYS.34870879._000015.pool.root.1'] ** http://bigpanda.cern.ch/job?pandaid=7301159559 ==> modificationHost: hpdar06c05s12 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://grid-dc.t2.mpcdf.mpg.de:1094//atlasdatadisk/rucio/mc20_13TeV/6f/7b/DAOD_PHYS.34870879._000016.pool.root.1'] ** http://bigpanda.cern.ch/job?pandaid=7301163301 ==> modificationHost: hpdar06c05s12 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://grid-dc.t2.mpcdf.mpg.de:1094//atlasdatadisk/rucio/mc20_13TeV/fd/8f/DAOD_PHYS.34870879._000002.pool.root.1'] ** http://bigpanda.cern.ch/job?pandaid=7301158308 ==> modificationHost: hpdar06c05s12 pilot:::1361 Remote file could not be opened:Remote file(s) could not be opened: ['root://grid-dc.t2.mpcdf.mpg.de:1094//atlasdatadisk/rucio/mc20_13TeV/39/79/DAOD_PHYS.34870879._000008.pool.root.1'] 20337308 »
Tue 08 Sep 2026 17:05:01warningAM-01-AANL->AM-01-AANL: AutoInclusion policy Last-Two-From-All not passed. See jobs ** http://bigpanda.cern.ch/job?pandaid=7301266536 ==> modificationHost: aipanda404.cern.ch pilot:::9000 Unknown error ddm:::200 expected output 684a87e9-777a-43fe-9fc5-ebf85da8e8fd_73393.1.job.log.tgz is missing in pilot JSON ** http://bigpanda.cern.ch/job?pandaid=7301264607 ==> modificationHost: aipanda066.cern.ch pilot:::9000 Unknown error ddm:::200 expected output dac22cce-196c-4d78-a6b8-2938fb61d8cc_81504.1.job.log.tgz is missing in pilot JSON 20337306 »
Tue 08 Sep 2026 17:05:01blacklistingChecking these sites currently test because of HC: ['AM-01-AANL', 'ANALY_LRZ_VP', 'BNL_GCP', 'CYFRONET', 'LRZ-LMU', 'LRZ-LMU_TEST', 'PUHTI', 'UKI-NORTHGRID-LIV-HEP', 'UM6P', 'UNIBE-LHEP-UBELIX', 'UNIGE-BAOBAB', 'Vega_GPU']  
Tue 08 Sep 2026 17:00:06testinfoTest 20337331 was assigned to hammercloud-ai-47.cern.ch. 20337331 »
Tue 08 Sep 2026 17:00:02queuecontrolLUMI was set to 'online' by an external agent.  
Tue 08 Sep 2026 16:50:33warningThe following sites need more test jobs for the PFT template 1370: CYFRONET_EOS, DE-TARDIS, INFN-NAPOLI-ATLAS, LRZ-LMU_TEST, UKI-SCOTGRID-ECDF_CLOUD, UKI-SOUTHGRID-BHAM-HEP, UKI-SOUTHGRID-SUSX, UNIGE-BAOBAB, Vega 20337320 »
Tue 08 Sep 2026 16:50:33warningThe following sites need more test jobs for the PFT template 1371: CYFRONET_EOS, DE-TARDIS, LRZ-LMU_TEST, SiGNET-NSC_MCORE, UKI-SCOTGRID-ECDF_CLOUD, UKI-SOUTHGRID-BHAM-HEP, UKI-SOUTHGRID-SUSX, UNIGE-BAOBAB, Vega 20337320 »
Tue 08 Sep 2026 16:50:33warningThe following sites need more test jobs for the PFT_MCORE template 1372: SARA-MATRIX_HPC, UKI-SOUTHGRID-SUSX_TEST 20337306 »
Tue 08 Sep 2026 16:50:33warningThe following sites need more test jobs for the ARM template 1367: FZK-LCG2_ARM 20337316 »
Tue 08 Sep 2026 16:50:33warningThe following sites need more test jobs for the AFT template 1368: ANALY_BNL_VP, CYFRONET_EOS, DESY-HH_NAF, UNIGE-BAOBAB, wuppertal_VP 20337308 »
«« first Page 1 of 224 next » last »»