[DONE] Storage maintenance and downtime 2023-11-30 08:00 CET

TSD will be upgrading software on the storage system Thursday, 2023-11-30 from 08:00 CET. We expect storage instability on the Windows and Linux vms throughout the day.

Around 10:00 the storage system will be shut down for an estimated 15min, which means network storage is inaccessible on all TSD hosts (Windows and Linux) as well as on our central services (file import/export, etc). To be on the safe side, please close any programs and log off from your vm prior to the downtime.

A maintenance reservation has been set on Colossus from 08:00. This means any jobs that cannot complete before the downtime will remain pending until after the maintenance completes. They'll resume automatically.

Our automation should fix any file system hangs that may occur, and we will be on standby to fix any remaining issues that do not automatically recover.

Apologies for the short notice, we've been in dialogue with IBM to alleviate storage instability and want to act on their latest recommendations as fast as possible.

---

Update 11:47: The software has been upgraded and Colossus jobs have been resumed. A few NFS hosts may have to be rebooted still.