TSD Operational Log - Page 11
*UPDATE*
/cluster should now be available on the new p<NUM>-submit.tsd.usit.no hosts. You can reach these by SSH from your login hosts, and from Windows you can access it using the Putty.
Modules are also available on these hosts, so that you can test your pipelines with the new software.
*END UPDATE*
Dear TSD users,
Unfortunately, something has happened with the mounts of /cluster on the new RHEL7 submit hosts created for the new Colossus cluster. We're working on it and will update this notice as soon as it is fixed.
Our apologies for the inconvenience.
--
Best regards,
The TSD team
The new cluster, Colossus 3, is now up, and the old cluster is turned off.
We are experiencing issues with some services, which may lead to some users being unable to login to TSD. We are investigating the cause of this and working on fix.
Some of our users are currently experiencing problems with Modules on Linux VM.
We are working on resolving this issue.
The TSD self service portal https://selfservice.tsd.usit.no is currently unavailable, and attempted logins will result in a 502 error.
We are investigating, and will update this message as we make progress.
--
Best regards,
The TSD-team
Dear TSD User
We are experiencing issues with thinlinc login. We are working to fix this. Until then, it will not be possible to login to linux VMs.
Regards
TSD
We are experiencing issues with one of the the BeeGFS file system nodes at the moment. To fix this we will try to restart a part of the IO system, which may cause hangs on VMs, and may cause parts of /cluster to be unavailable. If this does not work, then we will have to reboot the node.
There will be a scheduled upgrade of PostgreSQL to V11 on 13.02.2019, between 07:00 - 15:00 CET.
During this downtime, the applications running PostgreSQL will not work, as we will restart the database in your project. Other services inside TSD will continue working as normal.
We're currently experiencing issues with the export of /cluster from Colossus. Something went wrong during our nightly builds and we are working on solving the issue.
--
The TSD-team
Update to web-based file uploads. After this change files uploaded with https://data.tsd.usit.no and the tsd-api-client will be located in /data/durable/file-import/pXX-member-group, instead of the previous location: /data/durable/file-api.
The SPSS license is currently not valid. We are trying to update it as soon as possible. Thanks for your patience.
TSD
Dear TSD-users,
Unfortunately, our services are currently unavailable due to a DNS-issue. We're aware of the problem and working on solving this as quickly as possible.
Our apologies for the inconvenience.
Best regards,
The TSD-team.
There are currently problems with the 2 factor authentication, which makes new logins to TSD impossible. (Existing connections are not affected.)
A side effect is also that syncronization of new QR code keys has stopped.
Update: The reason for the downtime was a failed synchronization. Everything should be working now, including newly generated QR codes. If you are still experiencing problems, please contact us.
We will perform a scheduled system upgrade of Colossus starting at 2019-01-03, 10:00 until 2019-01-04, 10:00.
UPDATE (2019-01-04, 12:35)
The upgrade of Colossus is complete.
UPDATE (2019-01-04, 09:45)
We are experiencing a slight delay, and hopefully Colossus will be available for use by 13:00 CET.
Lately, the usage of Colossus has increased, and most of the time every node is busy running the jobs. The Infiniband network on one of the storage nodes has been unstable lately, causing decreased performance and lower availability. We will attempt to correct this on Friday between 09:00 - 11:00 CET.
We will place a reservation on the queue system during the aforementioned timeframe. This way the new jobs that overlap the timeframe will not start, while allowing currently running jobs to finish.
Submitted jobs that are not specified to finish before Friday at 09:00 will remain queued until the maintenance is finished.
We are sorry for the inconvenience.
TSD@USIT
TSD is inaccessible for the moment, and all services are affected. We are working to correct the issue.
TSD@USIT
Some of Our users are unable to login to TSD through ThinLinc. We are working on resolving this issue.
TSD@USIT
Dear TSD users,
Due to a network issue, the service is currently unavailable. We're working on solving this as quickly as possible, and will update this message as we progress.
Our apologies for the inconvenience.
--
Best regards,
The TSD team
Login to TSD through view-ous.tsd.usit.no is not working for the moment, and we are working on resolving this issue.
TSD@USIT
We are experiencing problems with the FileLock. Until the issue is solved user are advised to use the new Web File uploader for imports: https://www.uio.no/english/services/it/research/sensitive-data/use-tsd/import-export/index.html#toc4
The planned maintenance work in TSD has started. The TSD services will not be available today between 13:00 - 16:00 CET.
(Update: 15:56): All services are working.
Kind Regards,
TSD@USIT
Due to storage capacity problems, jobs have been pending on Colossus. We will update this status when we have resolved this issue.
Several issues makes most services in TSD unavailable. We are investigating the problems, and will come back with an update.
Update (12:05): Because one of the virtualisation clusters crashed, many VMs were forcefully restarted. Most services is back up now, but some services will need manual interactions.
Update (12:30): All services should behave normally now.
We are experiencing issues with some services, which may lead to some users being unable to login to TSD. We are investigating the cause of this and working on fix.
Solved: Some services hanged after a unplanned reboot of a part of our infrastructure last night. We have restarted the services that seemed to be affected, and all services should be up now.
On first of October between 13:00 - 16:00 CET, our team of engineers will perform an infrastructure upgrade. This upgrade is necessary, as we need more VLANs for our increasing growth of projects in TSD.
The downtime will affect all our services, so please do not schedule any long running jobs during this time. Please, also save your data before the maintenance window, and follow our Operation Log for the update:
http://www.uio.no/english/services/it/research/sensitive-data/log/
We are sorry for the inconvenience.
Kind Regards,
TSD@USIT