Data import
We now have several new data import services, each tailored to addressing different needs: easy once-off file uploads, more convenient command-line file uploads, upload of large data volumes with restart and incremental synchronization, and real-time delivery of Nettskjema.
Easy web-based file upload
Our new web-based data upload service available at https://data.tsd.usit.no, makes once-off file upload as simple as possible. You can log in from anywhere in the world, without having to download any software, and upload files directly to you project area. Once the upload is finished, the data is immediately available for usage, without any waiting time. File sizes up to 10GB are supported. Read more about the service here: https://www.uio.no/english/services/it/research/sensitive-data/use-tsd/import-export/import-data-using-data.tsd.usit.no.html.
API (Application Programming Interface) based file upload
For researchers who prefer working on the command-line, it is possible to use the same upload service with the TSD API client: https://github.com/unioslo/tsd-api-client. There is no limitation on the size of data uploads when using the TSD API client. However, if the network connection drops, then one will have to restart the upload. This is, therefore, appropriate for data between 10 and roughly 100GB.
Many researchers, however, have large volumes of data. Often these data have to be transfered over unreliable network connections. These large data volumes are also often subject to change, and researchers would therefore like to synchronize the changes into TSD incrementally. For this we have developed an S3 API, this is an object storage service, compatible with Amazon Web Services' S3 API. This service, which is available on request, allows uploads with restart capabilities, and incremental synchronization. We have successfully tested this with 1TB data transfer. To use the service one has to install the TSD API client: https://github.com/unioslo/tsd-api-client.
Real-time delivery of Nettskjema to your "/durable" folder
We are very excited to announce the introudction of real-time delivery of Nettskjema submissions. Using our new data transfer infrastructure we are able to store Nettskjema submissions directly in the project area, without any delay. This will support faster iteration in your workflows.
This transition will bring some changes: today, submissions are stored in the project's import area, such as /tsd/pXX/fx/import/sns/GPG-KEY-ID. Submissions that are delivered via the new solution will be stored in your durable area, such as /tsd/pXX/data/durable/nettskjema-submissions/GPG-KEY-ID. We will therefore create this new directory in your project's data/durable folder.
To ensure that all existing workflows can be transitioned without disrupting ongoing work we will run the existing and new data collection methods in parallel for some weeks. We will notify you in advance when we plan to switch off the old delivery method.
GDPR compliance
With the recent enforcement of the GDPR, collecting and managing consent has become more important than ever. This is a complex challenge in terms of security and data management. As part of the BigMed project, TSD is developing a Dynamic Consent System. This system will allow researchers to collect digitially signed consent forms from respondents, which will be stored in TSD. Signatures will be fully-fledged adwanced signatures using BankID with the DiFi signature system. TSD project members can then query these consents from inside TSD, to ensure that workflows are specified in accordance with the consents granted by respondents. Additionally, respondents who submitted their consent to participate in specific studies will have access to their consents via an external portal. This portal will allow them to revoke existing consents, grant new ones, and get an overview of which projects their data is being used in. Taken together, this service will make it easier for TSD projects to be compliant with the GDPR. The Dynamic Consent System will be available no later than January 2019. We are alos working on enabling the PI / research project to use this channel to give information back to their research objects.
Further, to enable you to be in compliant with GDPR we will shortly provide you with updated data processor agreements so that you may use this to get your paperwork compliant.
Self service project administration
We are aware of the fact that the current routines for collecting user credentials is too time consuming. To address this we have developed a self service portal, whereby users with Norwegian personal numbers (i.e. BankID) can log in, set their own password, and scan a QR code for their one-time codes. This will reduce waiting time to zero when users need to get their first login credentials, reset their password, or re-scan a QR code. The service will be launched in September 2018. Throughout the rest of 2018 we will be working to add new user registration to this self service portal.
We will announce the web adress as soon as this is in production.
Application deadline for TSD compute and storage resources from Uninett Sigma2 AS (24th og August, 2018)
As TSD became a national service under the Sigma2 umbrella (https://www.sigma2.no/) late 2017, any research project in need of storage from 10TiBs and upwards or usage of CPU on Colossus may apply Sigma2 Resource Allocation Commitee for storage and compute resources (https://www.sigma2.no/content/apply-e-infrastructure-resources). Storage will be provided at a discounted price, while CPU is provided for free as long as you are not in need of dedicated CPU hours (i.e. are willing to wait in the SLURM job submission queue, as everyone do today).
Prices for resources through Sigma2 RFK can be found here : https://www.uio.no/english/services/it/research/sensitive-data/access/prices/
NB The deadline for the application the next allocation period [1st October 2018 to 1st April 2019] is 24th of August 2018.
Apply here : https://www.sigma2.no/content/apply-e-infrastructure-resources