TSD Data Loader

TSD projects that are using Nettskjema can get a more streamlined and automatic data flow. This is a system tailored for research projects using Nettskjema in their data collection. It handles decryption of submissions as well as producing complete output files in your secure storage area within TSD containing all the original data registered in Nettskjema. The output files can be imported into standard software for further analysis.

What it does

Every night the TSD Data Loader automatically collects all Nettskjema responses from a TSD project import directory and creates a new updated file containing the submissions for a specific form. This means that every day there will be an updated raw data file (.csv) in the project's secure storage area for every Nettskjema activated by the project. This will be the master file with the complete data set, exactly as entered into the forms.

The TSD Data Loader aims at delivering the original research data «as is» and will not modify data in any sense. Further treatment and analysis of the data is not a task for this system and should rather be done by using dedicated software for statistics and alike.

How to open the .csv files

The TSD Data Loader creates an output folder with the original master files in your secure storage area. This folder is called "csvfiles" and is created in the "data\durable" folder. There will be one text file for each form. The Nettskjema form ID is represented in the filename with "internal_" as prefix (e.g. internal_69659.csv).

Data Loader also creates and uses certain hidden system directories in the durable folder. These are .decrypted, .handled, .failed etc — these are not user serviceable and changing them in any way may break Data Loader operation.

About the files

  • The files in the "csvfiles" folder are updated as long as new submissions are posted from Nettskjema. They will be overwritten on a nightly basis and should therefore not be edited or moved.
  • As they contain original research data they should be kept where they are for the duration of the study. Do not edit, delete or move the files.
  • The .csv files can safely be copied to another directory for further treatment and analysis.
  • The file format is comma-separated values encoded with UTF–8. When importing into standard software for analysis choose comma as delimiter and UTF–8 as encoding (see example below).

Automatic update frequency

Most projects are scheduled to update every night. This means that in the morning you will find all submissions up to the previous day.

Example reading data into Excel

  1. Open Excel
  2. Create a new file ("File" > "New" > "Empty workbook")
  3. Go to the "Data" tab and choose "From text".
  4. Select desired .csv file (with form ID part of its file name) in the "csvfiles" folder and choose "Import".
  5. Select the "Data with delimiter" for "Original data type" and "UTF-8" (you may find it as "65001 : Unicode (UTF-8)") for "File origin".
  6. In the next dialog select only comma for delimiter.
  7. Use "Finish" to close the dialog and "OK" on the next dialog to proceed without additional options.

Example reading data into SPSS

In order to read the .csv files into SPSS do the following:

  1. Open SPSS.
  2. Choose "File" from the menu and "Read Text Data".
  3. In the dialog select "Unicode (UTF–8)" and navigate to the "csvfiles" folder and open the .csv file.
  4. Answer «No» to whether your text file matches a predefined format.
  5. Select "Delimited" and "Yes, my variables are included at the top of the file".
  6. The actual data begin with line number 2, you may import all of them.
  7. Now select only comma as delimiter (deselect space, tab etc) and "Double quote" as text qualifier.
  8. You should now get a data preview. Go through all the variables (selecting their respective column in the preview) and check that the data format is correct (numeric, string, date etc).
  9. Choose "Next" and "Finish".
  10. Save the SPSS file as a new file somewhere within the "durable" folder (i.e. nettskjema-results) or your home folder.

Using syntax files in SPSS

If you go through the above procedure often you can save the SPSS syntax and reuse it when needed. You can then make a syntax in SPSS for importing the CSV file generated by Data Loader. This automation will help you speed up your workflow, e.g. if you need to check something in the data set often. The syntax for the import routine described above can be saved and reused (it comes up an option for "paste syntax" as one of the final steps). It's also possible to supplement it with syntax for frequencies etc.

Example-syntax for importing

Screenshot of SPSS window showing saved import procedure

You can also download the Nettskjema codebook as SPSS syntax to label variables and values.

References

Example of reading data into STATA

  1. Open STATA.
  2. From the menu choose "File", "Import" and then "Text Data (delimited *.csv, …)".
  3. Choose «Browse» and navigate to the .csv file.
  4. Select «UTF–8» as text encoding. Rest of the settings can be left as they are.
  5. Choose "OK".

Now the data set should be imported into STATA. Have a look at the STATA data editor (Browse) on the menu bar.

If you wish to save the import command for later, right click the command used in the «Review pane» (usually located on the left side) and choose «Send selected to Do-file Editor». The command will look something like the following:

Screenshot of the Do-file Editor window

Save the do-file somewhere in the "durable" folder or your home area. Next time you need to import this data file in STATA, you can easily right-click the .do file and choose «Execute (do)». If new submissions are delivered via Nettskjema, they will appear as cases with new lines in the data file. Very cool!

Data Loader reports

In addition to producing update files with original research data, the service also generates a report. The report gives a summary of everything that has been handled so far. It's located in the "durable" folder and is a plain text file that can be opened with text editors like Wordpad or Notepad.

The report lists how many forms that have currently been decrypted or otherwise handled and how many responses are collected. This is just for bookkeeping. It should state that a number of files are handled, decrypted etc, like "0 failed /0 unaccounted for". This means that all the data has been processed and the system works fine.

The guarantee here is that you will find all the submissions up to the previous day.

Requirements

There are a few requirements for successful installation of the Data Loader for a project:

  • The forms belonging to the project should be pre-coded using the Nettskjema codebook functionality. This is to ensure the data flow through the loader and also to be able to successfully import the data in applications like SPSS, Stata, Excel, R etc.
  • Maximum 999 variables per form allowed. In practice this means not having more than 999 questions in a single Nettskjema form.
  • Variable names in the codebook may be up to 60 characters long, and the first character must be a letter. Subsequent characters can be a combination of letters and numbers (no blank spaces in between). Variable names must also be unique within a form.
  • If you collect national ID numbers, the name of the corresponding variable should be either national_id_number or Fodselsnr.
  • Variable names defined in the Nettskjema codebook must not have characters from the following set: æ, ø, å, ;, ?, (, ), !, @, #, $, %, ^, *, +, =, [, ], <, >, ~, `, |, -, ., :.

Getting started with Data Loader

The service is a free add-on for TSD projects that are affiliated with UiO, provided they fulfill the above requirements. This solution is a “service-on-demand” and is currently not installed by default for new TSD projects. If you wish to get it installed for your project you need to contact our technical support. Note that there should be at least one Nettskjema answer registered in your TSD project before Data Loader can be installed. For projects that have institutional affiliations other than UiO please refer to the applicable section of the TSD price list.

For projects with specific needs and more advanced requirements to managing Nettskjema submissions we offer handling these using a relational database system. To negotiate this solution please contact our administrative support. Note that these advanced features that go beyond the standard Data Loader setup are associated with an initial setup cost and a yearly fee.

Support

If you need technical support with Data Loader, contact our technical support.

Published Feb. 17, 2017 1:23 PM - Last modified Sep. 3, 2020 4:50 PM