The UiO S3 solution is running on Cloudian HyperStore nodes with HAproxy in front. All hardware and networking componentes are owned by UiO and are located in USIT datacenters.

As access the S3 endpoint is open to the whole Internet it is crucial that data is stored with the proper security policies. Please see a description of policies and SSE-C encryption below.

Compatability with Amazon S3 is close to 100%, report any incompatibilites to us and we will ask Cloudian to improve it. As a result most of the AWS tips, tricks, and documentation works.

End user documentation

This section provides information you simply have to know. If we ever run into you we may even quiz you about this.

Crucial information

S3 endpoint https://s3-oslo.educloud.no:443

Bucket naming <ProjectID>-<dataclass>- e.g. 1001-green-demobucket

Apply by using this Nettskjema https://nettskjema.no/a/189898.

Getting started for owners of buckets with green and yellow data

Perform the steps below when your project application has been approved:

  1. Point your favourite browser to the CMC https://cmc.educloud.no (interface is restricted to UiO IP's)
  2. Select the group, according to the data level, so if you requested access for a yellow bucket, you choose the cloudian-yellow-owners, and log on with your UiO credentials for your system user.
  3. Setup IAM user(s) under https://cmc.educloud.no/Cloudian/iam/users/list.htm. Use a sensible naming policy, for example nettskjema-api-test and nettskjema-api-prod.
  4. Get your S3 Access Key and S3 Secret Key from https://cmc.educloud.no/Cloudian/iam/users/nettskjema-api-test/list.htm, where nettskjema-api-testis replaced with the username of your IAM users. Both this and the previous step can also be done through the IAM-API, note that you use the system user to create and delete IAM users.

Please note that you can also interact with your buckets and objects from the CMC.

Support

Use our RT queue when support is required, lagringstjenester@usit.uio.no.

Useful tips and tutorials

This section contains information that may be useful getting to know how S3 works, how files and buckets can be protected, as well as how to properly secure your credentials.

Set up credential storage

If you setup your credentials with awscli or the Powershell-module for AWS, it stores the credentials in a cleartext file under .aws/credentials. This should be avoided at all cost, and you should instead store the secrets in Vault or in a encrypted file using GPG or OpenSSL to encrypt the file.
Setup a script that let's you decrypt/retrieve the secret at will, in the file ~/.aws/config like this:

~ yourClient$ cat .aws/config
[default]
region = oslo
credential_process = /Users/Safet/retrieve_S3_vault.sh system_user_testbucket

[iam_user_testbucket1]
region = oslo
credential_process = /Users/Safet/retrieve_S3_vault.sh iam_user_testbucket1

The credential_process expects a script to output a JSON-file in the following format:

"Version": 1,
"AccessKeyId": "AKIA0123456787EXAMPLE",
"SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}'

Use one of the following examples to setup encryption or retrivial of the credentials, depending on wether you have access to Vault or not.

Using Vault for encryption and retrevial of S3 credentials

To use this method, you need to have the Hashicorp Vault client installed. In your prefered path, create the credential like this, where the name system_user_testbucke is the name of the key where you will be storing the JSON-data containing your access key and secret key.

~ yourClient$ vault kv put <your_secret_engine_path>/<prefered subfolder>/S3_credentials system_user_testbucket='{
"Version": 1,
"AccessKeyId": "AKIA0123456787EXAMPLE",
"SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}'

Next we create a shell script that can retrieve the JSON data so that awscli and the Powershell module can use it for credentials.

cat <<EOF > /Users/Safet/retrieve_S3_vault.sh
#!/bin/bash
vault kv get --field $1 it-cloudian/credentials/system_user_testbucket
EOF
chmod +x /Users/Safet/retrieve_S3_vault.sh

If you have more than one key stored in that Vault secret, you can now give the retrieve-script the name of the key, and it will try to request the secret from vault, like this:

~ yourClient$ ./retrieve_S3_vault.sh my_adminuser
{
"Version": 1,
"AccessKeyId": "AKIA0123456787EXAMPLE",
"SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

Once the script is tested and working you can add the script in the .aws/config-file:

[default]
region = oslo
credential_process = /Users/Safet/retrieve_S3_vault.sh system_user_testbucket

Notice here that the default in the config file referes to the default profile that will be used, unless specified otherwise when running the Powershell-module, or awscli.

Each time you now want to connect to your buckets, you need to make sure that you have logged in first with the Vault CLI and have a valid token.

Using Openssl to encrypt S3 credential

If you can't use Vault, you can instead make use of openssl and encrypt the S3 credentials with a password. Note that this guide assumes that you are using OpenSSL instead of LibreSSL, as pbkdf2 is not supported in LibreSSL. LibreSSL is the default SSL-library for Macs. See the following StackOverflow for how to solve that on Macs: https://stackoverflow.com/questions/61938795/openssl-pbkdf2-option-no-longer-supported

First create a JSON-file containing your credentials like this:

cat <<EOF > /Users/Safet/system_user_credentials.json
{
"Version": 1,
"AccessKeyId": "AKIA0123456787EXAMPLE",
"SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}
EOF

Then encrypt the file with OpenSSL, and delete the original file.

~ YourClient$ openssl enc -aes-256-cbc -md sha512 -pbkdf2 -iter 100000 -salt -in /Users/Safet/system_user_credentials.json -out /Users/Safet/system_user_credentials.enc
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
~ YourClient$ rm system_user_credentials

Next create a decryption script, that will be used by the awscli/Powershell AWS module to retrieve and decrypt the secrets:

cat <<EOF > /Users/Safet/decrypt.sh
openssl enc -aes-256-cbc -md sha512 -pbkdf2 -iter 100000 -salt -d -in $1 -pass env:AWS_CREDENTIAL_PASSWORD
EOF

Lastly edit the file .aws/config and add the decrypt.sh as a credential_process:

~ yourClient$ cat .aws/config
[default]
region = oslo
credential_process = /Users/Safet/decrypt.sh /Users/Safet/system_user_credentials.enc

In the decrypt.sh script above, there is a enviroment variable called $AWS_CREDENTIAL_PASSWORD. That variable needs to be set in each shell session that you will be using for S3 requests with the awscli or Powershell AWS module. The password for decryption should obviously not be stored in any files, but can be set in the shell where you will be running the AWS cli or Powershell modules like this, which will catch the user input, as well as hide the input while you are typing in the value for the variable AWS_CREDENTIAL_PASSWORD.

read -s -r AWS_CREDENTIAL_PASSWORD
export AWS_CREDENTIAL_PASSWORD

Once you close the shell, the variable is automatically removed, and next time you need to decrypt the values, you will have to re-set the value using the read-command as in the example above.

When you now try to run awscli or Powershell, it will automatically decrypt the file system_user_credentials.enc, and use the credentials stored in that file to send requests to the Cloudian S3 and IAM APIs.

S3 Using PowerShell

Even though numerous applications now has native support for S3 it may be useful to access the buckets and objects from command line or a script. In this section we describe how to do this using PowerShell. This is our preffered method on Windows, Linux, and MacOS.

Amazon has great documentation for Powershell and AWS:

Installing the AWS tools essentials consists of two steps, both performed inside Powershell

  1. Install the AWS tools installer Install-Module -Name AWS.Tools.Installer (accept the repo if it is untrusted)
  2. Install the S3 module Install-AWSToolsModule AWS.Tools.S3 -CleanUp and the IAM module Install-AWSToolsModule AWS.Tools.IdentityManagement -CleanUp.

Interacting with S3 using PowerShell and AWS

This section shows some typical commands and operations using PowerShell with AWS S3 Tool. All the examples here are using the default profile, setup in the previous section of the documentation.

Create a bucket with the name 1001-green-demo

New-S3Bucket -BucketName 1001-green-demo -EndpointUrl https://s3-oslo.educloud.no

Copy a local file to the 1001-green-demo bucket

Write-S3Object -BucketName 1001-green-demo -EndpointUrl https://s3-oslo.educloud.no -Key "remotefile.dat" -File .\localfile.dat

Download an object remotefile.dat to the current directory localfile.dat

Read-S3Object -BucketName 1001-green-demo -EndpointUrl https://s3-oslo.educloud.no -Key remotefile.dat -File localfile.dat

List all buckets

Get-S3Bucket -EndpointUrl https://s3-oslo.educloud.no

List objects in the bucket 1001-green-demo

Get-S3Object -BucketName 1001-green-demo -EndpointUrl https://s3-oslo.educloud.no

If you have multiple set of credentials and want to use the one called UiOProj0001 (which have been added by using e.g. Set-AWSCredential)

Get-S3Object -BucketName 1001-green-demo -EndpointUrl https://s3-oslo.educloud.no -Credential (Get-AWSCredential UiOProj0001)

Encrypting with SSE-C

There are several ways of encrypting S3 objects.

The UiO S3 solution currently support SSE-C (Server Side Encryption with Customer key). Data will be encrypted at rest and will not accidentaly be made available with a public URL. It provides a great additional level of security.

When using SSE-C the object and an encryption key is uploaded. At the S3 endpoint the object is encrypted with the encryption key and the key is then disgarded. The UiO S3 solution only supports https ensuring that the key is never transmitted in plain text.

Note If the encryption key is lost, so is the object!

Examples and tips:

Amazon has a great tutorial for this

https://aws.amazon.com/blogs/developer/s3-server-side-encryption-with-windows-powershell/

Example of uploading a file that is to be encrypted, and then downloading it.

#
# Bucket and file names
#
$initialfile  = "tutorial01.txt"
$bucket       = "1001-green-samples"
$objectkey    = "tutorial02.txt" 
$restoredfile = "tutorial01_restore.txt"

#
# The encryption key
#
$Aes = New-Object System.Security.Cryptography.AesManaged
$Aes.KeySize = 256
$Aes::GenerateKey
$Base64key = [System.Convert]::ToBase64String($Aes.Key)

#
# Upload the file and key (object will be encrypted at the endpoint)
#
try 
{
    Write-S3Object -EndpointUrl https://s3-oslo.educloud.no -File $initialfile -BucketName $bucket -Key $objectkey -ServerSideEncryptionCustomerProvidedKey $Base64key -ServerSideEncryptionCustomerMethod AES256
    Write-Host "Uploaded and encrypted with key = $Base64key"
}
catch [system.exception] 
{
	Write-Host  "Error: " $_.Exception.Message
}

#
# Download the file back to the current directory
#
try 
{
    Read-S3Object -EndpointUrl https://s3-oslo.educloud.no -BucketName $bucket -Key $objectkey -File $restoredfile  -ServerSideEncryptionCustomerProvidedKey $Base64key -ServerSideEncryptionCustomerMethod AES256
    Write-Host "The file $restoredfile has been copied to the cwd"
}
catch [system.exception]
{
	Write-Host  "Error: " $_.Exception.Message
}

Please note In the above example subsequent calls to $Aes::GenerateKey does not generate new keys.

ACLs

A newly created bucket is per default only accessible by the user who created the bucket, refered to as the owner of the bucket. So any kind of HTTP request to the bucket or object, only evaluates to ALLOW if the request comes from the bucket owner. Any other request will be denied.

In most scenarios though, a bucket needs different kinds of permissions, for example that the bucket is only accessible from certains IPs, and by certain users.

In Cloudian Hyperstore, there are three different ways of controlling authorization for a bucket, S3 ACLs, IAM policies and bucket policies. Since we are requiring that all projects use IAM users, you will have have to use IAM policies to secure your buckets, as bucket policies do not work with IAM users, and S3 ACLs should be avoided.

S3 ACLs

S3 ACL is a legacy way of enforcing access to a bucket, and one should avoid using it. Instead use bucket policies or IAM policies, both are more readable and enable fine-grained policies. When you use the CMC GUI, and set permissions on a bucket through Buckets & Objects → Buckets → Properties → Bucket Permissions, you are in fact setting up S3 ACL for the bucket. Follow this link on the CMC to read more about these permissions Set Custom S3 Permissions for a Bucket

IAM Users and Groups

You can create one or more IAM (Identity and Access Management) users and groups under your user. These users are only usable for buckets owned by your user. This is a way of enabling specific applications to use the object storage service without giving them the S3 access credentials associated with your root account and without giving them the full range of permissions associated with your root account. This can be useful if you want to have a bucket each for development, test and production, in which you use separate users for each bucket.

IAM Policy

By default, a IAM user has zero permissions, so if you create a bucket called test_s3_bucket, that bucket is per default private, and only available for the owner that created the bucket. So if you want to let the IAM-user testenviroment_iam_user1 get access to the bucket test_s3_bucket, you need to create a IAM Policy. Note that only the root user ie. your Cloudian user, owns all the files in the bucket, while the IAM-user is only given permissions to operate on the files and the bucket(s).

This newly created policy can then be attached to a IAM-user, giving the user acces to the bucket test_s3_bucket, from a certain range of IPs.

First create the Policy. In the following policy you are making sure that all S3-operations can only be done from a set of IPs, defined in the value list for the key aws:SourceIP. If possible, restrict the access for a IAM user based on the IPs of the servers where the user will be accessing the bucket from.

New-IAMPolicy -EndpointUrl https://iam.educloud.no:16443  -PolicyName AllowIAMuserExample -PolicyDocument '{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Sid": "AllowAllS3OperationsOnBucket",
             "Effect": "Allow",
             "Action": "s3:*",
             "Resource": [
                 "arn:aws:s3:::test_s3_bucket",
                 "arn:aws:s3:::test_s3_bucket/*"
             ],
             "Condition": {
                  "IpAddress": { "aws:SourceIp": ["129.240.2.75/32","129.240.6.27/32","193.156.42.128/25"]}
             }
         }
     ]
 }'

Then attach the policy to the user:

Register-IAMUserPolicy -EndpointUrl https://iam.educloud.no:16443 -UserName testenviroment_iam_user1 -PolicyArn arn:aws:iam::8f106a0c74098f36e1e9c233154ddae2:policy/AllowIAMuserExample

Check that the policy is attached:

PS S3_examples > Get-IAMAttachedUserPolicyList  -EndpointUrl https://iam.educloud.no:16443 -UserName testenviroment_iam_user1

PolicyArn                                                                PolicyName
---------                                                                ----------
arn:aws:iam::8f106a0c74098f36e1e9c233154ddae2:policy/AllowIAMuserExample AllowIAMuserExample

Bucket Policy

Bucket policies are used when you want to control who can access the bucket, be it from certain users, all users AKA public non-authenticated users, or from certain IPs, or filtering based on request headers. To see which options can be used in a Bucket Policy, see the following guide Hyperstore documentation - PutBucketPolicy.

Here we have included a couple of examples that might apply to most users, but if you need help with more advanced setup, please contact lagringstjenester@usit.uio.no.

Restrict access to a bucket for IP range/hosts

The following Bucket Policy makes sure that S3-actions on the bucket is only available for the IPs defined in the values for the key aws:SourceIp. Note that single IPs should still add the subnet suffix. So if you wanna only allow 129.240.6.149, it should still have /32 added.

Write-S3BucketPolicy -BucketName test_s3_bucket  -EndpointUrl https://s3-oslo.educloud.no  -Policy '{
   "Id": "OnlyAllowedIps",
   "Statement": [
     {
       "Sid": "OnlyAllowedIps",
       "Effect": "Deny",
       "Principal": "*",
       "Action": "s3:*",
       "Resource": [
            "arn:aws:s3:::test_s3_bucket",
          "arn:aws:s3:::test_s3_bucket/*"
       ],
       "Condition": {
        "NotIpAddress": { "aws:SourceIp": ["129.240.2.0/27","129.240.6.149/32","193.156.42.128/25"]}
       }
     }
   ]
 }'

Allow another Cloudian user access to bucket

If you want to add access for another user that is not the owner of the bucket, but has a non-IAM user in Cloudian, that can be done with the following bucket policy. After adding this policy to your bucket, all newly created objects in the bucket test_s3_bucket, can have all S3 actions performed on the new object, by the CanonicalUser with id ec224760c075c050472423eb347b203b, as well as the bucket owner. Any objects added to the bucket without the canned ACL header bucket-owner-full-control gets denied.

The reason for adding the header bucket-owner-full-control, is because without it, the other user is able to add files to the bucket, without the files being readable for the bucket owner.

Write-S3BucketPolicy -BucketName test_s3_bucket  -EndpointUrl https://s3-oslo.educloud.no -policy '{
    "Id": "AllowOtherUserAccess",
     "Statement": [
    {
        "Sid": "DenyWrongHeader",
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:*",
        "Resource": [
             "arn:aws:s3:::test_s3_bucket",
           "arn:aws:s3:::test_s3_bucket/*"
        ],
        "Condition": { 
            "StringNotEquals": { 
              "s3:x-amz-acl": "bucket-owner-full-control" 
            }
        }
    },
      {
        "Sid": "AllowOtherUserAccess",
        "Effect": "Allow",
        "Principal": {"CanonicalUser" : "ec224760c075c050472423eb347b203b"},
        "Action": "s3:*",
        "Resource": [
             "arn:aws:s3:::test_s3_bucket",
           "arn:aws:s3:::test_s3_bucket/*"
        ]
      }
    ] 
}'

Note that only root users, ie non-IAM users are the only ones that can be used in a Bucket Policy. The bucket owner is also the only one that can apply, change and delete bucket policies.

Security requirements when storing data

All human or machine generated data can be classified in one of four classes, green, yellow, red, black. For more information about these classes see https://www.uio.no/english/services/it/security/lsis/data-classes.html.

It is the responsibility of the user to ensure that the proper security attributes are enforced for the different data-classes.

Data class green

No restrictions are required. Data can be shared using public URLs.

Data class yellow

Data class red

Data class black

Storage of black data is not permitted.