Object Storage ¶

Important

The object storage is only available as a PILOT SERVICE. Do not use it for important data. The service is available in the BGO and OSL regions.

Warning

At this point in time, due to physical constraints on the backing hardware, the object storage service is not suited for very high volume usage. If you need either a high volume of objects, a high volume of transactions, or a high volume of storage capacity, please reach out to the NREC team in our slack channel, or via an email to support@nrec.no.

Access ¶

To gain access to the object storage pilot service please send an email to support@nrec.no and tell us the name of the project that should have access to object storage.

Usage ¶

Our object storage support two different APIs:

The endpoint URLs are

https://object.api.bgo.nrec.no
https://object.api.osl.nrec.no

for the BGO and OSL regions respectively.

Dashboard (Swift)¶

This is a simple web GUI where you can create containers (buckets) and upload and download files.

Openstack CLI (Swift)¶

You will need to install the python swiftclient for openstack. You will then be able to create containers (buckets) and upload and download files.

For Fedora and RHEL, Alma Linux, Rocky Linux and CentOS Stream 8.x and later:
```
# yum install python3-swiftclient
```

Amazon S3 Tools: s3cmd (S3)¶

Warning

Make sure you have export OS_INTERFACE=public in your RC-file used with openstack cli

To use the S3 API you will first need to create EC2 credentials. With openstack cli (version 3.8+) run:

openstack ec2 credentials create
openstack ec2 credentials list
openstack ec2 credentials show <id>

Install s3cmd:

For Fedora and RHEL, Alma Linux, Rocky Linux and CentOS Stream 8.x and later with EPEL enabled:
```
# yum install s3cmd
```

and create a config file ~/.s3cfg

[default]
access_key = <access_key>
host_base = object.api.bgo.nrec.no
host_bucket = object.api.bgo.nrec.no
secret_key = <secret_key>

See Amazon S3 Tools for more information

AWS CLI (S3)¶

AWS CLI can be installed on Fedora and RHEL, Alma Linux, Rocky Linux and CentOS Stream 8.x and later with EPEL enabled:

# yum install awscli

You need to create EC2 credentials, as described for the s3cmd usage. Configuration of the AWS CLI may be performed in several ways. A simple method is to use environment variables:

export AWS_ACCESS_KEY_ID=<access_key>
export AWS_SECRET_ACCESS_KEY=<secret_key>
export AWS_ENDPOINT_URL=https://object.api.<region>.nrec.no
export AWS_DEFAULT_REGION=None

See AWS CLI for more information.

Note

AWS_ENDPOINT_URL was introduced in AWS_CLI 2.13 (2023) and may not be available in the version supplied from the OS aws cli package. The endpoint url can be set by supplying it as a parameter:

aws --endpoint-url $AWS_ENDPOINT_URL command...

Public Access (S3)¶

To access a public object you will first set public ACL. The URL to access it will be on the form:

<endpoint>/<project_id>:<bucket>/<path-to-object>

Example:

https://object.api.bgo.nrec.no/<project-id>:<bucket>/<object>

Object Locking (S3)¶

Using the S3 object lock mechanism, you can use object lock concepts like retention period, legal hold, and bucket configuration to implement Write-Once-Read_Many (WORM) functionality. In the following example we will use AWS CLI to create a bucket and configure object locking for new objects put there.

Important

The object version(s), not the object name, is the defining and required value for object lock to perform correctly to support the GOVERNANCE or COMPLIANCE mode. You need to know the version of the object when it is written so that you can retrieve it at a later time.

First, create a bucket and enable object locking for the new bucket.

aws s3api create-bucket --bucket myimportantbackup --object-lock-enabled-for-bucket

Set a retention period for the bucket. In this example, we set a 30 days retention period. This will be the default for new objects put into this bucket.

$ aws s3api put-object-lock-configuration --bucket myimportantbackup \
--object-lock-configuration '{ "ObjectLockEnabled": "Enabled", \
"Rule": { "DefaultRetention": { "Mode": "GOVERNANCE", "Days": 30 }}}'

Note

You can choose either the GOVERNANCE or COMPLIANCE mode for the RETENTION_MODE in S3 object lock, to apply different levels of protection to any object version that is protected by object lock.

In GOVERNANCE mode, users cannot overwrite or delete an object version or alter its lock settings unless they have special permissions.

In COMPLIANCE mode, a protected object version cannot be overwritten or deleted by any user. When an object is locked in COMPLIANCE mode, its RETENTION_MODE cannot be changed, and its retention period cannot be shortened. COMPLIANCE mode helps ensure that an object version cannot be overwritten or deleted for the duration of the period.

Put an object into det bucket with a specific retention time set:

$ aws s3api put-object --bucket myimportantbackup --object-lock-mode GOVERNANCE \
--object-lock-retain-until-date "2023-12-30" --key governance-upload --body /tmp/testfile1

{
    "ETag": "\"224585ee94754d3d9095726275da863b\"",
    "VersionId": "BT5ILU5W8KCqi5BXHOptVDFb.JyHXFc"
}

Now upload another object, using the same key:

$ aws s3api put-object --bucket myimportantbackup --object-lock-mode GOVERNANCE \
--object-lock-retain-until-date "2023-12-30" --key governance-upload --body /tmp/testfile2

{
    "ETag": "\"404290d1d1cad1390cd77a0a56c960ec\"",
    "VersionId": "ynk5tyro6BufAQaKfPA0yg3vY6lKAh6"
}

List the object versions from the bucket:

$ aws s3api list-object-versions --bucket myimportantbackup

{
  "Versions": [
    {
        "ETag": "\"404290d1d1cad1390cd77a0a56c960ec\"",
        "Size": 126720,
        "StorageClass": "STANDARD",
        "Key": "governance-upload",
        "VersionId": "ynk5tyro6BufAQaKfPA0yg3vY6lKAh6",
        "IsLatest": true,
        "LastModified": "2023-12-07T14:05:42.423000+00:00",
        "Owner": {
            "DisplayName": "some-nrec-project",
            "ID": "a4549966f6e94851bb991c34aff828f0$a4549966f6e94851bb991c34aff828f0"
        }
    },
    {
        "ETag": "\"224585ee94754d3d9095726275da863b\"",
        "Size": 459164,
        "StorageClass": "STANDARD",
        "Key": "governance-upload",
        "VersionId": "BT5ILU5W8KCqi5BXHOptVDFb.JyHXFc",
        "IsLatest": false,
        "LastModified": "2023-12-07T13:57:10.669000+00:00",
        "Owner": {
            "DisplayName": "some-nrec-project",
            "ID": "a4549966f6e94851bb991c34aff828f0$a4549966f6e94851bb991c34aff828f0"
        }
    }
  ],
  "RequestCharged": null
}

List only the latest objects:

$  aws s3api list-objects --bucket myimportantbackup
{
    "Contents": [
        {
            "Key": "governance-upload",
            "LastModified": "2023-12-07T13:59:58.806000+00:00",
            "ETag": "\"404290d1d1cad1390cd77a0a56c960ec\"",
            "Size": 126720,
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "some-nrec-project",
                "ID": "a4549966f6e94851bb991c34aff828f0$a4549966f6e94851bb991c34aff828f0"
            }
        }
    ],
    "RequestCharged": null
}