ZFS as a swift object store

 This blog post goes through a feature of the ZFS Appliance that has been around for at least 3 years now. The Openstack Swift Object store.

When looking at the S3 API, and the OCI API, I forgot all about where it started.. With the Swift API.

I will go through the 3 APIs, and how they came about (from what I can find by reading through articles)..

It all started with the Swift API. Swift (V1) has simple authentication and a simple interface.

A URI to manage/access objects has the format of

HTTP://{object store server}/object/v1/{Account}/{bucket name}/{object name}.

In the case of ZFS, 

• Account - this is the share name.. "/export/swiftshare" for example
• Bucket name - The name of the bucket that was created
• Object name - name of the object.

Authentication with Swift while using curl is typically a 2 step process.  

First the authorization URI is called

HTTP://{object store server}/auth/v1.0/{Account}

The username and password is sent with the authentication URI.  The URI then returns an auth token which is used in the curl command line to manage buckets/objects.

Username/password authentication (v1.0) is one of the 3 choices.

1. Local username/password created on the ZDLRA.
2. LDAP user ZFS ties to
3. Keystone authentication server.

For all the testing I am doing on my ZFS simulator, I use a local user.

Before I go into how to configure and use the Swift interface on ZFS, I'll share what I was able to find out.

The Swift API has some limitations, and these limitations is what drove the move to S3.

As you probably noticed, the authentication and tracking of objects does not have enough details to support the segregation of users, and billing.

The S3 API takes the Swift API, and adds the ability to create separate tenants, set up billing, etc. All the things an enterprise needs to do.

With S3, you probably noticed that the authentication layer changed. It is based on secret name/secret rather than a username/password returning an auth token..

Well lets go through what it takes to configure the Swift interface.

First, most of the steps around configuring ZFS for an object store, I documented in my previous blog posts. 

If you look the posts below you see the steps on configuring a share,creating a local user on ZFS, and configuring the http service.

ZFS Appliance - Your on-premise cloud store

Managing authentication for a ZFS Object Store

For Swift, I will just go the steps specific to Swift.

All I need to do is enable swift. That's it !

Swift gets enabled just like enabling the S3 API, and the OCI API. Because I do not have a Keystone Authentication Server (which would be the OpenStack Identity Service), I didn't fill those values in.

NOTE: Authentication for swift is a little different from S3, or OCI. Both of the other APIs do not tie directly to the local user.  S3 uses "secrets", and OCI uses a PEM file, and a Fingerprint.

Accessing my Swift bucket.

First some links to documentation that will give you examples of these ways of connecting.

Swift Guide for ZFS OS 8.8 release  -- Current release as of writing.

Using ZFS as an object store  -- This is old, but has a lot of detail and great detail

API Guide OS 8.8 for Swift Docs -- Current documentation guide

Also, since the Swift implementation is OpenStack, there is a lot of examples and documentation (non-oracle) available across the web.

I was able to access my bucket one of 3 ways

The first 2 ways are very similar

Swift command line tool - python based tool to connect to swift and manage buckets

CURL - Command line took similar to swift.

In order to create a bucket, and upload an object ......

First I execute the  curl command to get the authentication token.

NOTE: my ZFS emulator is 10.0.0.110 and my share is /export/short

curl -i http://10.0.0.110/auth/v1.0/export/short -X GET -H "X-Auth-User: oracle" -H "X-Auth-Key: oracle123"

HTTP/1.1 200 OK
Date: Thu, 21 Jan 2021 18:54:38 GMT
Server: Apache
X-Content-Type-Options: nosniff
X-Storage-Url: http://10.0.0.110:80/object/v1/export/short
X-Auth-Token: ZFSSA_522d6355-9056-4a95-9060-c88648007993
X-Storage-Token: ZFSSA_522d6355-9056-4a95-9060-c88648007993
Content-Length: 0
X-Trans-Id: tx62e2f031f21640c29a2bf-006009cdee
Content-Type: text/html; charset=utf-8

Next I execute create a bucket .

From the output  above I can get the "Auth Token", and the Storage URL to manage the object store in curl. Note that the Auth Token will expire.

Create a container in curl

curl -i http://10.0.0.110:80/object/v1/export/short/bucketswift -X PUT -H "Content-Length: 0" -H "X-Auth-Token: ZFSSA_522d6355-9056-4a95-9060-c88648007993"

Create a container in swift

swift post container -A http://10.0.0.110:80/object/v1/export/short -U oracle -K oracle123

That's all there is to it with the Swift Object Store on ZFS.

Managing an Object Store on ZFS

 This blog post will cover how to access the object store on ZFS to create buckets and upload files. For S3, I am using Cloudberry, which I downloaded here. For OCI, I am using the OCI cli tool.

S3 access to ZFS

This is the easiest, since the S3 object store on ZFS is an S3 compatible interface.

In cloudberry add a new account and use the following for input.

Note that you need to enter the 4 fields above.

• Display Name -- What you want to call the new account entry
• Service Point  -- This is the ZFS interface for S3 in the form of 

HTTP://{the IP of the ZFS}/s3/v1/{share name}

• Access Key  -- This is the name you gave the S3 access key, when you added it to the ZFS
• Secret Key   -- This is the long string of characters that was returned by ZFS when you created the key.

That's it ! You can now use cloudberry to create buckets, upload files, sync object stores etc.

OCI access to ZFS

Install the CLI

Like the Oracle cloud, there is currently (as of me writing the blog post), no GUI tool like cloudberry that will connect to an OCI object store.  When connecting to the Oracle cloud, you can access the OCI object store through the S3 interface, but this is not possible on ZFS. Both the OCI and S3 object store are independent and cannot access buckets etc. in the other object store.

In order to access ZFS through OCI we start with downloading the OCI cli tool. Documentation on how do this can be found here.

In my install, I took the easy route  (and since I had a Ubuntu client with root access to play with). I installed it directly using "sudo pip install oci-cli"

Create a config file.

Once you have the OCI cli installed we need to set up a configuration file to be used.

The default file is ~/.oci/config, but this location can be changed when using the command if you access multiple OCI installations.

This is the contents of my file.

1 2 3 4 5 6 7

[DEFAULT] user=ocid1.user.oc1..oracle fingerprint=1e:6e:0e:79:38:f5:08:ee:7d:87:86:01:13:54:46:c6 key_file=/home/oracle/opc/oracle_private.pem tenancy=ocid1.tenancy.oc1..nobody region=us-phoenix-1 pass_phrase = oracle

Now to walk through each line.

1. This identified the entry. Since the config file can contain entries for multiple OCI locations, this entry is identified as the default entry to use (If I don't specify one).

2. This is the user ID.  Since I am using ZFS, the format is "ocid1.user.oc1..{zfs user}"

3. This is the fingerprint. I mentioned in the last blog post that this will be needed. This fingerprint identifies the API public_key entry on ZFS to use when matching the private API key being sent

4. This is the private key file. This contains the private API key that matches the public key that was added to the ZFS.

5. This is unimportant to ZFS, but is required to be set. Use the entry above.

6. Like #5. this is not used by ZFS but is needed by the OCI client.

7. This is optional. If the API private key was created with a pass_phrase, this the pass_phrase that matches the private key.

Create a bucket on OCI.

Almost there now ! We have everything in place for authentication, and we are ready to create an OCI bucket on ZFS for storing data.

The command is 

oci os bucket create --endpoint {OCI object store location} --namespace-name {location on the object store} --compartment-id {compartment in OCI} --name {new bucket name}

Now let's walk through what the parameters will be for ZFS

--endpoint               -> For my ZFS appliance, it is the url + oci

--namespace-name  -> This is the share on the ZFS.  "/export/short" in my config.

--compartment-id    -> This is also the share on the ZFS.  "/export/short" in my config.

--name                     -> the name of the bucket I want to create.

For my configuration below is the command and the output.. I now have a bucket created, and I am able upload data !

oci os bucket create --endpoint http://10.0.0.110/oci --namespace-name export/short --compartment-id export/short --name mynewbucket 

{
  "data": {
    "approximate-count": null,
    "approximate-size": null,
    "compartment-id": "export/short",
    "created-by": "oracle",
    "defined-tags": null,
    "etag": "a51c8ecbf1429f95b446c4413df9f494",
    "freeform-tags": null,
    "id": null,
    "is-read-only": null,
    "kms-key-id": null,
    "metadata": null,
    "name": "mynewbucket",
    "namespace": "export/short",
    "object-events-enabled": null,
    "object-lifecycle-policy-etag": null,
    "public-access-type": "NoPublicAccess",
    "replication-enabled": null,
    "storage-tier": "Standard",
    "time-created": "2021-01-05T16:15:05+00:00",
    "versioning": null
  },
  "etag": "a51c8ecbf1429f95b446c4413df9f494"
}

ADVANCED TOPIC -- SSL with OCI CLI

Now let's say I want to encrypt my connections to OCI and use the HTTPS server available on ZFS.

First I need to create a file containing the certificate. I can get the certificate by executing.

openssl s_client -showcerts -connect 10.0.0.110:443

This returns a lot of information, but within the output I can see the certificate, and I can copy and paste into a file.

Certificate chain
 0 s:CN = 10.0.0.110, description = https://10.0.0.110:215/#cert
   i:CN = 10.0.0.110, description = https://10.0.0.110:215/#cert
-----BEGIN CERTIFICATE-----
MIIDXDCCAkSgAwIBAgIIW+387wAAAAIwDQYJKoZIhvcNAQELBQAwPDETMBEGA1UE
AwwKMTAuMC4wLjExMDElMCMGA1UEDQwcaHR0cHM6Ly8xMC4wLjAuMTEwOjIxNS8j
Y2VydDAeFw0wNjAyMTUxODAwMDBaFw0zODAxMTkwMzE0MDdaMDwxEzARBgNVBAMM
CjEwLjAuMC4xMTAxJTAjBgNVBA0MHGh0dHBzOi8vMTAuMC4wLjExMDoyMTUvI2Nl
cnQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCyfgnTMtxgPEtrmCpB
J4gHngdqpRQWnUXA/OtWGymXME/+gAd5Q/8LZ74VpkHIwk3T7z/+tJVgL1HFmmbi
ZRSsXfSUgOBHm0doPn3VGbykz5MHWm3HHwjpPwvVhyeuVEkUfs/yiZ9B1WZrkr6U
ePNKlkbdL1VN5q2zuLdJ7+jn3HIiSS9j10i7HQVFEuzUAGdt3q0rp2MwaxSP6+cZ
hzMaI5IGBHuVkw2fGX1RdDB6uZpFEEhRSHURr5/3d+UgOprkMKp8Wph3kH0E2Nha
tGpSn2/6NM/Up/nDjfu2Dxm9A2aCwC56ShTckTTxE2HrgfSE9r/vEnkJEdSemH+X
9BuRAgMBAAGjYjBgMCYGCWCGSAGG+EIBDQQZFhdBdXRvbWF0aWNhbGx5IGdlbmVy
YXRlZDA2BgNVHREBAf8ELDAqggoxMC4wLjAuMTEwhwQKAABuhhZodHRwczovLzEw
LjAuMC4xMTA6MjE1MA0GCSqGSIb3DQEBCwUAA4IBAQAqxZk2knSBinWvTADkrvuS
C3vkeLyOLCRwABnGzZV80AAZ3tSVZt2JPXtg8uAVEj29J4VFw/I7HuneGL/faW9q
Qr9h+2WjvoT+m6lIfwELeaomZhkrLmJomGqSP1wfw5jaw3cpt0yOeS4RWUYb9eEe
bTH6laFBtSdbaI/uHslxpJwNRDwn8zBpAWmZk83UQ5CytH37yrFPRoHQWp+OqF+V
GYTPA4drxQ00nuelNfpHWMCjjMr0WxFz5rNJPMOAe2W1Xcr/MM1h04kGVwRtYsC0
4izqKtfiOHt0wMkSbYuSj1tIzdEzjVmxNSS7nv/znrMt+6SsdYQHMmaJ4+wHlJo4
-----END CERTIFICATE-----
---
Server certificate
subject=CN = 10.0.0.110, description = https://10.0.0.110:215/#cert

I want to copy the certificate including the "BEGIN CERTIFICATE" and "END CERTIFICATE" lines into a file. 

I now need to set my environment to see the certificate file and use it. In my case "/home/oracle/opc/wallet_cloud/zfs.cer"

export REQUESTS_CA_BUNDLE=/home/oracle/opc/wallet_cloud/zfs.cer

I can now view the buckets in my object store, and upload files encrypted.

oci os bucket list --endpoint https://10.0.0.110/oci --namespace-name export/short --compartment-id export/short 
{
  "data": [
    {
      "compartment-id": "export/short",
      "created-by": "oracle",
      "defined-tags": null,
      "etag": "a51c8ecbf1429f95b446c4413df9f494",
      "freeform-tags": null,
      "name": "mynewbucket",
      "namespace": "export/short",
      "time-created": "2021-01-05T16:15:05+00:00"
    }
  ]
}

The OCI documentation should give you everything you need to upload/download objects within a bucket.

Managing authentication for a ZFS Object Store

 As promised, I am continuing my blog series on how to work with ZFS as a cloud store.

My first blog post went through the steps of how to configure ZFS as an object store.

This post will go through how to create the authentication keys/secrets to access  Object Store.

OCI/S3 user management

The first thing to do is to create a user on the ZFS that will be used as the owner of the object store.

In my case I am going to use the "oracle" account, and ensure that the GUID is the same as the GUID I use on all my DB servers.

NOTE: Most of the information I used to go through the process was from this document.

Create the user

Start by logging into the web interface for the ZFS appliance and navigate to Configuration -> users .

Once on this page, click on the + sign next to users to create a new user. In the example, I had already created the Oracle user.

Now on the create user page, ensure the user is a "local" user, and the "User ID" is the same as the GUID I normally use for Oracle. After entering the information, click on "ADD" in the upper right hand corner to add the user.

Change share ownership

Now that I have the "oracle" user created, I am going to change ownership on my share that will be my object store.

In order to do this, I am going navigate to Shares -> SHARES . I see my object store share and highlight it and click on the pencil icon to edit the share.

I am now on the detail screen for my share, and I navigate to the Shares -> SHARES -> Access page.

On this page, I change the user to be Oracle, and ensure the permissions are open enough. Once this change is made click on the apply button in the top right hand corner.

OCI Authentication.

Create the API keys

Now we need to add to add the API key to authenticate the user to the Object store.
In the case of an OCI bucket, authentication is performed by using an X.509 certificate.
This is the same authentication used for an OCI bucket in the Oracle Cloud.

Instructions on how to create an API signing key can be found here.

In my case I used the linux command instructions, and the openssl command to create both a private and public key.  When completed, I had 2 files.

/home/oracle/opc/oracle_public.pem  

        -----BEGIN PUBLIC KEY-----

        sdfa23

        ....

        -----END PUBLIC KEY-----

/home/oracle/opc/oracle_private.pem  

       -----BEGIN RSA PRIVATE KEY----

        5dfgsret345

        ....

        -----END RSA PRIVATE KEY-----

Add the API key

Now that we generated the Keys, lets add them to the share so we can access the OCI object store.

We start by going to Configuration-> Services -> HTTP. Click on HTTP to bring up the next page.

On this page, we want to go to the OCI tab and add a new key.

On the "New Key" window that popped up, add the Oracle user, and paste in the public key. Once everything is entered click on add to create the key.

Once added make note of the Fingerprint.

S3 Authentication.

Create the Secret.

Unlike OCI, S3 authentication is done through a "secret".  The use of a "secret" is similar to the idea of a Key and a Passcode. You create a new access  key for the user, and you then you are provided a long string that is the "passcode" for this key.

Like creating API key for OCI, we start by going to the HTTP service.

This time we go the S3 tab under HTTP and click on the + sign to add a key.

Enter the oracle user, and give your key a name. Once complete, click on ADD to create the key.

Now you will see a window with the Secret Key.

SAVE THIS KEY. you will not be given this key again. You can copy and paste it, but better yet, save it in a file.

Authentication for S3 and OCI.

When you completed both of these actions you will have 2 authentication pieces that we will use to create buckets and access the object store in future blog posts.

S3 - You have an "ACCESSS_KEY" and a "SECRET_KEY" that will be used.

OCI - You have a file containing the private_key, public_key, and the fingerprint associated with the public_key to identify it.