If you are using the Oracle Object Store as part of the Oracle Public Cloud, "rclone" is an open source tool you can use to make things easier.
One of the things I really like about RCLONE is that it provides a command line like interface that is easy to use. If you have looked at the OCI cli tool, it requires a myriad of parameters. Below is the command I was using with OCI to view my list of buckets (I obfuscated some of the values).
oci os bucket list --endpoint https://objectstorage.us-ashburn-1.oraclecloud.com --namespace-name id20xxxxxofo --compartment-id ocid1.compartment.oc1..aaaaaaxxxxxxxxxxxxxxxxxcpqyvzzb4ykd3tyq --config-file ~/.oci/natdconfig
In order to use the OCI tool, I had to constantly keep a text file open to copy and paste commands.
In comparison, this is the command to list the buckets in my object store using rlcone.
rclone ls oci_bucket:
1) Configure compatibility for an S3 interface in the Public cloud.
In your public cloud council, in the top right hand corner, click on the "silhouette" that controls you settings. in the pull down menu click on "user settings" to bring up the window to configure you resources. Once there, click on "Customer Secret keys" and then "Generate Secret Key" bring up the window to add a secret key.
On this window give your secret key a name (like S3Key" in my case). When you click the "Generate Secret key" button, it will give you secret associated with key. SAVE THIS.
Once complete, you will have 2 items associated with your account
NAME: S3Key or whatever you named your key.
Access Key: ddddddddddddeeeeeeeeeffffffffggggg A uniquely identified key ID
Secret Key : dd32234sdwercfwe A system generated "secret"
2) Download rclone.
This can easily be done from the RCLONE.ORG site.
Note: You chose the platform you want to execute rclone on, then download the .zip file.
The .zip file contains the execute, and documentation.
Copy the "rclone" executable to the location of your choice and make it executable.
2) Configure Rclone.
You start by executing "rclone config". This will create a configuration file in ~/.config/rclone/rclone called rclone.conf. This is an interactive interface that will set the correct configuration parameters to be used.
This is an example of what I entered to connect to my Object Store using the S3 interface.
--> rclone config
Give this entry a unique name to identify the S3 object store.
Name> oci_s3 <-- my entry name in the config file
Type of storage to configure.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value..4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc)..
Storage> 4 <-- 4 identifies this as an S3 compatible object store
Choose your S3 provider.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value..13 / Any other S3 compatible provider..
provider> 13 <-- 13 identifies this as "other" S3 compatible object store
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).Only applies if access_key_id and secret_access_key is blank.Enter a boolean value (true or false). Press Enter for the default ("false").Choose a number from below, or type in your own value1 / Enter AWS credentials in the next step\ "false"
env_auth> 1 <-- 1 to identify that we are using "AWS compatible Key" for authentication
AWS Access Key ID.Leave blank for anonymous access or runtime credentials.Enter a string value. Press Enter for the default ("").
access_key_id> ddddddddddddeeeeeeeeeffffffffggggg <-- This is the Access key ID that was generated from my name in the public cloud
AWS Secret Access Key (password)Leave blank for anonymous access or runtime credentials.Enter a string value. Press Enter for the default ("").
secret_access_key> dd32234sdwercfwe --> The system generated key associated with my access key
Region to connect to.Leave blank if you are using an S3 clone and you don't have a region.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value1 / Use this if unsure. Will use v4 signatures and an empty region.\ ""
region> --> Leave blank
Endpoint for S3 API.Required when using an S3 clone.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value
endpoint> {namespace}.compat.objectstorage.{region}.oraclecloud.com --> Note that you will need to fill in your namespace from your account, and ensure the region is correct for the URL.
Location constraint - must be set to match the Region.Leave blank if not sure. Used when creating buckets only.Enter a string value. Press Enter for the default ("").
location_constraint> --> Leave blank
Canned ACL used when creating buckets and storing or copying objects.This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-aclNote that this ACL is applied when server side copying objects as S3doesn't copy the ACL from the source but rather writes a fresh one.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value1 / Owner gets FULL_CONTROL. No one else has access rights (default).\ "private"
acl> --> Leave blank
Edit advanced config? (y/n)y) Yesn) No (default)
y/n> --> Leave blank
Remote config--------------------[oci_s3]type = s3provider = Otherenv_auth = falseaccess_key_id = S3_Keysecret_access_key = ddddddd...endpoint = xxxxxxx.compat.objectstorage.us-ashburn-1.oraclecloud.com--------------------y) Yes this is OK (default)e) Edit this remoted) Delete this remote
y/e/d> y --> y to save this entry
3) Validate rclone.
Now let's verify what got create.
> cat ~/.config/rclone/rclone.conf
[oci_s3]type = s3provider = otherenv_auth = falseaccess_key_id =dd32234sdwercfwesecret_access_key = dddddxxxxxxendpoint = xxxxxxx.compat.objectstorage.us-ashburn-1.oraclecloud.comacl = authenticated-read
That's It. In my case
- the entry is "oci_s3"
- The access key for S3 is dd32234sdwercfwe"
- The secret associated with my S3 key is "dddddxxxxxx"
- The end point I am connecting to is "xxxxxxx.compat.objectstorage.us-ashburn-1.oraclecloud.com"
- "xxxxxxx" is my namespace
- "us-ashburn-1" is my region
4) Using rclone.
Now with rclone I can execute commands against my object store that are more linux like.
rclone mkdir oci_s3:mybucket --> will create a bucket named "mybucket"
rclone ls oci_s3: --> will list all my buckets
rclone ls oci_s3:mybucket --> will list all the objects in my bucket.
I can also use it to copy to and from my bucket.
rlcone copy /home/oracle/myfile.txt oci_s3:mybucket --> copies the file to the bucket.
Finally, a great command is sync to synchronize the contents of my on-prem to the cloud
rlcone sync /home/oracle/mydir/ oci_s3:mybucket --> this will sync the two locations
Now how fun with it !!
No comments:
Post a Comment