How-to Sync Data Between AWS S3 Storage and (Elastic Block Store) EBS

How-to Sync Data Between AWS S3 Storage and (Elastic Block Store) EBS

S3Sync is a tool written in Ruby to synchronize system folders (EBS) and S3 buckets. In this guide we will shown you how to sync your EBS data with S3 bucket and vice versa.

We will run an EBS-backed Linux instance and sync data between our instance and S3 bucket.

To execute the S3Sync from your instance, you need Ruby (version 1.8.7 or above ) installed in your local machine. If you are running a Windows instance, download and install Ruby from   http://rubyinstaller.org/downloads/  for a Windows local machine.

We will work with our Linux based instance.

1.  Connect to your Linux instance as described in (“Connecting to AWS Linux instance from a Windows Machine” – coming soon).

2.  Install Ruby by running the command: sudo yum install ruby.

3.  Check if Ruby is installed and running correctly by running the command: ruby –version.

4.  Download the S3Sync tool from http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz.

5.  Unzip the tar by running the command: sudo tar zxvf s4sync.tar.gz.

6.  Now go to the s3sync folder (d /home/ec2-user/s3sync).

7.  Create a new folder “certs”  by running the command: mkdir certs.

8.  Download a certificate file from http://mirbsd.mirsolutions.de/cvs.cgi/src/etc/ssl.certs.shar by running the command: sudo wget http://mirbsd.mirsolutions.de/cvs.cgi/src/etc/ssl.certs.shar.

9.  Now go back to the s3sync folder.

10.  Create the  s3config.yml file from the example file available.

11.  Modify the the  s3config.yml file, providing the right values as data (use vi editor to modify).

12.  Once the above setup is done, run the command: ruby s3cmd.rb listbuckets.  If the parameters’ setup is complete,  all S3 buckets in the US-Region are listed.

13.  Next, we will create a folder and some temporary data in our EC2 instance.

14.  We have created the testS3sync directory and have a few files available. Now, we can copy data from the testS3Sync folder to our bucket  s3syncTest by running the command:
sudo ruby s3sync.rb -v -r –make-dirs /home/ec2-user/testS3Sync/ s3syncTest:

16.  The data is copied to the S3 bucket  – s3syncTest. Check your bucket, it shows both the files available from EBS to S3 bucket.

17.  Let’s add a few more files to S3 bucket and download them to EBS.

18. From your instance, run the command: sudo ruby s3sync.rb -v -r –make-dirs s3syncTest: /home/ec2-user/testS3Sync/.

19.  The file is downloaded to your EBS volume.

 That is how to sync data between S3 and EBS using S3sync command.

Note:   S3Sync is a free tool but we recommend that you read the license terms and guidelines.

Keywords: Amazon AWS Cloud Services, EBS, S3 Bucket, HA, Availability, Backup, DR,  EBS-backed Linux instance, AWS Console,  Linux based Instance

There is 1 comment .

danneh3826 —

Or you could just use “s3cmd sync”, which does the same thing, built in Python (because, you know, Ruby sucks), and installable via “apt-get install s3cmd” or “yum install s3cmd”. Run s3cmd –configure, provide your AWS access and secret keys, and you’re good to go!

You must be to post a comment.

* As a bonus, you'll receive our weekly newsletter!

Hitchhiker's Guide to The Cloud

Newvem's eBook for Cloud Operations