rclone and Amazon Cloud Drive

I wanted to take a minute to post about the most useful tool I’ve stumbled across in recent months rclone.

Rclone is a command line tool written in Go designed to synchronise file between local and remote systems. I’ll let you read the homepage to see the complete list of destinations, but the 2 that are instantly useful for me are ‘Amazon Cloud Drive’ (for personal projects) and Amazon S3 (for work). In this mini post I wanted to describe how I’m using Rclone to backup the ~6000 images from my previous article.

At home I run a Ubuntu 16.04 server, with a raid 5 configuration, which contains 12Tb of raw unformatted storage. At last check I had ~500GB of images/videos; a combination of DSLR, personal film and family archives. My trusty MBP only has a 500GB SSD and hence a small portion of these files are stored locally.

Not a sound backup strategy to be found

As an Amazon Prime member, one of the perks is unlimited photo storage on their cloud platform, something I’d been utilising manually, uploading images using the web interface. With the completion of my family archive project I find myself manipulating individual images in random folders, ultimately negating to sync these changes to Amazon Cloud Drive.

In steps Rclone

I’ll not repeat the setup process because the documentation was unusually very good:

Installation, Configuration, Syncing

Once I’d installed and configured access to Rclone, it was just a case of pointing the app at my local folder and my desired remote destination. In my case, this looked like:

rclone sync /local-folder/images/ "Amazon Prime Photos":/remote-images/ --verbose

I ran this initial command as a background process using nohup which is still running (both Cloud Drive and my uploads are quite slow).

Once this has completed. I’ll create a simple cron job that runs the sync task each morning. Something like:

* 1 * * *   rclone sync /local-folder/images/ "Amazon Prime Photos":/remote-images/

Below is some sample Rclone output with the verbose option (–verbose) turned on:

2017/05/01 11:00:38 INFO  : digital/slr/raw/2014/08/31/IMG_8350.dng: Copied (new)
2017/05/01 11:00:48 INFO  : digital/slr/raw/2008/11/17/IMG_2342.dng: Copied (new)
2017/05/01 11:00:54 INFO  : digital/slr/raw/2014/09/02/IMG_8813.dng: Copied (new)
2017/05/01 11:00:57 INFO  : digital/compact/2008/11/sort/img_1134.jpg: Copied (new)
2017/05/01 11:01:00 INFO  : 
Transferred:   135.374 GBytes (1.108 MBytes/s)
Errors:                 4
Checks:              1292
Transferred:        14889
Elapsed time:  34h45m1.1s
Transferring:
 *       digital/slr/raw/2008/12/12/IMG_2387.dng: 12% done, 239.511 kBytes/s, ETA: 43s
 *       digital/slr/raw/2013/10/10/IMG_8206.dng: 97% done, 172.390 kBytes/s, ETA: 1s
 *       digital/slr/raw/2013/10/10/IMG_8207.dng:  5% done, 399.817 kBytes/s, ETA: 27s
 *       digital/slr/raw/2014/09/03/IMG_9066.dng: 43% done, 302.925 kBytes/s, ETA: 24s

2017/05/01 11:01:02 INFO  : digital/slr/raw/2013/10/10/IMG_8206.dng: Copied (new)
2017/05/01 11:01:30 INFO  : digital/slr/raw/2008/12/12/IMG_2387.dng: Copied (new)
2017/05/01 11:01:31 INFO  : digital/slr/raw/2014/09/03/IMG_9066.dng: Copied (new)

That’s it. Free, automatic backups to a destination of my choice. If I ever find myself changing storage provider Rclone has a lot of other possibilities (I’m holding out for a Google Photos integration)…