An guide for point 3,from an linux server administrator,maybe someone finds this usefull,and include all this in an amazon cloud drive app or something to be able to connect amazon cloud drive in nextcloud.
What we want is the ability to mount the CloudDrive in Linux and encrypt everything before uploading.
Let’s get cracking.
Part I: acd_cli
yadayada wrote acd_cli, hosted on github.
This jewel is a python application that connects and mounts(!) the
CloudDrive via fuse to Linux. Enough words, more installing! For CentOS7
it’s:
yum install epel-release
yum install python34-setuptools w3m
easy_install-3.4 pip
pip3 install --upgrade git+https://github.com/bgemmill/acd_cli.git@master
First we install the epel repository that is needed for Python 3.4
and pip (3.4 edition) which we are installing with command 2 and 3 (w3m
is a text based browser). Command 4 installs a fork(!) of acd_cli that
does some work-around needed, but more on that later on.
Try launching the command:
[arrakis ~]# acd_cli
usage: acd_cli [-h] [-v | -d] [-nl] [-c {never,auto,always}]
[-i {full,quick,none}] [-u]
{version,v,sync,s,old-sync,clear-cache,cc,tree,t,children,ls,dir,find,f,find-md5,fh,find-regex,fr,upload,ul,overwrite,ov,stream,st,download,dl,cat,create,c,mkdir,list-trash,lt,trash,rm,restore,re,move,mv,rename,rn,resolve,rs,add-child,ac,remove-child,rc,usage,u,quota,q,metadata,m,mount,umount,delete-everything,init,i,test}
…
acd_cli: error: the following arguments are required: action
Now you need to authorize acd_cli with your amazon account, launch:
[arrakis ~]# acd_cli init
For the one-time authentication a browser (tab) will be opened at https://tensile-runway-92512.appspot.com/.
Please accept the request and save the plain-text response data into a file called “oauth_data” in the directory “/root/.cache/acd_cli”.
Press a key to open a browser.
Hit a key, and (I am assuming you are running a headless server) w3m
will open. Navigate in there, enter password, click OK all the way to
something like this:
{
“access_token”: “4c8lwGmDqF2WZbcYDHI9MZ3040ATLKPU0fQkKTMNYYtx4gqC5bqAlUVvKyZ”,
“exp_time”: 1472910000.2017406,
“expires_in”: 3600,
“refresh_token”: “nDQVHrRvgU1MI1QoLuUAPHtz7CsLp3RIANxyQ2KyMf5EvoxyyEAUeLNarax”,
“token_type”: “bearer”
}
Copy that text or directly save that in a file and place it at
~/.cache/acd_cli/oauth_data, then(!) resume the process. Keep that file
as private as possible, that opens your vault without any further
authorizations. Now try your connection:
$ /bin/acd_cli sync
Getting changes…
Inserting nodes.
If that’s all you’re seeing: acd_cli works and just synced to your account.
Part II: The mounting
At this point we can actually do our first mount, but before let’s agree on some basics:
We will use ‘/var/clouddrive’ as a base for all other stuff.We will store all encryption related files in ‘~/encryption’We will mount the clouddrive on ‘/var/clouddrive/encrypted/’ (encrypted files view)Your current, unencrypted data that you want to encrypt and upload resides in ‘/media’
Create the directories:
base=’/var/clouddrive’
mkdir -p ${base}
mkdir -p ${base}/encrypted
mkdir -p ${base}/decrypted
Let’s mount Amazon CloudDrive right in there for the first time:
acd_cli mount ${base}/encrypted/
The command should exit without any errors, check that it works:
[plex@arrakis ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
[…]
ACDFuse 100T 1G 100T 1% /var/clouddrive/encrypted
[plex@arrakis ~]$ ls -lha /var/clouddrive/encrypted
total 0
drwxrwxrw- 1 acd acd 0 Aug 31 13:14 .
drwxr-xr-x 4 acd acd 38 Aug 31 10:33 …
drwxrwxrw- 1 acd acd 0 Aug 31 13:13 Documents
drwxrwxrw- 1 acd acd 0 Aug 31 13:14 Pictures
drwxrwxrw- 1 acd acd 0 Aug 31 13:14 Videos
The mount is active and you can even list the things that are in the
CloudDrive. Compare that to the web-interface and it should match. If
you don’t care about encryption then your journey can end here. It
should not, however.
We do not need the mount for now, dismount with
umount /var/clouddrive/encrypted
Part III: The disciphers of Data
Disciples, Disciphers… As long as you
keep reading and preach the word of encryption it should matter not.
Less talk, we have data to encrypt. I did create a “Storage” directory
inside the CloudDrive which I’ll use for the encfs. You can encrypt all
of it, but you might want to have, at some point, an unencrypted area
for sharing or whatnot. Better keep your options open.
install encfs:
yum install encfs
Here is the thing: You do have a lot of
data right now that you want to upload and fast. rsync comes to mind;
but rsync and acd_cli do not play nice at this time (extremely high cpu
load, extremely low throughput), that’s why we clone the fork of
acd_cli– still, uploading is a mess. This is why we will not use rsync
for the initial upload but some other file. More on that later; lets set
up the encryption. For that we will utilize encfs in its reverse mode,
that is, it will mount a virtual encrypted view of the data that it will encrypt on the fly. Normally encfs works the other way around: It mounts encrypted data that that it decrypts on the fly.
Normal Mode: On Harddisk: Encrypted files; encfs-mount shows decrypted files.Reverse Mode: On Harddisk: Decrypted files; encfs-mount shows encrypted files.
But since your data is decrypted on your, say, NAS, we will upload the encrypted view of that. Setup time.
Before you start the commands below,
create a password. I suggest using ‘pwgen’ with ‘pwgen -s 128 1’, this
will create a single, secure 128 chars long password.
base=’/var/clouddrive’
mkdir -p ${base}/reverse
encfs --reverse /media /var/clouddrive/reverse
Creating new encrypted volume.
Please choose from one of the following options:
enter “x” for expert configuration mode,
enter “p” for pre-configured paranoia mode,
anything else, or an empty line will select standard mode.
?> x
Manual configuration mode selected.
The following cypher algorithms are available:
- AES : 16 byte block cipher
– Supports key lengths of 128 to 256 bits
– Supports block sizes of 64 to 4096 bytes
- Blowfish : 8 byte block cipher
– Supports key lengths of 128 to 256 bits
– Supports block sizes of 64 to 4096 bytes
Enter the number corresponding to your choice: 1
Selected algorithm “AES”
Please select a key size in bits. The cypher you have chosen
supports sizes from 128 to 256 bits in increments of 64 bits.
For example:
128, 192, 256
Selected key size: 256
Using key size of 256 bits
Select a block size in bytes. The cypher you have chosen
supports sizes from 64 to 4096 bytes in increments of 16.
Alternatively, just press enter for the default (1024 bytes)
filesystem block size: 4096
Using filesystem block size of 4096 bytes
The following filename encoding algorithms are available:
- Block : Block encoding, hides file name size somewhat
- Block32 : Block encoding with base32 output for case-sensitive systems
- Null : No encryption of filenames
- Stream : Stream encoding, keeps filenames as short as possible
Enter the number corresponding to your choice: 1
Selected algorithm “Block”"
reverse encryption - chained IV and MAC disabled
Enable per-file initialization vectors?
This adds about 8 bytes per file to the storage requirements.
It should not affect performance except possibly with applications
which rely on block-aligned file io for performance.
y/[n]:
Configuration finished. The filesystem to be created has
the following properties:
Filesystem cypher: “ssl/aes”, version 3:0:2
Filename encoding: “nameio/block”, version 4:0:2
Key Size: 256 bits
Block Size: 4096 bytes
File holes passed through to ciphertext.
Now you will need to enter a password for your filesystem.
You will need to remember this password, as there is absolutely
no recovery mechanism. However, the password can be changed
later using encfsctl.
New Encfs Password:
Verify Encfs Password:
That should have created your initial
reverse view, try ‘ls -lha /var/clouddrive/reverse/’. If you see a bunch
of garbage: It worked. You also have a new file in /media/, named
.encfs6.xml. This is your supplemental encryption file that, combined
with your password will decrypt the data. Keep both at a super-safe
place. I you lose either you will no longer be able to access your data.
Move your file to our per-determined location:
mkdir ~/encryption
mv /media/.encfs6.xml ~/encryption/reverse.xml
We will store all encryption related things in ~/encryption. Also
create a file ‘~/encryption/reverse.pass’ and on a single line paste
your password there. This is optional and will help with auto-mounting.
Part IV: The Syncing
At this point you should be able to
mount amazon CloudDrive (which is not mounted currently) and having a
reverse-encfs mount up and running, let’s sync. Like I said, syncing
with rsync is currently not possible. So we will employ rclone,
a command line tool that can handle most Cloud-Provides, including
Amazon CloudDrive. Depending on your server, download the appropriate
binary from http://rclone.org/downloads/.
I am using a raspberry Pi3 for the upload, which maxes my 4Mb/s nicely
with plenty of resources to space. The RP3 utilizes the ARM binary for
Linux. Download, Unpack and place rclone in ‘/usr/local/bin/rclone’. For
further installation help consult their installation docs.
You’ll then need to do a first-time setup that you only need to do once, they also have a very detailed howto for that: http://rclone.org/amazonclouddrive/
Done? Now simply sync the encrypted files to your CloudDrive:
rclone sync /var/clouddrive/reverse remote:Storage/
Replace ‘remote’ with whatever you named
your remote in rclone during the setup. If you took my advice and
created a ‘Storage’ directory then that command will upload all the
encrypted files into your CloudDrive. I really, really recommend
launching that sync command from within a screen or tmux shell. If all worked well you’ll see (once a minute) a status output like this:
2016/09/02 10:19:26
Transferred: 928.978 GBytes (3.890 MBytes/s)
Errors: 0
Checks: 2
Transferred: 899
Elapsed time: 67h56m1.2s
Transferring:
- …1JrzoMcLjfGxQTbL1zsG3hCu23FI05r19H6Ek4l148: 94% done. avg: 1020.0, cur: 834.5 kByte/s. ETA: 4m1s
- …6jftoXHq-0n-AzZxg/JLV-hmzY,QGSqaLmmLKSPvcI: 3% done. avg: 739.5, cur: 775.2 kByte/s. ETA: 19m59s
- …ClFXawwAnT,8aoErCjOH-aZP2HyHDGrmdQ3XNHhp97: 66% done. avg: 892.0, cur: 717.1 kByte/s. ETA: 11m52s
- …VMBMmHAa23,LVkjy2zyNHHUvv4kOWI3xXBmRZg0Jk5: 31% done. avg: 905.0, cur: 734.7 kByte/s. ETA: 53m26s
Once some files have uploaded, check your CloudDrive web-interface, you should only see encrypted files inside ‘/Storage’:
So let it run and finish uploading. If
the rclone command aborts/interrupts at some point it’s safe to restart
it. It will ignore already-uploaded files so resume is a bliss. rclone
survived several DSL disconnections so far without dropping out at all. I
am still on my first-run on that command.
Part V: Accessing your Data
At some point you might want to access your data. Let’s also assume
you want to access the data on another server. Repeat the acd_cli and
encfs installation as we did earlier. You can skip reverse mounting and
rclone completly. Create directories:
base=’/var/clouddrive’
mkdir -p ${base}
mkdir -p ${base}/encrypted
mkdir -p ${base}/decrypted
mkdir -p ~/encryption
Copy the ‘~/.cache/acd_cli/oauth_data’
file from your rclone-upload server to this server so you can skip the
authorization altogether. Also copy the ‘~/encryption/reverse.*’ files
likewise. Next, create a ‘~/encryption/amazon-mount.sh’ file, paste this
inside:
#!/bin/bash
base="/var/clouddrive"
encfs_pass="cat ~/encryption/reverse.pass"
if [ ! -e ${base} ] ; then
mkdir -p ${base}
mkdir -p ${base}/encrypted
mkdir -p ${base}/decrypted
fi
if [ “$1” == “mount” ] ; then
acd_cli sync
acd_cli mount ${base}/encrypted/ || exit 1
ENCFS6_CONFIG=’~/encryption/reverse.xml’ encfs --extpass="/bin/echo ${encfs_pass}" ${base}/encrypted/Storage ${base}/decrypted
exit 0
fi
if [ “$1” == “umount” ] ; then
fusermount -u ${base}/decrypted
fusermount -u ${base}/encrypted
exit 0
fi
echo "Please supply mount or umount."
and make it executable:
chmod 0700 ~/encryption/amazon-mount.sh
This script will mount amazon CloudDrive and encfs als dismount is after. Like so:
$ ~/encryption/amazon-mount.sh mount
Getting changes…
Inserting nodes.
$
This will result in two mounted directories:
$ df -h | grep cloud
ACDFuse 100T 912G 100T 1% /var/clouddrive/encrypted
encfs 100T 912G 100T 1% /var/clouddrive/decrypted
You can now use the decrypted directory to your hearts desire, minus rsync for now.