So recently I got subscription to OneDrive in an effort to keep my files on cloud to have some peace of mind. Although needless to say, I don't want to trust any cloud storage. So what's the solution? Well, encryption! By encrypting our files we can have best of both worlds: Have a cloud storage and preventing corporates to snoop on them too.
Yes, encryption is not child's play, especially if you want to use open source tools and it comes with responsibility understanding the intricacies behind it. I wanted a solution where I could easily upload files to OneDrive from my Mint PC. And I think I have got this great solution!
Hence I have created this thread to document the procedure and also to keep track of this adventure I have embarked upon.
So, I got to view this video at a time, when I needed it the most:
.
The video talks about how we can use rclone to upload stuff to cloud after encryption and as a seamless process. Here's the high level steps:
1. Install rclone and rsync
2. Setup a config file for rclone (where you provide it access to your cloud storage
3. Setup encryption on that cloud storage you just setup in previous step
4. That's it. Now you can use simple commands to upload stuff and download it from cloud too.
And although the video makes it very clear, I am going to document the steps with a later edit of the second post of this thread. Meanwhile I just wanted to quickly get this post out there, since I got very excited by successfully trying it. The rclone utility also comes with a local web based GUI where you can browse your encrypted files and download them too! It's all very exciting for me at the moment.
So stay tuned for the updates and share if you already use similar solution!
Steps to install and use rclone to upload and download files from OneDrive on Linux Mint (should work with any Ubuntu based distro):
1. Installing rclone
Installing rclone from the following command is preferred, since when I tried it from the official repository I got an old version of rclone with missing features.
Code:
curl *rclone.org/install.sh | sudo bash
2. Type following command to create a config file for rclone:
Code:
rclone config
3. Now enter n to create a new remote and enter a name for the new remote, eg:
Code:
vyom@vyom-mint-main:~$ rclone config
2019/10/21 10:06:09 NOTICE: Config file "/home/vyom/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> VyomCloud
In the above example, I choose VyomCloud as the name for my cloud storage option.
In the next step you need to choose the provider of cloud storage which you want to use, I choose 20, which is for Onedrive:
Code:
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / 1Fichier
\ "fichier"
2 / Alias for an existing remote
\ "alias"
3 / Amazon Drive
\ "amazon cloud drive"
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)
\ "s3"
5 / Backblaze B2
\ "b2"
6 / Box
\ "box"
7 / Cache a remote
\ "cache"
8 / Dropbox
\ "dropbox"
9 / Encrypt/Decrypt a remote
\ "crypt"
10 / FTP Connection
\ "ftp"
11 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
12 / Google Drive
\ "drive"
13 / Google Photos
\ "google photos"
14 / Hubic
\ "hubic"
15 / JottaCloud
\ "jottacloud"
16 / Koofr
\ "koofr"
17 / Local Disk
\ "local"
18 / Mega
\ "mega"
19 / Microsoft Azure Blob Storage
\ "azureblob"
20 / Microsoft OneDrive
\ "onedrive"
21 / OpenDrive
\ "opendrive"
22 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
23 / Pcloud
\ "pcloud"
24 / Put.io
\ "putio"
25 / QingCloud Object Storage
\ "qingstor"
26 / SSH/SFTP Connection
\ "sftp"
27 / Union merges the contents of several remotes
\ "union"
28 / Webdav
\ "webdav"
29 / Yandex Disk
\ "yandex"
30 / http Connection
\ "http"
31 / premiumize.me
\ "premiumizeme"
4. Entered 20 since it's for Onedrive:
Code:
Storage> 20
** See help for onedrive backend at: *rclone.org/onedrive/ **
5. Next it will ask you to enter App client ID and App client secret. Just press enter to leave it blank.
6. Next it will ask to Edit advanced options. Just press n to leave it.
7. Now it will ask to use auto config. Enter Y since you may want to use that:
Code:
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> Y
The above step will open browser where you need to sign in with the cloud credentials and give rclone the required permissions. After which rclone with bring back the token ID automatically.
8. In the next option, it will ask you the type of cloud plan you have. I entered 1 since mine was Onedrive personal plan.
Code:
Log in and authorize rclone for access
Waiting for code...
Got code
Choose a number from below, or type in an existing value
1 / OneDrive Personal or Business
\ "onedrive"
2 / Root Sharepoint site
\ "sharepoint"
3 / Type in driveID
\ "driveid"
4 / Type in SiteID
\ "siteid"
5 / Search a Sharepoint site
\ "search"
Your choice> 1
Found 1 drives, please select the one you want to use:
0: (personal) id=xxxxxxxxxxxxxxxxx
Chose drive to use:> 0
Found drive 'root' of type 'personal', URL: *onedrive.live.com/?cid=xxxxxxxxxxxxxxxxx
Is that okay?
y) Yes
n) No
y/n> y
As can be seen you will be shown the cloud URL which rclone fetched and will ask to confirm if it's right. You can verify it by opening the URL and confirming if it's the right cloud storage.
Just confirm by pressing Y.
9. Next it will show you a very long string which is the token that it fetched. Just confirm by pressing y.
Once you do that, you will be shown a list of remote you just created:
Code:
Current remotes:
Name Type
==== ====
VyomCloud onedrive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q>
At this point you can use rclone to sync files between local and cloud storage. But without encryption. In next steps we will create encrypted channel based on the remote we just created:
10. Enter n to create a new remote and give it a name similar to following (so that you can easily identify):
Code:
name> VyomCloudEncrypt
11. Now enter the option 9 since it will allow to use encryption/decryption for that remote we just created:
Code:
Storage> 9
12. In the next step you need to enter the name of the remote you just created, a colon and then the name of the folder which you map against this cloud storage. In my example since I wanted to store all files which I upload in a single directory called, "Encrypted" I choose the name: VyomCloud:Encrypted
Code:
Remote to encrypt/decrypt.
Normally should contain a ':' and a path, eg "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
Enter a string value. Press Enter for the default ("").
remote> VyomCloud:Encrypted
13. In the next two steps, you can choose if you want to encrypt file names and directory names. I choose not to encrypt directory names but do encrypt file name.
Code:
How to encrypt the filenames.
Enter a string value. Press Enter for the default ("standard").
Choose a number from below, or type in your own value
1 / Don't encrypt the file names. Adds a ".bin" extension only.
\ "off"
2 / Encrypt the filenames see the docs for the details.
\ "standard"
3 / Very simple filename obfuscation.
\ "obfuscate"
filename_encryption> 2
Option to either encrypt directory names or leave them intact.
Enter a boolean value (true or false). Press Enter for the default ("true").
Choose a number from below, or type in your own value
1 / Encrypt directory names.
\ "true"
2 / Don't encrypt directory names, leave them intact.
\ "false"
directory_name_encryption> 2
14. In the next options rclone will let you create two passwords which will be used for encryption and decryption and will also let you generate passwords based on bit strength:
Code:
Password or pass phrase for encryption.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank
y/g/n> g
Password strength in bits.
64 is just about memorable
128 is secure
1024 is the maximum
Bits> 128
Your password is: a58JeIjbHWWeZUw5LNJfHA
Use this password? Please note that an obscured version of this
password (and not the password itself) will be stored under your
configuration file, so keep this generated password in a safe place.
y) Yes
n) No
y/n> y
Password or pass phrase for salt. Optional but recommended.
Should be different to the previous password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank
y/g/n> g
Password strength in bits.
64 is just about memorable
128 is secure
1024 is the maximum
Bits> 128
Your password is: aMfq-o39icmNQFJjAScdLw
Use this password? Please note that an obscured version of this
password (and not the password itself) will be stored under your
configuration file, so keep this generated password in a safe place.
y) Yes
n) No
y/n> y
Remote config
--------------------
[VyomCloudEncrypt]
type = crypt
remote = VyomCloudEncrypt:Encrypted
filename_encryption = standard
directory_name_encryption = false
password = *** ENCRYPTED ***
password2 = *** ENCRYPTED ***
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
VyomCloud onedrive
VyomCloudEncrypt crypt
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q>
Finally you have now created two remotes. One associated with OneDrive and another an encrypted layer to that cloud. If you use the second remote, the files will be uploaded after encryption. So now let's upload a directory of files to cloud and see if it works:
15. Copying files to cloud after encryption:
Code:
vyom@vyom-mint-main:~$ rclone -v copy ~/Downloads/test VyomCloudEncrypt:test
2019/10/21 10:47:15 INFO : Encrypted drive 'VyomCloudEncrypt:test': Waiting for checks to finish
2019/10/21 10:47:15 INFO : Encrypted drive 'VyomCloudEncrypt:test': Waiting for transfers to finish
2019/10/21 10:47:17 INFO : TestTextfile.txt: Copied (new)
2019/10/21 10:47:17 INFO : Test PDF File.pdf: Copied (new)
2019/10/21 10:47:17 INFO : naturepic.jpg: Copied (new)
2019/10/21 10:47:17 INFO : Landscape-Photography-steps.jpg: Copied (new)
2019/10/21 10:47:19 INFO : pexels-photo-247478.jpeg: Copied (new)
2019/10/21 10:47:19 INFO :
Transferred: 1.763M / 1.763 MBytes, 100%, 373.090 kBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 5 / 5, 100%
Elapsed time: 4.8s
Here's the file on cloud:
16. To download the file after decryption you can use following command:
Code:
rclone -v copy VyomCloudEncrypt:test ~/test
Above command will download and decrypt files and save it in test folder of your home directory.
17. Using web interface to rcopy:
Code:
rclone rcd --rc-web-gui
Using above command will open a web UI in your browser, and you can view and download/upload files from Encrypted folder using it:
Disclaimer: Use steps in this guide at your own risk. Always test encryption/decryption before uploading important things. Keep the configuration file and passwords backed up, since without it, you may loose access to your cloud data. You can retrieve the files, but you may not be able to decrypt it.
Just an info,rclone uses api available for public use by cloud storage provider for its functioning & is bound by the limits of that api.e.g.earlier google api was "misused" by people uploading 10/100s of TB of encrypted data to their unlimited google drive account(another misuse,google drive business account comes with unlimited storage provided minimum number of users under that business account are 5(aka $50 at $10 per user per month) but google usually don't enforce this even for a single user business drive account(aka $10 per month). Because of this google revoked its earlier version api & the new api comes with restrictions like xyz no. of requests can be made within 24 hours which limits the amount of data that can be uploaded/downloaded using rclone(I think the limit is ~750gb in 24 hours for google drive now).
Google Drive API Ban, what is it?
API is really what most “cloud” based applications use for communication between the client and the cloud, think of it as browsing the web every time you hit google.com that would be considered 1 API hit.
So when you upload (PUT) or download (GET) a file you make an API POST or API GET request and providers like amazon/google/Microsoft/Dropbox use rate limiting by API quota access so each file is not 1 API request as they break each file down into possibly hundreds or thousands of requests so depending on file size and other factors 1 file can be 1000 API requests to upload or download.
^^ Yes, I heard about that.
But even with any restrictions on API, I have no other choice. M$ don't provide a client for Linux anyway, so using rclone is the only best option.
It's actually no big deal,I just added the info for future reference.Only someone uploading/downloading TB of data in 24 hours will likely get hit with MS api restrictions.
Hi Guest we just wanted to alert you to a major change in the forum. We will no longer be allowing the posting of outgoing links. Please use the attachment feature to attach media to your posts.
