Automatic backups to Dropbox / cloud storage

newstaircase

Hello everyone, hope you're ok,

I understand that Opalstack keeps daily, weekly and monthly backups which is amazing, but I also want to take my own backups and store them elsewhere. I currently use Dropbox (at the moment just for SQL dumps, but I would want to backup source code and media files in this way too), but I run an scp command (using Lingon to automate it) to copy from the Opalstack servers to the Dropbox folders on my local computer. This then syncs automatically, but obviously all this only works if my computer is running. I am looking for something that will be sure to run and sync.

I understand that you cannot do ‘scp’ into Dropbox directly, however, I have found that there is a program called ‘rclone’ which I could install and configure to achieve a similar result.

I just wondered if anyone had used rclone successfully, or anything similar? I am not precious about using Dropbox either, but I think it would probably be a good fit.

I understand that rclone would be running on Opalstack so would have memory and bandwidth implications on my account. I don't know what these would be yet, but I'm hoping not too much.

Any advice would be greatly appreciated 😀 I feel embarrassed asking this, but I am stuck so thought I would reach out.

sean

rclone is fairly easy to install and use on an Opalstack shell user account:

SSH to your shell user account.

Download rclone and install into your ~/bin directory:

   mkdir ~/bin
   wget https://downloads.rclone.org/v1.58.0/rclone-v1.58.0-linux-amd64.zip
   unzip rclone-v1.58.0-linux-amd64.zip
   mv rclone-v1.58.0-linux-amd64/rclone ~/bin/

Run rclone config to set up your Dropbox connection:
- Choose n for New remote
- Enter a name for the new remote. You'll use this name when copying files to the remote, so keep it short and simple. I used dbx.
- Enter dropbox as the remote storage type.
- Leave client_id and client_secret blank.
- Enter n to skip advanced config
- Enter Y to do auto config
At this point you'll be prompted to visit a URL like http://127.0.0.1:53682/auth?state=SoMeRaNdOmTeXtHeRe. Before you can visit that URL in your desktop web browser you'll need to open a SSH tunnel to the port mentioned in the URL (53682 in this case). To open the SSH tunnel, open a second terminal window on your system and run the following command:
```
   ssh shelluser@opalN.opalstack.com -L 53682:127.0.0.1:53682 -N
```
After you run the above command your terminal will connect to the server and wait. There will be no shell prompt.
Visit the URL from step 4 in your desktop web browser to authorize rclone to connect to your Dropbox.
Go back to the terminal window running your SSH tunnel and press CTRL-C to terminate it.
Go back to the terminal window running your original SSH session from step 1:
- Enter y to accept the new config.
- Enter q to quit the config tool.

At this point you're ready to use rclone, for example to sync ~/backups to a Dropbox folder named backups:

~/bin/rclone sync ~/backups dbx:backups/

Hope this helps!

newstaircase

sean You are a star! I am so grateful for all your timely and detailed responses to my many questions. I will look forwarding to trying to get this set up and tested tomorrow 😃

sean

You're very welcome!

Mike

I just tried this and when I get to step 4 and open a second shell and enter in ssh myname@opal5.opalstack.com -L 53682:127.0.0.1:53682 -N, I'm given a password prompt but then I get the message "bind: Address already in use" Have you seen that problem before?

klynton

Hey Mike,

Yeah that error means that someone else is using that port number. You can get your own port number to use by creating a Proxy Port application and using the port it assigns you to do step 4.

elena

Should it also work for One Drive Business https://rclone.org/onedrive/ ?

And what would be the best way of backing up the files of some 80 apps, jailed in different home ssh user directories, while using minimum disk space on Opalstack? Would I have to install rclone for each user? Or could I have one single ssh user running the rclone task for all the 80 apps?

I am also considering using https://updraftplus.com/ as most of my websites are basic Wordpress apps.

Any suggestions?

Thank you very much!

sean

elena sorry I missed your question!

I do not use OneDrive so I'm not certain, but I think it should work with OneDrive Business. You might be able to get a definitive answer from the rclone author.

If you're primarily concerned with backing up WP then a plugin like Updraft is a good choice for that.

Mike

Klynton - this is followup to your suggestion to create a proxy port application..

I have created a port - 22825. Would I then in step 4 use this command - ssh myname@opal5.opalstack.com -L 22825:127.0.0.1:53682 -N ? Or should I used the 22825 port in both parts of the syntax?

I have tried this using the command I list above and it asks for my password but once I put the password in, nothing else happens on the screen and in the browser window I still get a site refused to connect error when trying to go to http://127.0.0.1:53682/auth?state=hfwWsjvD5wqLhBSnxx4nSg

Thank you for your assistance.

sean

Mike sorry for the confusion here. If you're seeing "bind: Address already in use" then it means someone on the server is using port 53682, which is the port rclone wants to use. Changing the port on your tunnel won't fix that.

My first suggestion is wait a few minutes and try running the rclone config again. I just went the through the steps and got "address already in use" the first time, but it worked correctly 20 minutes later.

If you can't get that to work, then please do the following:

Install rclone on your own computer.
Repeat the config steps 1-3 we've provided, except for the last step. This time, enter "n" when prompted to do auto-config.
In a separate terminal run rclone authorize "dropbox" on your own computer and follow the onscreen prompts. You'll be issued an authorization token.
Back in your Opalstack terminal, paste the token at the "config_token" prompt and press Enter to complete the setup.

Mike

sean Thanks Sean, I feel like I'm really close! I get the access token from the terminal on my computer and it looks something like this:

Paste the following into your remote machine --->
{"access_token":"xxxxxx","token_type":"bearer","refresh_token":"xxxxxx","expiry":"2022-03-24T14:34:41.8815391-04:00"}
<---End paste

Back in the OpalStack terminal - I get to the Waiting for code... and paste in the following:

{"access_token":"xxxxxx","token_type":"bearer","refresh_token":"xxxxxx","expiry":"2022-03-24T14:34:41.8815391-04:00"}

but when I press Enter, nothing seems to happen. Am I not pasting the right information? I tried both with and without the curly brackets but got the same results.

I appreciate your help!

Mike

sean

Mike from your description it seems like you've pasted the correct info (ie including the opening/closing brackets).

When you get to the point where nothing seems to happen, is your terminal session still working? For example, if you hit CTRL-C does it interrupt the process and drop you back to the prompt?

Mike

sean Hi Sean, yes I can interrupt the process with Ctrl- C. Is the expected behavior to receive a certain message and go back to the command prompt?

sean

Mike yes, the expected behavior is that you'll get a prompt to accept the final config like so:

Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine

y) Yes (default)
n) No
y/n> n
Option config_token.
For this to work, you will need rclone available on a machine that has
a web browser available.
For more help and alternate methods see: https://rclone.org/remote_setup/
Execute the following on the machine with the web browser (same rclone
version recommended):
	rclone authorize "dropbox"
Then paste the result.
Enter a value.
config_token> {"access_token":"xxxxx","token_type":"bearer","refresh_token":"xxxxx",
"expiry":"2022-03-24T16:18:48.275952-06:00"}
--------------------
[dbx]
type = dropbox
token = {"access_token":"xxxxx","token_type":"bearer","refresh_token":"xxxxx",
"expiry":"2022-03-24T16:18:48.275952-06:00"}
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote

I just tried this on your server and had no trouble so I'm not sure why it wouldn't work for you.

Please try it once more and if you run into the same issue, then please email Opalstack support and provide the value you're pasting for config_token - we'll then log in as your Opalstack shell user and try to complete the config ourselves.

bici

I just installed this via the command wget but nothing happens after the download and the move to ~/bin
-bash-4.2$ rclone config
-bash: rclone: command not found

even though clone is on the directory.

is the version correct?

nick

bici As per the original instructions you will have to reference the path to the binary command to use it, eg. ~/bin/rclone

Otherwise, edit the ~/.bash_profile file in your home dir, and update the PATH to include the path to ~/bin, eg: PATH=$HOME/bin:$PATH:$HOME/.local/bin

bici

sorry to be so dense, but this command?? ssh shelluser@opalN.opalstack.com -L 53662:127.0.0.1:53452 -N

do I use my shell user name bici or actually shelluser?

neither seemed to work. although the first time I was asked for a password? password to my ssh account I assume

nick

bici You will use the shell user name that you created in the dashboard, then authenticate with its password. If needed, verify the shell user name or change it's password here on the apps page: https://my.opalstack.com/apps/

Also, be sure to specify the server number. So let us suppose your server is opal4 and your shell user is bici, your command would look like this:

ssh bici@opal4.opalstack.com -L 53682:127.0.0.1:53682 -N

Ref:
https://docs.opalstack.com/user-guide/shell-users/#changing-a-shell-users-password

bici

nick Thank-you thank-you ! It Worked!

so in the good old days on webfaction I was able to backup apps and DBs to my strongspace via rsync.

So I am nowable to backup the Dbs to my dropbox /backups dir.
If I wanted to clone my apps dirs to dropbox can i simply use :
~/bin/rclone sync ~/apps dbx:apps/ after having created the apps dir on dropbox?

cheers!

nick

bici That sounds about right. That command should copy data from local apps dir to the remote one you made on dropbox.

bici

nick great.
another question.
does running rclone overwrite existing files on the remote dropbox directory or add changed files?
In the case where I backup the apps directories I would hope the files on dropbox are overwritten with the new files.