HOWTO migrate stuff automatically from WebFaction!

sean

End-of-life for Opalstack API v0 and the WebFaction migration tool

‼️ If your WebFaction account is now closed then the automatic migration procedures below are not going to work for you. You'll need to create your apps, databases, domains etc manually via your Opalstack dashboard and set up your sites manually.

As we've mentioned on @opalstack and elsewhere in the forum here we've been building a suite of migration tools to help incoming WF customers move their stuff over to our service.

I'm happy to say that we've deployed the migration stuff across the fleet and the following migrations can be performed automatically!

Wordpress sites (including the app, DB, DB user, and domains)
Generic PHP apps
Generic port listeners (basically anything that listens on a port at WF: Django, Rails, "custom app listening on port", websockets etc)
MySQL databases and users (MariaDB on our side)
PostgreSQL databases and users
Email addresses and mailboxes (more info)

The migration tool is wrapped up in a command called wfmigrate.

Example: Migrating a Wordpress site

Here's a quick example of how to use it to migrate a WP site:

Create a shell user via your Opalstack dashboard if you don't have one already: https://docs.opalstack.com/user-guide/shell-users/
Go to https://my.opalstack.com/tokens/ and create a new API token if you don't have one already. Make a note of the token key (not its name) as you will need that in your migration config later. The key is the long sequence of characters in the second column on the token page in the dashboard.
SSH to your Opalstack server as your shell user: https://docs.opalstack.com/user-guide/server-access/#ssh-access
Make a copy of our boilerplate config file for the WP migrator (Do this every time to ensure that you're using the latest configuration required by the migrator):
```
cp /opt/app_migrators/wf_wordpress/config-template.ini ./config.ini
```
The boilerplate config files are located at:
- /opt/app_migrators/wf_generic_dns/config-template.ini
- /opt/app_migrators/wf_generic_domains/config-template.ini
- /opt/app_migrators/wf_generic_email/config-template.ini
- /opt/app_migrators/wf_generic_maildir/config-template.ini
- /opt/app_migrators/wf_generic_maria/config-template.ini
- /opt/app_migrators/wf_generic_php/config-template.ini
- /opt/app_migrators/wf_generic_php_symlink/config-template.ini
- /opt/app_migrators/wf_generic_port/config-template.ini
- /opt/app_migrators/wf_generic_psql/config-template.ini
- /opt/app_migrators/wf_generic_staticonly/config-template.ini
- /opt/app_migrators/wf_generic_staticonly_symlink/config-template.ini
- /opt/app_migrators/wf_wordpress/config-template.ini
Edit the config file config.ini that you just created to plug in your Opalstack info and WF info. Note the following:
- The value for apitoken is the key (not the name) of the token you created in in step 2.
- The server for your WebFaction info is the short server hostname, like webNNN or wf-NN-NN-NN-NN. Do not include ".webfaction.com" in the server name.

Run a simulation to do a pre-migration check. This will ensure that you don't have any conflicting items already created on Opalstack:

wfmigrate wordpress ./config.ini --mode simulate

You'll see a lot of output go by as wfmigrate inventories all of your items and simulates the migration. There are couple of things to look out for:
- WARNING messages will appear if the simulation encounters an item on your Opalstack server that has the same name as the item on your WebFaction server. Such items will be deleted and replaced when you run the live migration in the next step, so pay attention here! If there's something you don't want to delete, back it up before you run the live migration.
- CRITICAL messages will appear if the simulation runs into a problem that prevents it from performing the migration, like a bad password, a SSH key that can't be set, etc. The simulation will terminate at that point.

Once you've run a successful simulation you can perform the live migration with the do mode option:

wfmigrate wordpress ./config.ini --mode do

wfmigrate will then run to create all of the necessary items in your Opalstack account and move everything from WF over to our side. All of the above info about WARNING and CRITICAL messages applies to this step as well.

That's pretty much it!

Tip: if you notice a database connection error after migrating WP, then check your DB_PASS in wp-config.php. If the migrated password has multiple sets of quotation marks around it then correct that, eg change `""password123"" to "password123". If you think the password might be incorrect, then reset it and put the correct password in your wp-config.php.

The other items work similarly, and you can run wfmigrate --help to see all available options:

$ wfmigrate --help
usage: wfmigrate [-h]
                 {account,domains,dns,maria,psql,email,email-data,php,php-symlink,port,staticonly,staticonly-symlink,wordpress}
                 ...

WF -> Opalstack Migrator

positional arguments:
  {account,domains,dns,maria,psql,email,email-data,php,php-symlink,port,staticonly,staticonly-symlink,wordpress}
                        Migration Component
    account             Account (config builder)
    domains             Domains
    dns                 DNS Overrides
    maria               MariaDB (for MySQL databases)
    psql                PostgreSQL (for PostgreSQL databases)
    email               Email
    email-data          Email Data (Import & Deduplication)
    php                 Generic PHP
    php-symlink         Generic Symlink to PHP
    port                Generic Proxy-Port (custom app listening on port)
    staticonly          Generic Static-Only (no .htaccess)
    staticonly-symlink  Generic Symlink to Static-Only (no .htaccess)
    wordpress           Wordpress

optional arguments:
  -h, --help            show this help message and exit

You can also get help on a specific migrator, eg:

$ wfmigrate wordpress --help 
usage: wfmigrate wordpress [-h] --mode {simulate,do}
                           [--loglevel {debug,info,warning,error,critical}]
                           config_path

positional arguments:
  config_path           Path to config.ini

optional arguments:
  -h, --help            show this help message and exit
  --mode {simulate,do}  Mode of operation
  --loglevel {debug,info,warning,error,critical}
                        Logging verbosity level

Caveats

⚠️ Copy the configuration template every time. We're always updating wfmigrate so you need to have the correct configuration every time you run it.

⚠️ When you migrate a MySQL db from WebFaction, the password for the MariaDB user created on our end will not be the same. You'll need to either change the DB password to match your old one from WF, or update your application settings to use the new DB password. You can find the new DB password in your Opalstack account notifications.

⚠️ The maria and psql migration options do not migrate databases from WebFaction private database instances. It can only migrate MySQL databases hosted on WebFaction's shared MySQL created via WebFaction's dashboard.

⚠️ When you migrate a port listener the migrator will create a matching app on Opalstack and copy the app files over from WebFaction, but the app will almost certainly not work right away. Once the app has been copied you'll need to adjust whatever configuration it uses, for example filesystem paths are different, server IPs, maybe your shell user name, etc. Binaries like Apache and Nginx that are copied over might need to be completely rebuilt from source.

⚠️ READ THE WARNINGS when you run the simulation step. If you see something like "htaccess file contains incompatible directive: SetHandler" or similar then you will need to edit your .htaccess after the migration those directives. If you don't do this then your site visitors will see your site's PHP code instead of the rendered HTML.

⚠️ At this time PostGIS databases cannot be migrated automatically. Instead you'll need to do the following:
1. Export the database at WF and copy the exports to Opalstack.
2. Create new PostgreSQL databases on Opalstack.
3. Contact support to have PostGIS enabled.
4. Import your data.

⚠️ We only support PHP versions 5.6, 7.3, 7.4, and 8.0. If you're migrating a WP or PHP app then set the the version to the closest match.

Migrating multiple apps, sites, and databases

The migrator has an account option that you can use to generate migration templates and commands to migrate all of your apps or databases in a single run.

To migrate multiple apps:

Copy the account migration template:

mkdir -p ~/migrations/apps
cd ~/migrations/apps
cp /opt/app_migrators/wf_account/config-template.ini account.ini

Edit account.ini to provide your Opalstack and WF account info.

Run the migrator to generate your app migration config:

wfmigrate account ./account.ini --buildconfigs=apps

The migrator will run and generate the following items:
- A ini file for each app to be migrated
- A file migration_commands.txt containing the commands to run for each app migration. These are initially set to run as simulations.
Edit migration_commands.txt to remove any apps that you don't want to migrate.

Run the simulations:

bash migration_commands.txt

After the simulation is done, correct any problems that were noted (especially incompatible PHP versions as noted above and htaccess directives) and then edit migration_commands.txt to change the simulate option to do.

Run the live migration:

bash migration_commands.txt

The migrator will then run to migrate each application.

**The multiple app migration does not migrate your sites, only your apps. Proceed to the next step to migrate your sites.

To migrate multiple sites:

Copy the account migration template:

mkdir -p ~/migrations/sites
cd ~/migrations/sites
cp /opt/app_migrators/wf_account/config-template.ini account.ini

Edit account.ini to provide your Opalstack and WF account info.

Run the migrator to generate your app migration config:

wfmigrate account ./account.ini --buildconfigs=sites

The migrator will run and generate the following items:
- A ini file for each site to be migrated
- A file migration_commands.txt containing the commands to run for each site migration. These are initially set to run as simulations.
Edit migration_commands.txt to remove any sites that you don't want to migrate. If you have HTTP>HTTPS redirect sites at WebFaction then leave those out of the migration. You can enable HTTPS redirection in your Opalstack dashboard after the sites are migrated.

Run the simulations:

bash migration_commands.txt

After the simulation is done, correct any problems that were noted and then edit migration_commands.txt to change the simulate option to do.

Run the live migration:

bash migration_commands.txt

The migrator will then run to migrate each site.

To migrate multiple databases:

Copy the account migration template:

mkdir -p ~/migrations/dbs
cd ~/migrations/dbs
cp /opt/app_migrators/wf_account/config-template.ini account.ini

Edit account.ini to provide your Opalstack and WF account info.

Run the migrator to generate your app migration config:

wfmigrate account ./account.ini --buildconfigs=dbs

The migrator will run and generate the following items:
- A ini file for each app to be migrated
- A file migration_commands.txt containing the commands to run for each app migration. These are initially set to run as simulations.
Edit migration_commands.txt to remove any databases that you don't want to migrate. ‼️If you've already migrated your Wordpress apps then remove your WP databases from the file.
Edit each ini file to include your WF database credentials. ‼️Don't forget this step!

Run the simulations:

bash migration_commands.txt

After the simulation is done, correct any problems that were noted and then edit migration_commands.txt to change the simulate option to do.

Run the live migration:

bash migration_commands.txt

The migrator will then run to migrate each database.

webscience

sean Dear Sean, thank you so much for your support to the WF exodus members.
I have various WF accounts, and on those accounts i have various webapps, most of which are wordpress installs.

Since I would not like to make mistakes, please forgive me to have some very basic questions.

Assume the following situation:
At WF i have in the webapps directory a folder "digk' which contains a typical wordpress installation.
/home/webscience/webapps/digk
Question 1: When I login to OpalStack via ssh, to which directory do i copy the boilerplate config.txt? I'm confused if i first should create an app (lets say digk), cd to that folder (e.g. /home/webscience/apps/digk) and THEN do the copy, OR does the script does all of that by itself and i can run it from the /home/webscience/ folder (ie. i copy the boilerplate to /home/webscience/config.ini)?

Question 2: Do i need to create a new API Token for every webapp (read wordpress) migration?

Question 3: Since I use the opal5 server, I assume that the config.ini server ip should be (ifconfig -a): 178.162.198.197?
Question 4: What does the 'auto_site_url' option exactly do? And maybe related to that on which url will i find the migrated wordpress? And since the DNS still needs to be updated, how can I test it? So far I know, the url has to match the wordpress config settings.

Thanks again!
Best,
Ronald

webscience

sean You guys are simply amazing! I did a migration of an old wp install, and it wasn't compatible with php 2.7. So i changed the DNS back to WF and updated ancient stuff and got it working. Then did the migration again (i expected problems by doing it twice), but no! Everything just works!! Amazing, thank you, you really saved so much trouble and that is priceless!

sean

Updated the HOWTO for an updated version of wfmigrate. Main changes:

Added the locations of all of the migrator boilerplate config.
Added some info about the messages you might see when it's running.
Updated the --help examples.

etienneh

I just want to say - this is incredible - the fact that you've paused whatever work you're doing to help all these WebFaction customers is so awesome.
I know it's a huge business opportunity for you, but you could've merely listed some instructions and that was it. Having a simulation/dry-run feature is sweet 🙂

Thanks!!!

adam_w

Way to go Opalstack! I'll echo @etienneh 's comment on the sweet simulation feature.

sean

webscience When I login to OpalStack via ssh, to which directory do i copy the boilerplate config.txt?

You can copy the file to any location you wish. Don't create the app first, since the migrator will do that for you when you run it. My example copies it to whatever your current directory is. You don't need to use config.ini as the name, so if you need a bunch of config to script several migrations you can create individual files for each one and then pass each filename at runtime.

webscience Do i need to create a new API Token for every webapp (read wordpress) migration?

Nope, you can use a single API token for all of them.

webscience Since I use the opal5 server, I assume that the config.ini server ip should be (ifconfig -a): 178.162.198.197?

Yes, 178.162.198.197 is opal5's IP. You can also get that info from your dashboard at https://my.opalstack.com/

webscience What does the 'auto_site_url' option exactly do?

That option tells our system to adjust your Wordpress general settings to set the Home and Site URLs to match your site configuration. WebFaction did this for you as well but didn't give you any way to turn it off.

webscience And maybe related to that on which url will i find the migrated wordpress?

The Wordpress migrator migrates the entire WP site from WF, ie the app, the database, and the website setup with the domains. So, the site will be set up with the domains that you had set up on the site at WebFaction.

webscience And since the DNS still needs to be updated, how can I test it?

Create an entry in your local system's hosts file to point your domain/subdomain at your Opalstack server IP, or add your free opalstacked.com subdomain to the site after the migration is complete.

webscience

sean Thank you Sean. I think i filled out everything correctly but do get this error:
Traceback (most recent call last): File "/usr/local/bin/wfmigrate", line 66, in <module> migrator = __import__(COMPONENTS[args.component]['module']).Migrator(cfg, args) File "/opt/app_migrators-1.3.1/wf_wordpress/migrator.py", line 27, in __init__ self.wf_cfg = WfCfg(cfg['webfaction']) File "/opt/app_migrators-1.3.1/lib/config_types/wf_cfg.py", line 12, in __init__ self.update_derived_attrs() File "/opt/app_migrators-1.3.1/lib/config_types/wf_cfg.py", line 16, in update_derived_attrs self.wf_session_id, self.wf_account_cfg = self.wf_api.login(self.wf_account, self.wf_password, self.wf_server.capitalize(), 2) File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python3.6/xmlrpc/client.py", line 1452, in __request verbose=self.__verbose File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python3.6/xmlrpc/client.py", line 1170, in single_request return self.parse_response(resp) File "/usr/lib64/python3.6/xmlrpc/client.py", line 1342, in parse_response return u.close() File "/usr/lib64/python3.6/xmlrpc/client.py", line 656, in close raise Fault(**self._stack[0]) xmlrpc.client.Fault: <Fault 1: 'LoginError'>

My password contains various characters which are not alphanummerical, might that be it?

psykhe

sean Create an entry in your local system's hosts file to point your domain/subdomain at your Opalstack server IP, or add your free opalstacked.com subdomain to the site after the migration is complete.

I have done this, but my free opalstacked.com subdomain does a 301 redirect to the old domain. I must overlook something but I'm totally lost as to what for the moment.

rrr

Great tutorial.

I added my OpalStack IP address to /etc/hosts and visited my domain, but I get a text version of the WordPress PHP code rather than it being executed. I know this is usually a permissions issue I can fix with chmod +x ...

To be safe, what is the command I should run to get WordPress to run when I visit the site root instead of displaying the PHP code?

sean

rrr you probably have a SetHandler directive in your .htaccess that's pointing to some stuff that WF had. Since the handler doesn't exist here, Apache's just serving up the PHP as plain text. If you remove the directives you should be good to go.

rrr

sean Indeed, I had a SetHandler pointing to a really old version of PHP. I removed that, but now I get a blank page. Am I at the point where I should reach out to support? Or should I start a separate thread?

sean

rrr try one last thing, add this alternate handler to the top of .htaccess in your app directory:

AddHandler application/x-httpd-php7 .php
Action application/x-httpd-php7 /php73-bin/php-cgi
<FilesMatch \.php$>
    SetHandler application/x-httpd-php7
</FilesMatch>

Please let me know how that goes.

rrr

sean Thanks! I still get a blank page, but I checked the logs and it looks like I have a theme that is causing a ton of problems (Atahualpa) as it uses deprecated functions in PHP, so I will have to manually swap it or something.

sean

rrr What version of PHP were you running when this site worked?

rrr

sean I think it was 5.3. I was able to get the site up and running, I just have to fix the theme as it seems my customizations were lost somehow, but that is something specific to that theme as it's a weird one. It's not a standard WP theme that "just works."

sean

rrr ah yeah, 5.3 is a no-go here. Sorry!

marsanyi

Sean: kudos from me, too, on the script. The logging output is particularly interesting, and I'm interested in the API that you've constructed for making these sorts of scripts work. Nice work!

I, too, had a bogus "SetHandler" instruction in .htaccess that resulted in the same symptom as rrr above: plaintext php output instead of executing the script. Followed your prescription, removed the handler reference, works perfectly.

Again, nice job. I spent a couple of hours trying to manually rsync and configure a copy of my (simple) WordPress installation on my local machine, and still didn't get it right. This is exactly why I'm happy to pay for a managed server.

allrite

@sean I'm getting the following error:

wfmigrate wordpress ./config.ini --mode simulate
Traceback (most recent call last):
  File "/usr/local/bin/wfmigrate", line 66, in <module>
    migrator = __import__(COMPONENTS[args.component]['module']).Migrator(cfg, args)
  File "/opt/app_migrators-1.2.1/wf_wordpress/migrator.py", line 27, in __init__
    self.wf_cfg = WfCfg(cfg['webfaction'])
  File "/opt/app_migrators-1.2.1/lib/config_types/wf_cfg.py", line 12, in __init__
    self.update_derived_attrs()
  File "/opt/app_migrators-1.2.1/lib/config_types/wf_cfg.py", line 16, in update_derived_attrs
    self.wf_session_id, self.wf_account_cfg = self.wf_api.login(self.wf_account, self.wf_password, self.wf_server.capitalize(), 2)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1170, in single_request
    return self.parse_response(resp)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1342, in parse_response
    return u.close()
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 656, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: 'Not allowed to login to machine "Web614.webfaction.com"'>

I've used and tested the credentials to log into webfaction and they work. Any ideas?

klynton

allrite Usually when we see this error in the migrators it's caused by either a typo in the config file or the wrong server/account name being specified in the config file.

allrite

klynton Thanks, but I've managed to both ssh and log into the Webfaction control panel using the credentials specified in the webfaction section of the config.ini file, which I assume would be the section responsible for the "Not allowed to login" error.

allrite

allrite Ah, the important thing is to leave out the .webfaction.com part of the server name!

klynton

allrite aha! I think the issue my be your password had a special character that Python tried to interpret. We've updated the app migrator. Can you try running it again?

psykhe

allrite I was in the same situation, had been pulling my hair for over an hour, your suggestion to leave out the 'webfaction.com' part in the [webfaction]server parameter did it for me! Thanks very much.

allrite

klynton It appears to have completed successfully now. I also removed the .webfaction.com from the server address.