HOWTO migrate your email addresses and mailboxes from WebFaction

kappleby

sean Which config template does email-data use?
I note the wf_generic_maildir one, in which case I will need to have the gzipped mailbox files copied to my opalstack account?

sean

kappleby /opt/app_migrators/wf_generic_maildir/config-template.ini is the correct template:

[opalstack]
account = my_opalstack_account_name
apitoken = my_opalstack_api_token
imap_server = imap1.us.opalstack.com

[mailsync]
source_basedir = ~/wf_archive_delivery/mailboxes
import = True
dedup = False

[mailusers]
opal_mailuser_1 = opal_mailuser_1_password
opal_mailuser_2 = opal_mailuser_2_password
opal_mailuser_3 = opal_mailuser_3_password

Note the source_basedir = ~/wf_archive_delivery/mailboxes option - that directory needs to contain your unzipped mailboxes, ie the user directories that each contain a Maildir directory.

If it crashes or fails due to memory, IMAP errors, etc it's safe to repeat the run. DO NOT delete your ~/.mbsync directory between runs.

If you notice duplicate messages in your mailbox after everything is synced, then run again with the import and dedup options flipped, eg:

import = False
dedup = True

You can also use dedup = True from the start, and the migrator will make a de-duplication pass after the migration is complete.

tmciver

I think I need a bit of hand-holding here.

sean Note the source_basedir = ~/wf_archive_delivery/mailboxes option - that directory needs to contain your unzipped mailboxes, ie the user directories that each contain a Maildir directory.

Where do I get "unzipped mailboxes"? Am I supposed to ssh into WF and zip up mailboxes? Is that documented somewhere? I was under the impression that the migration of email data would happen over the network when the script was run as it was for syncing the email accounts.

sean

Pro tip for the email-data migrator:

The migrator tends to fail occasionally, usually because we have to reload the IMAP service on the mail servers frequently to install new mail users.

When that happens, it's safe to re-run the migrator but doing that over and over manually isn't fun.

Instead you can wrap the command in a while loop like so:

while ! wfmigrate email-data maildir.ini --mode do; do sleep 5; done

This is saying "if this command fails wait five seconds and run it again".

Use this loop only if you're certain that your migration configuration is valid - if the failures are due to an error in your config then repeating the command won't accomplish anything.

reecardo

[opal1 ~]$ wfmigrate email ./config.ini --mode simulate
2020-12-13 20:56:52 : INFO : Simulation started
2020-12-13 20:56:54 : ERROR : Mailuser 'info' error for field 'name': Mail User with this name already exists.

So I guess someone else already as a mail user named info? Any way we could just prefix the new mail user with the account username, so the already taken info turns into prefix_info?

sean

reecardo Any way we could just prefix the new mail user with the account username, so the already taken info turns into prefix_info?

Sorry, not possible at this time. You'll need to put info in exclude_mailboxes in your config and migrate that one to a different mail user manually later.

rdelrossi

In the midst of simulating a mail data import I'm getting a persistent error at the same spot:

C: 0/1  B: 9/15  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0
Error: channel robertdelrossi: slave Untitled cannot be opened.
C: 1/1  B: 15/15  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0

## The following error was encountered during operation: Command mbsync -a -c /home/rdelrossi/mbsync_robertdelrossi.conf --noop returned nonzero exit status 1
##
## Please ensure that you are modifying the latest version of the configuration template:
##
##     /opt/app_migrators/wf_generic_maildir/config-template.ini
##
## If you have any questions, please contact Support, and include the follwing error information for debugging:

2020-12-13 23:45:11 : ERROR : Command mbsync -a -c /home/rdelrossi/mbsync_robertdelrossi.conf --noop returned nonzero exit status 1
Traceback (most recent call last):
  File "/usr/local/bin/wfmigrate", line 102, in <module>
    migrator.run()
  File "/opt/app_migrators-1.10.3/wf_generic_maildir/migrator.py", line 67, in run
    self.simulate()
  File "/opt/app_migrators-1.10.3/wf_generic_maildir/migrator.py", line 48, in simulate
    self.import_all(simulate=True)
  File "/opt/app_migrators-1.10.3/wf_generic_maildir/migrator.py", line 24, in import_all
    self.opal_cfg.imap_server['hostname'], mailuser_name, mailuser_pass, simulate=simulate,
  File "/opt/app_migrators-1.10.3/lib/generic_importer_maildir.py", line 135, in do_import
    self.run_mbsync(mailuser_name, simulate=simulate)
  File "/opt/app_migrators-1.10.3/lib/generic_importer_maildir.py", line 92, in run_mbsync
    raise RuntimeError(f"Command {' '.join(cmd)} returned nonzero exit status {result.returncode}")
RuntimeError: Command mbsync -a -c /home/rdelrossi/mbsync_robertdelrossi.conf --noop returned nonzero exit status 1

sean

rdelrossi I've pinged the migrator developer and will follow up later after they've looked at this.

sean

rdelrossi Ok, here's what's happening:

You have an existing folder in your mail user directory here at Opalstack.
The migrator runs a program mbsync to do the actual import from your archive to your mail user.
The migrator was designed to import mail into your mail user ie it does not sync folders and mail back down from your mail user directory into your archive.
(here's the fun part we just learned) mbsync returns a failure exit code whenever it completes a run that results with the source and destination are out of sync, even if you configure it to do exactly that and it does it successfully.

We don't want to ignore failure exit codes, and we don't want to modify your source mail archives, so we don't have a quick fix to make this go away.

Here are three ways you can deal with it:

Trust that it's done everything successfully when you see those "Slave Whatever cannot be opened" errors and a "mbsync ... returned nonzero exit status 1" traceback. The developer assures me that this is the case for this particular scenario, ie using wfmigrate to load mail into a populated mailbox.
Start with an empty mailbox on our side and avoid the problem altogether.
Run mbsync or imapsync on your own without the migrator: Importing Email from Another Provider | Opalstack Support Center

sean

tmciver You will need to get your mailbox archive from WebFaction and they will drop that into your home directory there.

Once you have it there you can copy it to your Opalstack server and unzip it.

Once it's unzipped you should be ready to run the migrator.

The migrator was designed this way because in most cases the account owner (you) won't know all of their mailbox passwords.

If you do know your WF mailbox passwords and you'd rather do an IMAP-to-IMAP sync yourself we have documentation for that here: Importing Email from Another Provider | Opalstack Support Center

charles_h

sean

Hi Sean,

Regarding the archived emails, how can I get that? Ask WF staffs help me zip the mailbox and save it on the server?
Or I can do it myself?

Thanks

jake74

sean hey Sean, I'm getting the same error as rdelrossi with the RuntimeError: Command mbsync -a -c /home/ME/mbsync_MAILBOX.conf --noop returned nonzero exit status 1

I checked the mailbox and it was empty.

Did you both solve this? I see the conversation went off forum.

rdelrossi

sean Thanks for the reply, Sean. So, I deleted all the mailboxes (i.e., Email | Mail Users) and re-ran the simulation but got the same error. I noted upon deletion of the mailboxes that "Hard delete will happen 24 hours after being marked," and was wondering if that might be why I am still getting that error.

sean

rdelrossi I noted upon deletion of the mailboxes that "Hard delete will happen 24 hours after being marked," and was wondering if that might be why I am still getting that error.

Yes, exactly that. If you want to delete the mailboxes right away use the trash icon that appears in the row for the mailboxes that are pending deletion.

Your Wordpress problem is off-topic for this thread, but I'll look into it and will ping you via email.

rdelrossi

Transferring a site over from WebFaction. The wfmigrate simulation passes without a problem, but on the actual migration I get a persistent error here:

2020-12-14 15:41:32 : INFO : Extracting DB User password from rdelrossi_rsnsta.opalmigration-local.passwd
2020-12-14 15:41:32 : INFO : Writing sqlpasswd file to rdelrossi_rsnsta.opalmigration-local.passwd
2020-12-14 15:41:32 : INFO : Changing app config DB credentials in /home/rdelrossi/apps/rsnstatssite/wp-config.php
2020-12-14 15:41:32 : INFO : Importing DB using sqlpasswd file rdelrossi_rsnsta.opalmigration-local.passwd
ERROR 1045 (28000): Access denied for user 'rdelrossi_rsnsta'@'localhost' (using password: YES)

## The following error was encountered during operation: Command "['mysql', '--defaults-extra-file=rdelrossi_rsnsta.opalmigration-local.passwd', '-u', 'rdelrossi_rsnsta', 'rdelrossi_rsnsta']" exited with status 1
##
## Please ensure that you are modifying the latest version of the configuration template:
##
##     /opt/app_migrators/wf_wordpress/config-template.ini
##
## If you have any questions, please contact Support, and include the follwing error information for debugging:

2020-12-14 15:41:32 : ERROR : Command "['mysql', '--defaults-extra-file=rdelrossi_rsnsta.opalmigration-local.passwd', '-u', 'rdelrossi_rsnsta', 'rdelrossi_rsnsta']" exited with status 1
Traceback (most recent call last):
  File "/usr/local/bin/wfmigrate", line 102, in <module>
    migrator.run()
  File "/opt/app_migrators-1.10.3/wf_wordpress/migrator.py", line 214, in run
    self.migrate()
  File "/opt/app_migrators-1.10.3/wf_wordpress/migrator.py", line 190, in migrate
    self.wf_migrator_maria.import_db()
  File "/opt/app_migrators-1.10.3/lib/wf_migrator_maria.py", line 159, in import_db
    raise RuntimeError(f'Command "{cmd}" exited with status {result.returncode}')
RuntimeError: Command "['mysql', '--defaults-extra-file=rdelrossi_rsnsta.opalmigration-local.passwd', '-u', 'rdelrossi_rsnsta', 'rdelrossi_rsnsta']" exited with status

rdelrossi

sean Thank you and apologies for the mis-post. Really appreciate your quick and helpful replies, Sean. As for emailing me, given the current of affairs, please use robert.delrossi at gmail.com.

-- Robert

rdelrossi

sean Sean, I've been able to delete all the mailboxes but one, robertdelrossi. The trash can icon doesn't seem to clear it. Is this just a matter of waiting a day for the "hard delete?" Sorry to be so persistent about this, but being down on email is a big deal.

mikz

Thanks very much for developing this tool, and thanks to the helpful WebFaction support person who recommended OpalStack. I'm pleased to see things looking fairly familiar here.

I was getting the following error:

2020-12-14 16:44:01 : ERROR : <Fault 1: 'Not allowed to login to machine "Web1"'>
Traceback (most recent call last):
  File "/usr/local/bin/wfmigrate", line 101, in <module>
    migrator = __import__(module).Migrator(cfg, args)
  File "/opt/app_migrators-1.10.3/wf_generic_email/migrator.py", line 18, in __init__
    self.wf_cfg = WfCfg(cfg['webfaction'])
  File "/opt/app_migrators-1.10.3/lib/config_types/wf_cfg.py", line 10, in __init__
    self.update_derived_attrs()
  File "/opt/app_migrators-1.10.3/lib/config_types/wf_cfg.py", line 18, in update_derived_attrs
    self.refresh_api_connection()
  File "/opt/app_migrators-1.10.3/lib/config_types/wf_cfg.py", line 14, in refresh_api_connection
    self.api_session_id, self.wf_account_cfg = self.api.login(self.wf_account, self.wf_password, self.wf_server.capitalize(), 2)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1170, in single_request
    return self.parse_response(resp)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1342, in parse_response
    return u.close()
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 656, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: 'Not allowed to login to machine "Web1"'>

Update

I found the problem. The server property in my config.ini needed to be set to the server I found under Account Services←(linky!) in my WebFaction control panel (e.g. Web123), and not the hostname I SSH to.

I'm leaving the error message posted here, in case anybody else makes the same mistake.

sean

mikz regarding the WF server name, please follow the example in the config template. The template shows web1 as an example, ie just the short hostname without the domain. For example, if your WF server is web555.webfaction.com then use web555.

Regarding the blank entries in your config, a space after = doesn't matter to wfmigrate. Your config example looks OK to me. If you still have concerns then the simulate mode is there for you to test your config.

mikz

Another question: The bottom of my config.ini looks like this:

[email]
include_domains =
include_subdomains = True

exclude_domains =
exclude_subdomains = True

exclude_mailboxes =

It says to leave the domains blank, if I want to include all of them. I don't remember how Python handles empty strings in .ini files—I'm not sure if I ever tried, TBH. But is this the right way? Does it matter whether there are spaces after the equals signs? Thanks.

mikz

On my first live run, I got this error:

2020-12-14 17:16:38 : INFO : Creating domains: [{'name': 'domain5.com'}, {'name': 'junction.oneofmydomains.net'}, {'name': 'anotherofmydomains.com'}, {'name': 'domain3.org'}, {'name': 'thejunction.oneofmydomains.net'}, {'name': 'domain4.org'}, {'name': 'mywfusername.webfactional.com'}, {'name': 'oneofmydomains.net'}, {'name': 'domain6.org'}]
2020-12-14 17:16:38 : DEBUG : performing POST domain/add/ with [{'name': 'domain5.com'}, {'name': 'junction.oneofmydomains.net'}, {'name': 'anotherofmydomains.com'}, {'name': 'domain3.org'}, {'name': 'thejunction.oneofmydomains.net'}, {'name': 'domain4.org'}, {'name': 'mywfusername.webfactional.com'}, {'name': 'oneofmydomains.net'}, {'name': 'domain6.org'}]
2020-12-14 17:16:38 : DEBUG : Starting new HTTPS connection (1): my.opalstack.com:443
2020-12-14 17:17:28 : DEBUG : https://my.opalstack.com:443 "POST /api/v0/domain/add/ HTTP/1.1" 400 137
2020-12-14 17:17:28 : DEBUG : {"error": "Domain name \"mywfusername.webfactional.com\" is already in use by another user, if you believe you own this domain contact support."}
2020-12-14 17:17:28 : DEBUG : got resp: <Response [400]> and result {'error': 'Domain name "mywfusername.webfactional.com" is already in use by another user, if you believe you own this domain contact support.'}

## The following error was encountered during operation: Unexpected status_code: 400
##
## Please ensure that you are modifying the latest version of the configuration template:
##
##     /opt/app_migrators/wf_generic_email/config-template.ini
##
## If you have any questions, please contact Support, and include the follwing error information for debugging:

2020-12-14 17:17:28 : ERROR : Unexpected status_code: 400
Traceback (most recent call last):
  File "/usr/local/bin/wfmigrate", line 102, in <module>
    migrator.run()
  File "/opt/app_migrators-1.10.3/wf_generic_email/migrator.py", line 92, in run
    self.migrate()
  File "/opt/app_migrators-1.10.3/wf_generic_email/migrator.py", line 71, in migrate
    self.wf_migrator_email.create_domains()
  File "/opt/app_migrators-1.10.3/lib/wf_migrator_email.py", line 144, in create_domains
    created = self.opal_cfg.api.domains.ensure(needed)
  File "/opt/app_migrators-1.10.3/lib/opalapi.py", line 155, in ensure
    created = self.create(tocreate, wait=wait)
  File "/opt/app_migrators-1.10.3/lib/opalapi.py", line 140, in create
    create_result = self.api.post(create_url, tocreate, ensure_status=[200])
  File "/opt/app_migrators-1.10.3/lib/opalapi.py", line 532, in post
    return self.request_result(urlpath, dataObj, 'POST', ensure_status)
  File "/opt/app_migrators-1.10.3/lib/opalapi.py", line 525, in request_result
    resp, result = self.request(urlpath, dataObj, method, ensure_status)
  File "/opt/app_migrators-1.10.3/lib/opalapi.py", line 521, in request
    raise RuntimeError(f'Unexpected status_code: {resp.status_code}')
RuntimeError: Unexpected status_code: 400

I've changed the names to protect the guilty, e.g. 'mywfusername' is a pseudonym for my WebFaction username. But 'webfactional.com' is verbatim, which I imagine is the actual cause of this exception. I think 'mywfusername.webfactional.com' was a test server I set up when I first moved to WebFaction, back in the 2000s. I don't really need it, but I don't see how to delete it in the WebFaction control panel.

I'll keep trying, but can you tell me if I can just rerun wfmigrate after that? Or do I have to run some clean-up step first?

Also, most of my domains actually point to the same mailbox, including multiple subdomains (e.g. 'thejunction.oneofmydomains.net' and 'junction.oneofmydomains.net' point to the same web server, and both have a catch-all going to the same mailbox). Is this going to be a problem?

Update

I added the domain it choked on to exclude_domains, and then it seemed to work. I'll write a separate post if anything looks messed up. Thanks.

sean

mikz it seems like must have a mywfusername.webfactional.com email address set up at WF. If you delete that address this problem should go away.

Or, try exclude_domains = mywfusername.webfactional.com in your config and give it another go.

sean

mikz Also, most of my domains actually point to the same mailbox, including multiple subdomains (e.g. 'thejunction.oneofmydomains.net' and 'junction.oneofmydomains.net' point to the same web server, and both have a catch-all going to the same mailbox). Is this going to be a problem?

Nope!

sean

mikz I've changed the names to protect the guilty, but 'mywfusername' is indeed my WebFaction username.

👍 More people should do this.

« Previous Page Next Page »