HOWTO migrate your email addresses and mailboxes from WebFaction

Axel

sean after the maildirs were installed and the DNS was changed I ran mbsync for an account and had this output:

C: 0/1 B: 0/0 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Password (opalstack):
Password (oldhost):
C: 0/1 B: 0/15 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479362758, expected 1286210106; slave got 1479362758, expected 1286210106).
C: 0/1 B: 3/15 M: +0/0 *0/0 #0/0 S: +1765/1982 *0/0 #0/0
IMAP error: unexpected EOF from mail.us.opalstack.com (209.58.145.201:993)
C: 1/1 B: 4/15 M: +0/0 *0/0 #0/0 S: +1765/1982 *0/0 #0/0

Are the UIDVALIDITY error and unexpected EOF error anything to worry about ?

sean

Axel check your mailbox in webmail or an IMAP client to see if all of your folders etc are working.

If everything seems OK and you want to do another mbsync run to catch up, then that should be fine.

I don't think clearing out the mbsync cache as others have suggested is the way to go. I think for the purpose of importing mail into our system, a subsequent run should be: mbsync --pull --all

I've updated the import doc to include that --pull option.

Axel

sean I tried the sync action on another mailbox with mbsync --pull --all but it didn't pull the 6 missing messages from webfaction to opalstack, but it did put a copy of each and every message that was already there in the Opalstack mailbox (except for most, not all, of the Inbox - weird). So I now have to get rid of a couple of thousand duplicates ... Any ideas how to do this, and how to get those 6 missing messages over ? I wonder if it isn't a better idea that I delete all messages except the ones I can manually verify to NOT be present at webfaction and then to get the maildir zip again from webfaction so that Opalstack can then install them while keeping the new messages ?

This was the output:

C: 0/1 B: 0/0 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Password (opalstack):
Password (oldhost):
C: 0/1 B: 0/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479143676, expected 1286210106; slave got 1479143676, expected 1286210106).
C: 0/1 B: 1/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 2/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 3/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 4/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 20/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Warning: lost track of 174 pulled message(s)
C: 0/1 B: 21/30 M: +0/0 *0/0 #0/0 S: +520/3225 *0/0 #0/0
Socket error on mail.us.opalstack.com (209.58.145.201:993): timeout.
C: 1/1 B: 22/30 M: +0/0 *0/0 #0/0 S: +520/3225 *0/0 #0/0

sean

Axel there's a deduplication process described here that should help with the duplicate messages: https://community.opalstack.com/d/124-migration-tool-for-webfaction-mailboxes/39

If you'd like we could instead wipe your mailbox here and replace it with the most recent backup (usually a day or less old).

If it's just 6 messages or so I'd just drag them over from one account to the other in a desktop IMAP client.

Axel

sean

The sync errors turned out to be manageable (although tedious) so I didn't try the deduplication process (and risk more errors). The duplication errors were not consistent - it seems that mbsync sometimes lost track of where it was, skipping some messages and then finally copped out completely as most messages weren't duplicated at all. The output I quoted earlier on seems to reflect this somewhat flawed process. Also, the duplication errors following the mbsync actions already occurred after running "mbsync -a" (ie. running "mbsync --pull --all" afterwards didn't do anything as far as I can tell). After this experience I'm not confident that mbsync is a reliable method for large files.

To fix the few missing messages I used the download feature in roundcube at webfaction, which saves an email as an .eml file, which I then imported into roundcube at opalstack (the importer doesn't accept spaces etc in the name). The arrival date stamps show incorrect (since the messages were imported today) but the sent dates are correct. It's a bit unfortunate it's not possible to save multiple emails as an .eml file - lots of clicking to do the transfers for just a few emails.

colinguyon

Hello, sorry for the simple question, but when I try to migrate emails, I get this:

mbsync --pull --all
C: 0/1  B: 0/0  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0
Password (opalstack): 
Password (oldhost): 
IMAP command 'LOGIN <user> <pass>' returned an error: NO [AUTHENTICATIONFAILED] Authentication failed.
C: 1/1  B: 0/0  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0

I guess I don't put the correct passwords, but to be sure, what passwords should I put? The password of the email account?
Migrating accounts with wfmigrate email ./config.ini --mode do was successful, but are the passwords the same as on webfaction? Anyway to be sure I changed the password of an account and tried mbsync, but I get the same error.

EDIT: Direct Import (recommended) is recommended, so is it the preferred way in the case of a migration from Webfaction? Also, is there any "roundcube" or similar equivalent with Opalstack?

EDIT2: I asked Webfaction to give me the Maildir for all my email accounts, so I put this in my homedir at Opalstack and I sent an email to the support 😉 (but please tell me if I can do it faster myself)

Thank you

sean

colinguyon I guess I don't put the correct passwords, but to be sure, what passwords should I put?

For Opalstack use your mail user's password
For WebFaction (oldhost) use your mailbox password.

colinguyon EDIT: Direct Import (recommended) is recommended, so is it the preferred way in the case of a migration from Webfaction?

Yes, but we have some upcoming automation to help with that. Watch the forum for an announcement later today 🙂

colinguyon Also, is there any "roundcube" or similar equivalent with Opalstack?

Yes, please see: https://help.opalstack.com/article/105/webmail

colinguyon I asked Webfaction to give me the Maildir for all my email accounts, so I put this in my homedir at Opalstack and I sent an email to the support 😉 (but please tell me if I can do it faster myself)

We'll get to it as soon as possible.

colinguyon

Great, thank you 🙂

christer

I used this and then the IMAP-to-IMAP Transfer as adviced here: https://help.opalstack.com/article/107/importing-email-from-another-provider. Worked perfectly

natebeaty

This is incredibly handy for clients that just have a ton of email forwards. Saves so much time.

sean

We rolled out an email-data migrator component a couple of days ago that can migrate your mailbox data:

# wfmigrate --help
usage: wfmigrate [-h]
                 {domains,maria,psql,email,email-data,php,php-symlink,port,staticonly,staticonly-symlink,wordpress}
                 ...

WF -> Opalstack Migrator

positional arguments:
  {domains,maria,psql,email,email-data,php,php-symlink,port,staticonly,staticonly-symlink,wordpress}
                        Migration Component
    domains             Domains
    maria               MariaDB (for MySQL databases)
    psql                PostgreSQL (for PostgreSQL databases)
    email               Email
    email-data          Email Data (Import & Deduplication)
    php                 Generic PHP
    php-symlink         Generic Symlink to PHP
    port                Generic Proxy-Port (custom app listening on port)
    staticonly          Generic Static-Only (no .htaccess)
    staticonly-symlink  Generic Symlink to Static-Only (no .htaccess)
    wordpress           Wordpress

optional arguments:
  -h, --help            show this help message and exit

You do the email migrator to make the addresses and mailboxes then the email-data migrator to copy everything over.

The email data migrator works best if you're starting with an empty Opalstack mailbox but it should work if you've already copied stuff over.

Note that you will need to get your mailbox archives from WebFaction and transfer them to your Opalstack shell user before you can run the email-data migrator to import them.

All of these new migration features work like the other features: copy the config template, edit to put in your info, simulate, and run.

Hoping to have all of the new features more-fully documented this weekend.

kappleby

sean Which config template does email-data use?
I note the wf_generic_maildir one, in which case I will need to have the gzipped mailbox files copied to my opalstack account?

dragonxi

What is the right setting for server ?
[webfaction]
account = my_webfaction_account_name
account_pass = my_webfaction_account_password
ssh_pass = my_webfaction_ssh_password
server = web1 <-- this as in example ? or Web533 as in my WebFaction Control panel ?
Did a dry run of the migration:
wfmigrate email ./config2dragon.ini --mode simulate
Simulation started
Creating mailusers: []
Creating domains: []
Creating addresses: []
Simulation finished: []

Verified that it run with the configuration you've provided.
no warnings

Run it!

wfmigrate email ./config2dragon.ini --mode do

Migration completed OK

Tested:
https://my.opalstack.com/mailusers/ <-- No Mail Users
https://my.opalstack.com/mail/ <-- No Email Address

What went wrong ?
How to proceed ?

dragonxi

Does the software display statistics such as
Created xx mailboxes
Created yy addresses
after it has completed migration ok ?

sean

dragonxi i've answered your other post.

sean

kappleby /opt/app_migrators/wf_generic_maildir/config-template.ini is the correct template:

[opalstack]
account = my_opalstack_account_name
apitoken = my_opalstack_api_token
imap_server = imap1.us.opalstack.com

[mailsync]
source_basedir = ~/wf_archive_delivery/mailboxes
import = True
dedup = False

[mailusers]
opal_mailuser_1 = opal_mailuser_1_password
opal_mailuser_2 = opal_mailuser_2_password
opal_mailuser_3 = opal_mailuser_3_password

Note the source_basedir = ~/wf_archive_delivery/mailboxes option - that directory needs to contain your unzipped mailboxes, ie the user directories that each contain a Maildir directory.

If it crashes or fails due to memory, IMAP errors, etc it's safe to repeat the run. DO NOT delete your ~/.mbsync directory between runs.

If you notice duplicate messages in your mailbox after everything is synced, then run again with the import and dedup options flipped, eg:

import = False
dedup = True

You can also use dedup = True from the start, and the migrator will make a de-duplication pass after the migration is complete.

tmciver

I think I need a bit of hand-holding here.

sean Note the source_basedir = ~/wf_archive_delivery/mailboxes option - that directory needs to contain your unzipped mailboxes, ie the user directories that each contain a Maildir directory.

Where do I get "unzipped mailboxes"? Am I supposed to ssh into WF and zip up mailboxes? Is that documented somewhere? I was under the impression that the migration of email data would happen over the network when the script was run as it was for syncing the email accounts.

sean

Pro tip for the email-data migrator:

The migrator tends to fail occasionally, usually because we have to reload the IMAP service on the mail servers frequently to install new mail users.

When that happens, it's safe to re-run the migrator but doing that over and over manually isn't fun.

Instead you can wrap the command in a while loop like so:

while ! wfmigrate email-data maildir.ini --mode do; do sleep 5; done

This is saying "if this command fails wait five seconds and run it again".

Use this loop only if you're certain that your migration configuration is valid - if the failures are due to an error in your config then repeating the command won't accomplish anything.

reecardo

[opal1 ~]$ wfmigrate email ./config.ini --mode simulate
2020-12-13 20:56:52 : INFO : Simulation started
2020-12-13 20:56:54 : ERROR : Mailuser 'info' error for field 'name': Mail User with this name already exists.

So I guess someone else already as a mail user named info? Any way we could just prefix the new mail user with the account username, so the already taken info turns into prefix_info?

sean

reecardo Any way we could just prefix the new mail user with the account username, so the already taken info turns into prefix_info?

Sorry, not possible at this time. You'll need to put info in exclude_mailboxes in your config and migrate that one to a different mail user manually later.

rdelrossi

In the midst of simulating a mail data import I'm getting a persistent error at the same spot:

C: 0/1  B: 9/15  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0
Error: channel robertdelrossi: slave Untitled cannot be opened.
C: 1/1  B: 15/15  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0

## The following error was encountered during operation: Command mbsync -a -c /home/rdelrossi/mbsync_robertdelrossi.conf --noop returned nonzero exit status 1
##
## Please ensure that you are modifying the latest version of the configuration template:
##
##     /opt/app_migrators/wf_generic_maildir/config-template.ini
##
## If you have any questions, please contact Support, and include the follwing error information for debugging:

2020-12-13 23:45:11 : ERROR : Command mbsync -a -c /home/rdelrossi/mbsync_robertdelrossi.conf --noop returned nonzero exit status 1
Traceback (most recent call last):
  File "/usr/local/bin/wfmigrate", line 102, in <module>
    migrator.run()
  File "/opt/app_migrators-1.10.3/wf_generic_maildir/migrator.py", line 67, in run
    self.simulate()
  File "/opt/app_migrators-1.10.3/wf_generic_maildir/migrator.py", line 48, in simulate
    self.import_all(simulate=True)
  File "/opt/app_migrators-1.10.3/wf_generic_maildir/migrator.py", line 24, in import_all
    self.opal_cfg.imap_server['hostname'], mailuser_name, mailuser_pass, simulate=simulate,
  File "/opt/app_migrators-1.10.3/lib/generic_importer_maildir.py", line 135, in do_import
    self.run_mbsync(mailuser_name, simulate=simulate)
  File "/opt/app_migrators-1.10.3/lib/generic_importer_maildir.py", line 92, in run_mbsync
    raise RuntimeError(f"Command {' '.join(cmd)} returned nonzero exit status {result.returncode}")
RuntimeError: Command mbsync -a -c /home/rdelrossi/mbsync_robertdelrossi.conf --noop returned nonzero exit status 1

sean

rdelrossi I've pinged the migrator developer and will follow up later after they've looked at this.

sean

rdelrossi Ok, here's what's happening:

You have an existing folder in your mail user directory here at Opalstack.
The migrator runs a program mbsync to do the actual import from your archive to your mail user.
The migrator was designed to import mail into your mail user ie it does not sync folders and mail back down from your mail user directory into your archive.
(here's the fun part we just learned) mbsync returns a failure exit code whenever it completes a run that results with the source and destination are out of sync, even if you configure it to do exactly that and it does it successfully.

We don't want to ignore failure exit codes, and we don't want to modify your source mail archives, so we don't have a quick fix to make this go away.

Here are three ways you can deal with it:

Trust that it's done everything successfully when you see those "Slave Whatever cannot be opened" errors and a "mbsync ... returned nonzero exit status 1" traceback. The developer assures me that this is the case for this particular scenario, ie using wfmigrate to load mail into a populated mailbox.
Start with an empty mailbox on our side and avoid the problem altogether.
Run mbsync or imapsync on your own without the migrator: Importing Email from Another Provider | Opalstack Support Center

« Previous Page Next Page »