HOWTO migrate your email addresses and mailboxes from WebFaction

aa11

I just ran it again on the exact same config file.

It seems to no longer be getting tripped up on GET /api/v0/mail/list/ HTTP/1.1. It finds conflicting forwards and notes:

2020-11-28 16:27:34 : WARNING : Address ADDRESS@EXAMPLE.com will be DELETED!

Which is expected. (I assume, in these cases, the version that exists at OS will be deleted, only the be immediately recreated by migrate ??)

But then it runs into another error:

2020-11-28 16:27:37 : INFO : Creating mailusers: []
Traceback (most recent call last):
  File "/usr/local/bin/wfmigrate", line 71, in <module>
    migrator.run()
  File "/opt/app_migrators-1.6.2/wf_generic_email/migrator.py", line 61, in run
    self.simulate()
  File "/opt/app_migrators-1.6.2/wf_generic_email/migrator.py", line 36, in simulate
    needed = self.wf_migrator_email.generate_needed_domains()
  File "/opt/app_migrators-1.6.2/lib/wf_migrator_email.py", line 90, in generate_needed_domains
    for address in self.generate_needed_addresses()
  File "/opt/app_migrators-1.6.2/lib/wf_migrator_email.py", line 90, in <setcomp>
    for address in self.generate_needed_addresses()
AttributeError: 'NoneType' object has no attribute 'split'

I ran it with --loglevel debug and it seems that the error occurs after migrate is done looping through the final mailuser in its queue.

This is the final readout (with --loglevel debug) before the error above:

2020-11-28 16:33:06 : DEBUG : performing GET mailuser/read/UUID-HERE
2020-11-28 16:33:06 : DEBUG : Starting new HTTPS connection (1): my.opalstack.com:443
2020-11-28 16:33:06 : DEBUG : https://my.opalstack.com:443 "GET /api/v0/mailuser/read/UUID-HERE HTTP/1.1" 200 186
2020-11-28 16:33:06 : DEBUG : {"id": "UUID-HERE", "name": "MAILUSERNAME", "pending_deletion": false, "ready": true, "imap_server": "UUID-2-HERE", "init_created": true}
2020-11-28 16:33:06 : DEBUG : got resp: <Response [200]> and result {'id': 'UUID-HERE', 'name': 'MAILUSERNAME', 'pending_deletion': False, 'ready': True, 'imap_server': 'UUID-2-HERE', 'init_created': True}

Thanks for the fix on the first part. Sorry if you're in the middle of working on this and you already know about this error.

sean

aa11 do you have an email address defined at WF that has no local part (ie the part before @domain.com)?

aa11

sean do you have an email address defined at WF that has no local part (ie the part before @domain.com)?

That's a good thought, but not that I can see.

I ran migrate the other day and it was fine. I'm pretty sure I've only made minor changes to the emails, at WF, since then.

I added one address (at WF) to test out migrate (via do and it was created at OS as expected).
I've made no changes to the domain specified as include_domains =

sean

aa11 Please check your addresses on in your Opalstack dashboard and see if you have a blank one defined here. If you find one, please delete it and try the migration again.

aa11

sean

That was it!!

There was a blank address at Opalstack.

Not sure how it got that way. But I know why the issue started happening. I ran (with do) migrate on a different domain yesterday. It was successful. It seems to have both correctly brought over Forward A but also created a blank version (local part) as well.

I'm pretty sure I couldn't have created that, because I don't think the CP allows that. So I guess, for some reason the migrator was allowed to sneak that in somehow?

I'm sorry I don't have the logs from that run of migrate.

In case it helps, this was the format of the address, and forward, in question:
a.b@example.com -> c+d@anotherdomain.com

I'd suspect the . or +. Except other email addresses, with the same format, were successfully created by migrate at OS (i.e.: Without creating the blank email address).

tl;dr: If you're reading this and have similar issues, make sure you have no email addresses, at Opalstack or WebFaction, with a BLANK local part.

Thanks @sean !!

Axel

sean after the maildirs were installed and the DNS was changed I ran mbsync for an account and had this output:

C: 0/1 B: 0/0 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Password (opalstack):
Password (oldhost):
C: 0/1 B: 0/15 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479362758, expected 1286210106; slave got 1479362758, expected 1286210106).
C: 0/1 B: 3/15 M: +0/0 *0/0 #0/0 S: +1765/1982 *0/0 #0/0
IMAP error: unexpected EOF from mail.us.opalstack.com (209.58.145.201:993)
C: 1/1 B: 4/15 M: +0/0 *0/0 #0/0 S: +1765/1982 *0/0 #0/0

Are the UIDVALIDITY error and unexpected EOF error anything to worry about ?

sean

Axel check your mailbox in webmail or an IMAP client to see if all of your folders etc are working.

If everything seems OK and you want to do another mbsync run to catch up, then that should be fine.

I don't think clearing out the mbsync cache as others have suggested is the way to go. I think for the purpose of importing mail into our system, a subsequent run should be: mbsync --pull --all

I've updated the import doc to include that --pull option.

Axel

sean I tried the sync action on another mailbox with mbsync --pull --all but it didn't pull the 6 missing messages from webfaction to opalstack, but it did put a copy of each and every message that was already there in the Opalstack mailbox (except for most, not all, of the Inbox - weird). So I now have to get rid of a couple of thousand duplicates ... Any ideas how to do this, and how to get those 6 missing messages over ? I wonder if it isn't a better idea that I delete all messages except the ones I can manually verify to NOT be present at webfaction and then to get the maildir zip again from webfaction so that Opalstack can then install them while keeping the new messages ?

This was the output:

C: 0/1 B: 0/0 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Password (opalstack):
Password (oldhost):
C: 0/1 B: 0/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479143676, expected 1286210106; slave got 1479143676, expected 1286210106).
C: 0/1 B: 1/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 2/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 3/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 4/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 20/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Warning: lost track of 174 pulled message(s)
C: 0/1 B: 21/30 M: +0/0 *0/0 #0/0 S: +520/3225 *0/0 #0/0
Socket error on mail.us.opalstack.com (209.58.145.201:993): timeout.
C: 1/1 B: 22/30 M: +0/0 *0/0 #0/0 S: +520/3225 *0/0 #0/0

sean

Axel there's a deduplication process described here that should help with the duplicate messages: https://community.opalstack.com/d/124-migration-tool-for-webfaction-mailboxes/39

If you'd like we could instead wipe your mailbox here and replace it with the most recent backup (usually a day or less old).

If it's just 6 messages or so I'd just drag them over from one account to the other in a desktop IMAP client.

Axel

sean

The sync errors turned out to be manageable (although tedious) so I didn't try the deduplication process (and risk more errors). The duplication errors were not consistent - it seems that mbsync sometimes lost track of where it was, skipping some messages and then finally copped out completely as most messages weren't duplicated at all. The output I quoted earlier on seems to reflect this somewhat flawed process. Also, the duplication errors following the mbsync actions already occurred after running "mbsync -a" (ie. running "mbsync --pull --all" afterwards didn't do anything as far as I can tell). After this experience I'm not confident that mbsync is a reliable method for large files.

To fix the few missing messages I used the download feature in roundcube at webfaction, which saves an email as an .eml file, which I then imported into roundcube at opalstack (the importer doesn't accept spaces etc in the name). The arrival date stamps show incorrect (since the messages were imported today) but the sent dates are correct. It's a bit unfortunate it's not possible to save multiple emails as an .eml file - lots of clicking to do the transfers for just a few emails.

colinguyon

Hello, sorry for the simple question, but when I try to migrate emails, I get this:

mbsync --pull --all
C: 0/1  B: 0/0  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0
Password (opalstack): 
Password (oldhost): 
IMAP command 'LOGIN <user> <pass>' returned an error: NO [AUTHENTICATIONFAILED] Authentication failed.
C: 1/1  B: 0/0  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0

I guess I don't put the correct passwords, but to be sure, what passwords should I put? The password of the email account?
Migrating accounts with wfmigrate email ./config.ini --mode do was successful, but are the passwords the same as on webfaction? Anyway to be sure I changed the password of an account and tried mbsync, but I get the same error.

EDIT: Direct Import (recommended) is recommended, so is it the preferred way in the case of a migration from Webfaction? Also, is there any "roundcube" or similar equivalent with Opalstack?

EDIT2: I asked Webfaction to give me the Maildir for all my email accounts, so I put this in my homedir at Opalstack and I sent an email to the support 😉 (but please tell me if I can do it faster myself)

Thank you

sean

colinguyon I guess I don't put the correct passwords, but to be sure, what passwords should I put?

For Opalstack use your mail user's password
For WebFaction (oldhost) use your mailbox password.

colinguyon EDIT: Direct Import (recommended) is recommended, so is it the preferred way in the case of a migration from Webfaction?

Yes, but we have some upcoming automation to help with that. Watch the forum for an announcement later today 🙂

colinguyon Also, is there any "roundcube" or similar equivalent with Opalstack?

Yes, please see: https://help.opalstack.com/article/105/webmail

colinguyon I asked Webfaction to give me the Maildir for all my email accounts, so I put this in my homedir at Opalstack and I sent an email to the support 😉 (but please tell me if I can do it faster myself)

We'll get to it as soon as possible.

colinguyon

Great, thank you 🙂

christer

I used this and then the IMAP-to-IMAP Transfer as adviced here: https://help.opalstack.com/article/107/importing-email-from-another-provider. Worked perfectly

natebeaty

This is incredibly handy for clients that just have a ton of email forwards. Saves so much time.

sean

We rolled out an email-data migrator component a couple of days ago that can migrate your mailbox data:

# wfmigrate --help
usage: wfmigrate [-h]
                 {domains,maria,psql,email,email-data,php,php-symlink,port,staticonly,staticonly-symlink,wordpress}
                 ...

WF -> Opalstack Migrator

positional arguments:
  {domains,maria,psql,email,email-data,php,php-symlink,port,staticonly,staticonly-symlink,wordpress}
                        Migration Component
    domains             Domains
    maria               MariaDB (for MySQL databases)
    psql                PostgreSQL (for PostgreSQL databases)
    email               Email
    email-data          Email Data (Import & Deduplication)
    php                 Generic PHP
    php-symlink         Generic Symlink to PHP
    port                Generic Proxy-Port (custom app listening on port)
    staticonly          Generic Static-Only (no .htaccess)
    staticonly-symlink  Generic Symlink to Static-Only (no .htaccess)
    wordpress           Wordpress

optional arguments:
  -h, --help            show this help message and exit

You do the email migrator to make the addresses and mailboxes then the email-data migrator to copy everything over.

The email data migrator works best if you're starting with an empty Opalstack mailbox but it should work if you've already copied stuff over.

Note that you will need to get your mailbox archives from WebFaction and transfer them to your Opalstack shell user before you can run the email-data migrator to import them.

All of these new migration features work like the other features: copy the config template, edit to put in your info, simulate, and run.

Hoping to have all of the new features more-fully documented this weekend.

kappleby

sean Which config template does email-data use?
I note the wf_generic_maildir one, in which case I will need to have the gzipped mailbox files copied to my opalstack account?

dragonxi

What is the right setting for server ?
[webfaction]
account = my_webfaction_account_name
account_pass = my_webfaction_account_password
ssh_pass = my_webfaction_ssh_password
server = web1 <-- this as in example ? or Web533 as in my WebFaction Control panel ?
Did a dry run of the migration:
wfmigrate email ./config2dragon.ini --mode simulate
Simulation started
Creating mailusers: []
Creating domains: []
Creating addresses: []
Simulation finished: []

Verified that it run with the configuration you've provided.
no warnings

Run it!

wfmigrate email ./config2dragon.ini --mode do

Migration completed OK

Tested:
https://my.opalstack.com/mailusers/ <-- No Mail Users
https://my.opalstack.com/mail/ <-- No Email Address

What went wrong ?
How to proceed ?

dragonxi

Does the software display statistics such as
Created xx mailboxes
Created yy addresses
after it has completed migration ok ?

sean

dragonxi i've answered your other post.

« Previous Page Next Page »