HOWTO migrate your email addresses and mailboxes from WebFaction

sean

Axel there will always be a bit of propagation time, never really a clean switch. You usually have to watch mail at both ends until propagation is done, then do a final mailbox sync to get the Opalstack side caught up.

Curious as to what you mean by "pause email receipt" - what would you expect to happen to a message that was sent to your domain in this paused state?

Axel

sean ok so I should first change the DNS servers, then wait a day, and then move the old mail (I'm planning to use the zipped maildir method), assuming that is what you mean with "mailbox sync" ?

email is a "store-and-forward" system, right ? So I was thinking that email would be held at some other server (for at most a day) until it is accepted by the Opalstack server where I'd read it from.

sean

Axel sean ok so I should first change the DNS servers, then wait a day, and then move the old mail (I'm planning to use the zipped maildir method), assuming that is what you mean with "mailbox sync" ?

You could do it that way, or you could let us load your zipped maildirs in advance and then you can follow up with a quick mbsync after the switch.

Axel email is a "store-and-forward" system, right ? So I was thinking that email would be held at some other server (for at most a day) until it is accepted by the Opalstack server where I'd read it from.

The thing is, who is going to store it?

If the goal is to pause mail (ie don't attempt to deliver it to any MX server during DNS propagation) you'd have to configure every SMTP server that sends mail to your domain to defer delivery until some future date. If that's not done, then the mail is going to arrive at whichever MX the SMTP server picks up in DNS.

Axel

sean after the maildirs were installed and the DNS was changed I ran mbsync for an account and had this output:

C: 0/1 B: 0/0 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Password (opalstack):
Password (oldhost):
C: 0/1 B: 0/15 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479362758, expected 1286210106; slave got 1479362758, expected 1286210106).
C: 0/1 B: 3/15 M: +0/0 *0/0 #0/0 S: +1765/1982 *0/0 #0/0
IMAP error: unexpected EOF from mail.us.opalstack.com (209.58.145.201:993)
C: 1/1 B: 4/15 M: +0/0 *0/0 #0/0 S: +1765/1982 *0/0 #0/0

Are the UIDVALIDITY error and unexpected EOF error anything to worry about ?

aa11

sean

I think something has broken with the migrator (or maybe WF?).

I just ran it (with simulate) on the WF account/domain that I ran originally. Nothing has changed in the file.

Here's the readout:

2020-11-28 02:51:14 : INFO : Simulation started
2020-11-28 02:51:16 : WARNING : Email address EMAILADDRESSHERE points to nonexistent or excluded mailbox MAILBOXNAME
Traceback (most recent call last):
  File "/usr/local/bin/wfmigrate", line 71, in <module>
    migrator.run()
  File "/opt/app_migrators-1.6.2/wf_generic_email/migrator.py", line 61, in run
    self.simulate()
  File "/opt/app_migrators-1.6.2/wf_generic_email/migrator.py", line 30, in simulate
    self.wf_migrator_email.create_addresses(simulate=True)
  File "/opt/app_migrators-1.6.2/lib/wf_migrator_email.py", line 137, in create_addresses
    toretain, todelete, tocreate = self.opal_cfg.api.addresses.check_ensure(needed)
  File "/opt/app_migrators-1.6.2/lib/opalapi.py", line 58, in check_ensure
    existing = self.get_all()
  File "/opt/app_migrators-1.6.2/lib/opalapi.py", line 166, in get_all
    for existing_summary in self.api.get(self.model + '/list/', ensure_status=[200])[self.model + 's']
  File "/opt/app_migrators-1.6.2/lib/opalapi.py", line 507, in get
    return self.request_result(urlpath, None, 'GET', ensure_status)
  File "/opt/app_migrators-1.6.2/lib/opalapi.py", line 503, in request_result
    resp, result = self.request(urlpath, dataObj, method, ensure_status)
  File "/opt/app_migrators-1.6.2/lib/opalapi.py", line 499, in request
    raise RuntimeError(f'Unexpected status_code: {resp.status_code}')
RuntimeError: Unexpected status_code: 400

That initial 2020-11-28 02:51:16 : WARNING makes total sense. I got it before and it's accurate.

But it's also how I know (I think) that the migrator IS connecting to WF. So something is happening after that, as the migrator tries to complete one of it's tasks.

Of the top of my head: Maybe OS has hit a rate-limit?

sean please run the sim with --loglevel debug added to the command and then amend your original post with the output, thanks.

I think this is the relevant portion:

2020-11-28 03:26:51 : DEBUG : performing GET mail/list/
2020-11-28 03:26:51 : DEBUG : Starting new HTTPS connection (1): my.opalstack.com:443
2020-11-28 03:26:52 : DEBUG : https://my.opalstack.com:443 "GET /api/v0/mail/list/ HTTP/1.1" 400 18
2020-11-28 03:26:52 : DEBUG : {"error": "error"}
2020-11-28 03:26:52 : DEBUG : got resp: <Response [400]> and result {'error': 'error'}

It seems to get through the requests to GET /api/v0/domain/read/UUIDHERE HTTP/1.1.

But then errors out when it starts the requests to GET /api/v0/mail/list/ HTTP/1.1.

After that, it's the same Traceback as above.

@sean If you need to see everything (i.e.: directly from my account) you know where to find it ;-) Or let me know and I can open a ticket.

sean

aa11 please run the sim with --loglevel debug added to the command and then amend your original post with the output, thanks.

pjrobertson

I'm getting the following error when trying to copy over a single domain:

2020-11-28 09:21:49 : DEBUG : performing GET domain/list/
2020-11-28 09:21:49 : DEBUG : Starting new HTTPS connection (1): my.opalstack.com:443
2020-11-28 09:21:50 : DEBUG : https://my.opalstack.com:443 "GET /api/v0/domain/list/ HTTP/1.1" 200 97
2020-11-28 09:21:50 : DEBUG : {"domains": [{"id": "XXXXXX-86af-4b5b-a79a-f43329321bd4", "name": "username.opalstacked.com"}]}
2020-11-28 09:21:50 : DEBUG : got resp: <Response [200]> and result {'domains': [{'id': 'XXXXXX-86af-4b5b-a79a-f43329321bd4', 'name': 'username.opalstacked.com'}]}
2020-11-28 09:21:50 : DEBUG : performing GET domain/read/XXXXXX-86af-4b5b-a79a-f43329321bd4
2020-11-28 09:21:50 : DEBUG : Starting new HTTPS connection (1): my.opalstack.com:443
2020-11-28 09:21:50 : DEBUG : https://my.opalstack.com:443 "GET /api/v0/domain/read/XXXXXX-86af-4b5b-a79a-f43329321bd4 HTTP/1.1" 200 82
2020-11-28 09:21:50 : DEBUG : {"id": "XXXXXX-86af-4b5b-a79a-f43329321bd4", "name": "username.opalstacked.com"}
2020-11-28 09:21:50 : DEBUG : got resp: <Response [200]> and result {'id': 'XXXXXX-86af-4b5b-a79a-f43329321bd4', 'name': 'username.opalstacked.com'}
2020-11-28 09:21:50 : DEBUG : performing GET domain/list/
2020-11-28 09:21:50 : DEBUG : Starting new HTTPS connection (1): my.opalstack.com:443
2020-11-28 09:21:51 : DEBUG : https://my.opalstack.com:443 "GET /api/v0/domain/list/ HTTP/1.1" 200 97
2020-11-28 09:21:51 : DEBUG : {"domains": [{"id": "XXXXXX-86af-4b5b-a79a-f43329321bd4", "name": "username.opalstacked.com"}]}
2020-11-28 09:21:51 : DEBUG : got resp: <Response [200]> and result {'domains': [{'id': 'XXXXXX-86af-4b5b-a79a-f43329321bd4', 'name': 'username.opalstacked.com'}]}
2020-11-28 09:21:51 : DEBUG : performing GET domain/read/XXXXXX-86af-4b5b-a79a-f43329321bd4
2020-11-28 09:21:51 : DEBUG : Starting new HTTPS connection (1): my.opalstack.com:443
2020-11-28 09:21:52 : DEBUG : https://my.opalstack.com:443 "GET /api/v0/domain/read/XXXXXX-86af-4b5b-a79a-f43329321bd4 HTTP/1.1" 200 82
2020-11-28 09:21:52 : DEBUG : {"id": "XXXXXX-86af-4b5b-a79a-f43329321bd4", "name": "username.opalstacked.com"}
2020-11-28 09:21:52 : DEBUG : got resp: <Response [200]> and result {'id': 'XXXXXX-86af-4b5b-a79a-f43329321bd4', 'name': 'username.opalstacked.com'}
2020-11-28 09:21:52 : INFO : Creating domains: [{'name': 'example.com'}]
2020-11-28 09:21:52 : DEBUG : performing POST domain/add/ with [{'name': 'example.com'}]
2020-11-28 09:21:52 : DEBUG : Starting new HTTPS connection (1): my.opalstack.com:443
2020-11-28 09:21:53 : DEBUG : https://my.opalstack.com:443 "POST /api/v0/domain/add/ HTTP/1.1" 400 52
2020-11-28 09:21:53 : DEBUG : [{"name":["Domain with this name already exists."]}]
2020-11-28 09:21:53 : DEBUG : got resp: <Response [400]> and result [{'name': ['Domain with this name already exists.']}]
Traceback (most recent call last):
  File "/usr/local/bin/wfmigrate", line 71, in <module>
    migrator.run()
  File "/opt/app_migrators-1.6.2/wf_generic_email/migrator.py", line 70, in run
    self.migrate()
  File "/opt/app_migrators-1.6.2/wf_generic_email/migrator.py", line 51, in migrate
    self.wf_migrator_email.create_domains()
  File "/opt/app_migrators-1.6.2/lib/wf_migrator_email.py", line 132, in create_domains
    created = self.opal_cfg.api.domains.ensure(needed)
  File "/opt/app_migrators-1.6.2/lib/opalapi.py", line 155, in ensure
    created = self.create(tocreate, wait=wait)
  File "/opt/app_migrators-1.6.2/lib/opalapi.py", line 140, in create
    create_result = self.api.post(create_url, tocreate, ensure_status=[200])
  File "/opt/app_migrators-1.6.2/lib/opalapi.py", line 510, in post
    return self.request_result(urlpath, dataObj, 'POST', ensure_status)
  File "/opt/app_migrators-1.6.2/lib/opalapi.py", line 503, in request_result
    resp, result = self.request(urlpath, dataObj, method, ensure_status)
  File "/opt/app_migrators-1.6.2/lib/opalapi.py", line 499, in request
    raise RuntimeError(f'Unexpected status_code: {resp.status_code}')
RuntimeError: Unexpected status_code: 400

pjrobertson

Update: I realised it was to do with my domain 'example.com' being registered on two different Opalstack accounts (I have two accounts, one is a testing account, one is a live account). Once I deleted the domain from my testing account, it worked.

I sent a ticket about this - I don't see why I can't have a domain name registered to two different opalstack accounts.

aa11

I just ran it again on the exact same config file.

It seems to no longer be getting tripped up on GET /api/v0/mail/list/ HTTP/1.1. It finds conflicting forwards and notes:

2020-11-28 16:27:34 : WARNING : Address ADDRESS@EXAMPLE.com will be DELETED!

Which is expected. (I assume, in these cases, the version that exists at OS will be deleted, only the be immediately recreated by migrate ??)

But then it runs into another error:

2020-11-28 16:27:37 : INFO : Creating mailusers: []
Traceback (most recent call last):
  File "/usr/local/bin/wfmigrate", line 71, in <module>
    migrator.run()
  File "/opt/app_migrators-1.6.2/wf_generic_email/migrator.py", line 61, in run
    self.simulate()
  File "/opt/app_migrators-1.6.2/wf_generic_email/migrator.py", line 36, in simulate
    needed = self.wf_migrator_email.generate_needed_domains()
  File "/opt/app_migrators-1.6.2/lib/wf_migrator_email.py", line 90, in generate_needed_domains
    for address in self.generate_needed_addresses()
  File "/opt/app_migrators-1.6.2/lib/wf_migrator_email.py", line 90, in <setcomp>
    for address in self.generate_needed_addresses()
AttributeError: 'NoneType' object has no attribute 'split'

I ran it with --loglevel debug and it seems that the error occurs after migrate is done looping through the final mailuser in its queue.

This is the final readout (with --loglevel debug) before the error above:

2020-11-28 16:33:06 : DEBUG : performing GET mailuser/read/UUID-HERE
2020-11-28 16:33:06 : DEBUG : Starting new HTTPS connection (1): my.opalstack.com:443
2020-11-28 16:33:06 : DEBUG : https://my.opalstack.com:443 "GET /api/v0/mailuser/read/UUID-HERE HTTP/1.1" 200 186
2020-11-28 16:33:06 : DEBUG : {"id": "UUID-HERE", "name": "MAILUSERNAME", "pending_deletion": false, "ready": true, "imap_server": "UUID-2-HERE", "init_created": true}
2020-11-28 16:33:06 : DEBUG : got resp: <Response [200]> and result {'id': 'UUID-HERE', 'name': 'MAILUSERNAME', 'pending_deletion': False, 'ready': True, 'imap_server': 'UUID-2-HERE', 'init_created': True}

Thanks for the fix on the first part. Sorry if you're in the middle of working on this and you already know about this error.

sean

aa11 do you have an email address defined at WF that has no local part (ie the part before @domain.com)?

aa11

sean do you have an email address defined at WF that has no local part (ie the part before @domain.com)?

That's a good thought, but not that I can see.

I ran migrate the other day and it was fine. I'm pretty sure I've only made minor changes to the emails, at WF, since then.

I added one address (at WF) to test out migrate (via do and it was created at OS as expected).
I've made no changes to the domain specified as include_domains =

sean

aa11 Please check your addresses on in your Opalstack dashboard and see if you have a blank one defined here. If you find one, please delete it and try the migration again.

aa11

sean

That was it!!

There was a blank address at Opalstack.

Not sure how it got that way. But I know why the issue started happening. I ran (with do) migrate on a different domain yesterday. It was successful. It seems to have both correctly brought over Forward A but also created a blank version (local part) as well.

I'm pretty sure I couldn't have created that, because I don't think the CP allows that. So I guess, for some reason the migrator was allowed to sneak that in somehow?

I'm sorry I don't have the logs from that run of migrate.

In case it helps, this was the format of the address, and forward, in question:
a.b@example.com -> c+d@anotherdomain.com

I'd suspect the . or +. Except other email addresses, with the same format, were successfully created by migrate at OS (i.e.: Without creating the blank email address).

tl;dr: If you're reading this and have similar issues, make sure you have no email addresses, at Opalstack or WebFaction, with a BLANK local part.

Thanks @sean !!

sean

Axel check your mailbox in webmail or an IMAP client to see if all of your folders etc are working.

If everything seems OK and you want to do another mbsync run to catch up, then that should be fine.

I don't think clearing out the mbsync cache as others have suggested is the way to go. I think for the purpose of importing mail into our system, a subsequent run should be: mbsync --pull --all

I've updated the import doc to include that --pull option.

Axel

sean I tried the sync action on another mailbox with mbsync --pull --all but it didn't pull the 6 missing messages from webfaction to opalstack, but it did put a copy of each and every message that was already there in the Opalstack mailbox (except for most, not all, of the Inbox - weird). So I now have to get rid of a couple of thousand duplicates ... Any ideas how to do this, and how to get those 6 missing messages over ? I wonder if it isn't a better idea that I delete all messages except the ones I can manually verify to NOT be present at webfaction and then to get the maildir zip again from webfaction so that Opalstack can then install them while keeping the new messages ?

This was the output:

C: 0/1 B: 0/0 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Password (opalstack):
Password (oldhost):
C: 0/1 B: 0/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479143676, expected 1286210106; slave got 1479143676, expected 1286210106).
C: 0/1 B: 1/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 2/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 3/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 4/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Error: channel main: UIDVALIDITY of both master and slave changed
(master got 1479151396, expected 1330182929; slave got 1479151396, expected 1330182929).
C: 0/1 B: 20/30 M: +0/0 *0/0 #0/0 S: +0/0 *0/0 #0/0
Warning: lost track of 174 pulled message(s)
C: 0/1 B: 21/30 M: +0/0 *0/0 #0/0 S: +520/3225 *0/0 #0/0
Socket error on mail.us.opalstack.com (209.58.145.201:993): timeout.
C: 1/1 B: 22/30 M: +0/0 *0/0 #0/0 S: +520/3225 *0/0 #0/0

sean

Axel there's a deduplication process described here that should help with the duplicate messages: https://community.opalstack.com/d/124-migration-tool-for-webfaction-mailboxes/39

If you'd like we could instead wipe your mailbox here and replace it with the most recent backup (usually a day or less old).

If it's just 6 messages or so I'd just drag them over from one account to the other in a desktop IMAP client.

Axel

sean

The sync errors turned out to be manageable (although tedious) so I didn't try the deduplication process (and risk more errors). The duplication errors were not consistent - it seems that mbsync sometimes lost track of where it was, skipping some messages and then finally copped out completely as most messages weren't duplicated at all. The output I quoted earlier on seems to reflect this somewhat flawed process. Also, the duplication errors following the mbsync actions already occurred after running "mbsync -a" (ie. running "mbsync --pull --all" afterwards didn't do anything as far as I can tell). After this experience I'm not confident that mbsync is a reliable method for large files.

To fix the few missing messages I used the download feature in roundcube at webfaction, which saves an email as an .eml file, which I then imported into roundcube at opalstack (the importer doesn't accept spaces etc in the name). The arrival date stamps show incorrect (since the messages were imported today) but the sent dates are correct. It's a bit unfortunate it's not possible to save multiple emails as an .eml file - lots of clicking to do the transfers for just a few emails.

colinguyon

Hello, sorry for the simple question, but when I try to migrate emails, I get this:

mbsync --pull --all
C: 0/1  B: 0/0  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0
Password (opalstack): 
Password (oldhost): 
IMAP command 'LOGIN <user> <pass>' returned an error: NO [AUTHENTICATIONFAILED] Authentication failed.
C: 1/1  B: 0/0  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0

I guess I don't put the correct passwords, but to be sure, what passwords should I put? The password of the email account?
Migrating accounts with wfmigrate email ./config.ini --mode do was successful, but are the passwords the same as on webfaction? Anyway to be sure I changed the password of an account and tried mbsync, but I get the same error.

EDIT: Direct Import (recommended) is recommended, so is it the preferred way in the case of a migration from Webfaction? Also, is there any "roundcube" or similar equivalent with Opalstack?

EDIT2: I asked Webfaction to give me the Maildir for all my email accounts, so I put this in my homedir at Opalstack and I sent an email to the support 😉 (but please tell me if I can do it faster myself)

Thank you

sean

colinguyon I guess I don't put the correct passwords, but to be sure, what passwords should I put?

For Opalstack use your mail user's password
For WebFaction (oldhost) use your mailbox password.

colinguyon EDIT: Direct Import (recommended) is recommended, so is it the preferred way in the case of a migration from Webfaction?

Yes, but we have some upcoming automation to help with that. Watch the forum for an announcement later today 🙂

colinguyon Also, is there any "roundcube" or similar equivalent with Opalstack?

Yes, please see: https://help.opalstack.com/article/105/webmail

colinguyon I asked Webfaction to give me the Maildir for all my email accounts, so I put this in my homedir at Opalstack and I sent an email to the support 😉 (but please tell me if I can do it faster myself)

We'll get to it as soon as possible.

colinguyon

Great, thank you 🙂

« Previous Page Next Page »