Migration tool for Webfaction mailboxes

melvyn

Hello everybody!
Are there any tools available to migrate mailboxes from Webfaction to Opalstack?

sean

melvyn Are there any tools available to migrate mailboxes from Webfaction to Opalstack?

Not yet, but they're coming soon!

In the meantime, you can set up your mail users manually on our end and then migrate your email using one of the methods described here: Importing Email from Another Provider

samoube

Appreciate all the work you guys are doing. Do you have an ETA on the email tool, or info on what part of the migration it will help with?

Should I go ahead and manually create all the mailboxes, domains, and email addresses in Opal, or will the migration tool assist with that?

The other thing I hadn't quite figured out is the proper sequence for handling the migration and cutover. Was hoping to use the directory migration method. Should we get all the mail setup at Opal, cut over domains over (by changing the name servers), then migrate the mailboxes from WF? I wasn't sure if migrating the mailboxes after new mail is already being received to Opal risks overwriting anything, or if the process will merge the old with anything in the new mailbox.

Best,
Sam

sean

samoube Sam when the migration tool is ready it will create all of the dashboard stuff for you. Not ready yet tho, and no ETA. Obviously migrations are a high priority so we'll have it done as soon as possible.

sean

samoube The order is:

Add your domain to the panel here.
Add your mail users and addresses here.
Import your mail archives here.
When you're ready (ie when you've got your sites set up etc) point your domain at us.
While DNS is propagating, you will continue to receive mail at the old host. That will slowly taper off as DNS makes its way through the system, you'll see less mail arriving there and more mail arriving here.
Once you've stopped receiving mail at the old host (hopefully no more than a day or two) you can copy those last messages over.

kylotan

Add me to the list of people who want such a tool! I managed to migrate one mailbox over via the mbsync instructions but that program is pretty user-hostile. It requires a separate configuration file for each mailbox, and on one of my mailboxes it spat out the error channel main: UIDVALIDITY of both master and slave changed so I'm stuck.

andrebr

kylotan I have the same problem. Of 4 mailboxes, one had this problem today

vsajip

kylotan I had this problem, too. Between mailboxes, I deleted a folder ~/.mbsync created by the previous mbsync run. In theory mbsync can process multiple mailboxes in one go (by adding "channels" to the config file) but I prefer to check each mailbox as I go, so I didn't try this. The high RAM usage of mbsync might make multiple mailboxes in one run problematic, too.

I tried imapsync, too, but there's some issue where it can't log in to the Opalstack IMAP server! I've logged an issue for that. At least mbsync appears to have got through the mailboxes I've migrated so far.

sean

Agreed, we're seeing a few problems from customers having problems with mbsync. Turns out it really like to eat RAM, go figure :\

If mbsync isn't working for you then please get your mail archives from WF and upload them to a shell user on your web server, then create your mail users, then contact us to let us know so that we can load the mail for you.

samoube

sean Thanks Sean!

kylotan

Aha, clearing out the .mbsync directory seems to have fixed the obscure UIDs error I was getting. So, what I've done now is:

(a) created a separate config file in .mbsyncrc format for each mailbox - like the ones at the Importing Email from Another Provider link, but with different filenames, and with the Pass field set to the mailbox passwords for ease of use.

(b) Created the following sync_email script in my home directory so I can clear out the .mbsync directory and run the migration for a mailbox with a specific config file, like so:

# Clear out any previous run
cd ~/.mbsync
rm *
cd ..

# Sync fresh using the config provided as the argument
mbsync -a -c $1

Then made it executable via chmod u+x sync_email.

(c) I can sync over the latest emails by running the script with the relevant config file as an argument, e.g. ./sync_email mbsyncrc-kylotan

Judging by the web mail interface this seems to work, so hopefully I'll be able to run that, redirect my domain to Opalstack, then run it again to pull over any emails that came in after the first migration, with zero mail downtime.

Hope this helps someone!

sean

kylotan and vsajip you guys are awesome for sharing your solutions here, thank you! ❤️

kylotan

I spoke too soon - trying to migrate one of my larger mailboxes failed half-way. Starting it up again (without cleaning the .mbsync directory) managed another couple of thousand mails before the process and my shell session was forcibly terminated. 😃 Maybe this was the RAM problem mentioned above...

vsajip

Inspired by the example given by @kylotan, I've created a more generic Python script which should be able to migrate multiple mailboxes, as long as there are no out-of-memory problems. Here's the script as a GitHub Gist. Feel free to try it out and give feedback if you find it useful (or if you hit any problems with it). It just needs to be run in an unencumbered directory (as it creates configuration files and sync state directories in the current directory when it's run). It takes a CSV file as input, the format of which is described in the comment at the start of the script.

vsajip

Thanks to Sean reminding me that Opalstack doesn't support unsecured IMAP (Doh!) I got an imapsync-based solution apparently working. I've added it to the gist I linked to earlier. The new script is called migmail.py and I transferred a mailbox with over 22000 messages with memory usage around 350MB RAM, so comfortably inside a Value Stack plan limits :-)

It's run using python3 migmail.py CSV-FILE-WITH-MAILBOX-INFO and will create a log directory LOG_imapsync in the directory where it's run.

Any feedback welcome!

ggates1020

vsajip Thank for sharing! Question, can it be run more than once to grab new messages or would you have to remove the target mailbox and rerun?

kesavan

vsajip Wow! great.. the script is awesome. Migrated 80K mails in 8+ Hrs (18GB overall size)

aa11

vsajip Thanks to Sean reminding me that Opalstack doesn't support unsecured IMAP (Doh!) I got an imapsync-based solution apparently working.

I'm trying to get this working, but seem to be running into a similar security issue:

Host2 failure: can not open imap connection on host2 [mail.us.opalstack.com] with user [MAILUSERNAME]: timeout waiting 120s for data from server
Exiting with return value 10 (EXIT_CONNECTION_FAILURE) 1/50 nb_errors/max_errors

And if I add the --ssl2 flag to imapsync which @vsajip did in migmail.py:

Host2 failure: can not open imap connection on host2 [mail.us.opalstack.com] with user [MAILUSERNAME]: Unable to connect to mail.us.opalstack.com: IO::Socket::IP configuration failed SSL connect attempt failed because of handshake problems SSL wants a read first
Exiting with return value 10 (EXIT_CONNECTION_FAILURE) 1/50 nb_errors/max_errors

Although I'm not sure why I'd need to add that because that's also an imapsync default and I don't have any arguments to pass for that.

Other Info:

I'm forcing --port2 993 (for OS) because imapsync was telling me: "sslcheck did not detected open ssl port 993. Will use standard 143 port."
I also tried --host2 imap1.us.opalstack.com. Same result.
I tried resetting the password for my Mail User in the OS CP.
I can connect to this Mail User using a standard email app.
imapsync connects to WF without any problems.

I'm not sure which imapsync security options you updated @vsajip to get it to work.

I must be doing something dumb.

Thanks.

miopa

vsajip Just tried this, worked as expected out of the box.

I run it on the opalstack server, CSV file looks like

oldhost,olduser,oldpwd,newhost,newuser,newpwd,migrate
mail.webfaction.com,wfmailbox1,wfpass1,mail.de.opalstack.com,opalmailbox1,opalpass1,1

no errors, 6822 messages (1.620 GiB) transferred in less than 10 minutes

vsajip

ggates1020 You should be able to run imapsync again to just get new messages - try it yourself on one or two mailboxes. It seems to work on a folder-by-folder basis so I think you might get duplicates if you move stuff around in the source mailbox before re-syncing. I've stopped using mbsync because memory usage balloons to beyond my quota limits even on a modestly-sized mailbox.

Nis

Another option would be the OVH Mail Migrator, which I've used in the past (not with OpalStack, though).
https://omm.ovh.net

sean

Nis Another option would be the OVH Mail Migrator, which I've used in the past (not with OpalStack, though).

We used to recommend a similar service back in the WebFaction days but I never felt great about telling someone to hand their mail credentials over to a third-party website. 😬

pjrobertson

Any updates on when the migration tool to move messages from WF to Opalstack might be ready? I think this is the last missing piece of the puzzle we don't have.