Migration tool for Webfaction mailboxes

vsajip

Thanks to Sean reminding me that Opalstack doesn't support unsecured IMAP (Doh!) I got an imapsync-based solution apparently working. I've added it to the gist I linked to earlier. The new script is called migmail.py and I transferred a mailbox with over 22000 messages with memory usage around 350MB RAM, so comfortably inside a Value Stack plan limits :-)

It's run using python3 migmail.py CSV-FILE-WITH-MAILBOX-INFO and will create a log directory LOG_imapsync in the directory where it's run.

Any feedback welcome!

ggates1020

vsajip Thank for sharing! Question, can it be run more than once to grab new messages or would you have to remove the target mailbox and rerun?

kesavan

vsajip Wow! great.. the script is awesome. Migrated 80K mails in 8+ Hrs (18GB overall size)

aa11

vsajip Thanks to Sean reminding me that Opalstack doesn't support unsecured IMAP (Doh!) I got an imapsync-based solution apparently working.

I'm trying to get this working, but seem to be running into a similar security issue:

Host2 failure: can not open imap connection on host2 [mail.us.opalstack.com] with user [MAILUSERNAME]: timeout waiting 120s for data from server
Exiting with return value 10 (EXIT_CONNECTION_FAILURE) 1/50 nb_errors/max_errors

And if I add the --ssl2 flag to imapsync which @vsajip did in migmail.py:

Host2 failure: can not open imap connection on host2 [mail.us.opalstack.com] with user [MAILUSERNAME]: Unable to connect to mail.us.opalstack.com: IO::Socket::IP configuration failed SSL connect attempt failed because of handshake problems SSL wants a read first
Exiting with return value 10 (EXIT_CONNECTION_FAILURE) 1/50 nb_errors/max_errors

Although I'm not sure why I'd need to add that because that's also an imapsync default and I don't have any arguments to pass for that.

Other Info:

I'm forcing --port2 993 (for OS) because imapsync was telling me: "sslcheck did not detected open ssl port 993. Will use standard 143 port."
I also tried --host2 imap1.us.opalstack.com. Same result.
I tried resetting the password for my Mail User in the OS CP.
I can connect to this Mail User using a standard email app.
imapsync connects to WF without any problems.

I'm not sure which imapsync security options you updated @vsajip to get it to work.

I must be doing something dumb.

Thanks.

miopa

vsajip Just tried this, worked as expected out of the box.

I run it on the opalstack server, CSV file looks like

oldhost,olduser,oldpwd,newhost,newuser,newpwd,migrate
mail.webfaction.com,wfmailbox1,wfpass1,mail.de.opalstack.com,opalmailbox1,opalpass1,1

no errors, 6822 messages (1.620 GiB) transferred in less than 10 minutes

vsajip

ggates1020 You should be able to run imapsync again to just get new messages - try it yourself on one or two mailboxes. It seems to work on a folder-by-folder basis so I think you might get duplicates if you move stuff around in the source mailbox before re-syncing. I've stopped using mbsync because memory usage balloons to beyond my quota limits even on a modestly-sized mailbox.

Nis

Another option would be the OVH Mail Migrator, which I've used in the past (not with OpalStack, though).
https://omm.ovh.net

sean

Nis Another option would be the OVH Mail Migrator, which I've used in the past (not with OpalStack, though).

We used to recommend a similar service back in the WebFaction days but I never felt great about telling someone to hand their mail credentials over to a third-party website. 😬

pjrobertson

Any updates on when the migration tool to move messages from WF to Opalstack might be ready? I think this is the last missing piece of the puzzle we don't have.

klynton

We're working on it now and it's fairly close to being done.

GaelP

Hi. Is there a tool to move messages from WF to Opalstack now? I've tried with mbsync and imapsync without success.

sean

GaelP Not yet, but we hope to have it ready soon.

In the meantime if mbsync, imapsync, etc don't work for you then please try the Direct Import option.

aa11

I just tried with --debugssl 4

It still times out and ends with a OK free ctx error.

Host2 failure: can not open imap connection on host2 [mail.us.opalstack.com] with user [MAILUSERNAME]: error(-1) waiting 120s for data from server: Interrupted system call Interrupted system call
Exiting with return value 10 (EXIT_CONNECTION_FAILURE) 1/50 nb_errors/max_errors
Disconnecting from host1 mail.webfaction.com user1 MAILUSERNAME
DEBUG: .../IO/Socket/SSL.pm:1948: free ctx 140446561243520 open=140446561243520
DEBUG: .../IO/Socket/SSL.pm:1953: free ctx 140446561243520 callback
DEBUG: .../IO/Socket/SSL.pm:1956: OK free ctx 140446561243520

This is what I found about that:

I also found that it might have something to do with special characters in my password:

But I tried changing it to something simpler (sill with an * because the OS CP requires that) and got the same errors.

elena

I have been using mbsync to migrate mailboxes from Webfaction as described here and have had no problem. https://help.opalstack.com/article/107/importing-email-from-another-provider#imaptoimap-transfer

One after the other... If you use the same ssh user, just make sure you remove the contents of the folder /.mbsync after each run, as user vsajip explained here.

Thanks

aa11

elena One after the other... If you use the same ssh user, just make sure you remove the contents of the folder /.mbsync after each run, as user vsajip explained here.

Thanks for your suggestion.

I just tried mbsync.

Oooohhff.

I agree with what everyone wrote above. Not great.

It seems to work on the first run, but subsequent runs duplicate messages. (Yes, I cleared out the ~/.mbsync dir after each run.)

I rant it on a really small sample (less than 500 total messages) and it was slow (which I can deal with). But I can imagine how much ram it's taking up and, given the experiences above, wouldn't be surprised if it crashed on my main mail boxes. And getting duplicate messages on subsequent runs makes that a bad idea.

I still haven't been able to figure out the security/SSL issue with imapsync.

I just tried @vsajip's migratemail.py script. It too was able to login to WF but not OS. But it generated a slightly different error:

Host2: connecting and login on host2 [mail.us.opalstack.com] port [993] with user [EMAILBOX]
Host2 failure: can not open imap connection on host2 [mail.us.opalstack.com] with user [EMAILBOX]: Unable to connect to mail.us.opalstack.com: IO::Socket::IP configuration failed SSL connect attempt failed because of handshake problems SSL wants a read first
Exiting with return value 10 (EXIT_CONNECTION_FAILURE) 1/50 nb_errors/max_errors

I found the following, but not definitive fix:

aa11

miopa Just tried this, worked as expected out of the box.

I run it on the opalstack server, CSV file looks like

Thanks.

I've been running it on my local machine (not the source or destination). So maybe that was the problem.

I tried running it directly on the OS server and had the same issues. imapsync was even more finicky with the special characters in the password (something I've read in their docs). So I change both passes to something without so many special characters and it still didn't work.

@sean Can you check the server logs at imap1.us.opalstack.com or mail.us.opalstack.com (those might actually be the same) to see what the errors look like on that end?

Maybe I've been blacklisted because of failed login attempts in my early tests?? (Although I never saw any messages like that.)

Thanks!

sean

aa11 I can't reproduce the SSL error you're reporting, imapsync doens't complain at all when connecting to our IMAP servers.

Based on all of the feedback here I think I'll add some official docs for imapsync and make that the recommended tool instead of mbsync.

aa11

sean I can't reproduce the SSL error you're reporting, imapsync doens't complain at all when connecting to our IMAP servers.

Thanks for trying.

I'm completely at a loss.

I'm doing what should be out-of-the-box simple.

I've never been able to connect to the OS server. Whether from my machine or OS.
I switched to a pretty simple password.
I use --port2 993 because imapsync seems to default to 143 (but I've tried the default too).
I've tried putting the password directly in the command via --password2 and via an external file via --passfile2.
I've tried MAILUSERNAME@mail.us.opalstack.com (and imap1.us.opalstack.com) as --user2.

I even just created a new mail user at OS and tried to use that as --user2. Same result.

sean

aa11 feel free to email support with the exact details and we'll look into it.

elena

I meant removing the contents of the folder /.mbsync after each different user. I do not think you have to remove the contents if you are synchronizing the same user again, but I must admit that I have not tested running subsequent runs on the same user.

aa11

elena I meant removing the contents of the folder /.mbsync after each different user. I do not think you have to remove the contents if you not synchronizing the same user again, but I must admit that I have not tested running subsequent runs on the same user.

Thanks for clarifying.

I tried running mbsybc with and without removing the contents of ~/mysync. Always the same shell and email user.

Subsequent runs always duplicated emails in the destination.

« Previous Page Next Page »