Migration tool for Webfaction mailboxes

aa11

elena One after the other... If you use the same ssh user, just make sure you remove the contents of the folder /.mbsync after each run, as user vsajip explained here.

Thanks for your suggestion.

I just tried mbsync.

Oooohhff.

I agree with what everyone wrote above. Not great.

It seems to work on the first run, but subsequent runs duplicate messages. (Yes, I cleared out the ~/.mbsync dir after each run.)

I rant it on a really small sample (less than 500 total messages) and it was slow (which I can deal with). But I can imagine how much ram it's taking up and, given the experiences above, wouldn't be surprised if it crashed on my main mail boxes. And getting duplicate messages on subsequent runs makes that a bad idea.

I still haven't been able to figure out the security/SSL issue with imapsync.

I just tried @vsajip's migratemail.py script. It too was able to login to WF but not OS. But it generated a slightly different error:

Host2: connecting and login on host2 [mail.us.opalstack.com] port [993] with user [EMAILBOX]
Host2 failure: can not open imap connection on host2 [mail.us.opalstack.com] with user [EMAILBOX]: Unable to connect to mail.us.opalstack.com: IO::Socket::IP configuration failed SSL connect attempt failed because of handshake problems SSL wants a read first
Exiting with return value 10 (EXIT_CONNECTION_FAILURE) 1/50 nb_errors/max_errors

I found the following, but not definitive fix:

miopa

vsajip Just tried this, worked as expected out of the box.

I run it on the opalstack server, CSV file looks like

oldhost,olduser,oldpwd,newhost,newuser,newpwd,migrate
mail.webfaction.com,wfmailbox1,wfpass1,mail.de.opalstack.com,opalmailbox1,opalpass1,1

no errors, 6822 messages (1.620 GiB) transferred in less than 10 minutes

aa11

miopa Just tried this, worked as expected out of the box.

I run it on the opalstack server, CSV file looks like

Thanks.

I've been running it on my local machine (not the source or destination). So maybe that was the problem.

I tried running it directly on the OS server and had the same issues. imapsync was even more finicky with the special characters in the password (something I've read in their docs). So I change both passes to something without so many special characters and it still didn't work.

@sean Can you check the server logs at imap1.us.opalstack.com or mail.us.opalstack.com (those might actually be the same) to see what the errors look like on that end?

Maybe I've been blacklisted because of failed login attempts in my early tests?? (Although I never saw any messages like that.)

Thanks!

sean

aa11 I can't reproduce the SSL error you're reporting, imapsync doens't complain at all when connecting to our IMAP servers.

Based on all of the feedback here I think I'll add some official docs for imapsync and make that the recommended tool instead of mbsync.

aa11

sean I can't reproduce the SSL error you're reporting, imapsync doens't complain at all when connecting to our IMAP servers.

Thanks for trying.

I'm completely at a loss.

I'm doing what should be out-of-the-box simple.

I've never been able to connect to the OS server. Whether from my machine or OS.
I switched to a pretty simple password.
I use --port2 993 because imapsync seems to default to 143 (but I've tried the default too).
I've tried putting the password directly in the command via --password2 and via an external file via --passfile2.
I've tried MAILUSERNAME@mail.us.opalstack.com (and imap1.us.opalstack.com) as --user2.

I even just created a new mail user at OS and tried to use that as --user2. Same result.

sean

aa11 feel free to email support with the exact details and we'll look into it.

elena

I meant removing the contents of the folder /.mbsync after each different user. I do not think you have to remove the contents if you are synchronizing the same user again, but I must admit that I have not tested running subsequent runs on the same user.

aa11

elena I meant removing the contents of the folder /.mbsync after each different user. I do not think you have to remove the contents if you not synchronizing the same user again, but I must admit that I have not tested running subsequent runs on the same user.

Thanks for clarifying.

I tried running mbsybc with and without removing the contents of ~/mysync. Always the same shell and email user.

Subsequent runs always duplicated emails in the destination.

sean

aa11 I think the problem is that you're using MAILUSERNAME@mail.us.opalstack.com instead of just MAILUSERNAME.

aa11

I was able to get it to work... By running it on the OS server instead of my local machine.

I assume there's something wrong with certificates, my OS, or my install of imapsync.

I tried running imapsync on OS in the past (see above), but I guess those earlier settings weren't having it.

This is what I use and it seems to be running well:

imapsync --dry \
--host1 mail.webfaction.com --ssl1 --user1 WF_MAILUSERNAME --passfile1 ./pw-temp.txt \
--host2 mail.us.opalstack.com --ssl2 --user2 OS_MAILUSERNAME --passfile2 ./pw-temp.txt

I use the --passfile option so it doesn't get logged.
You don't need --ssl1, since WF seems to do ok with imapsync defaults. But I leave it just for symmetry.

I really appreciates the output (and functionality!?) over mbsync

I tried to get @vsajip's Python script to work (on OS) but couldn't because of special characters in my passwords. Which is a known gotcha with imapsync:

https://github.com/imapsync/imapsync/issues/99#issuecomment-416574189
https://github.com/imapsync/imapsync/issues/85
https://github.com/imapsync/imapsync/search?q=password&type=issues

I tried all manner of wrapping the passwords with single quotes, double quotes, quotes within quotes, and escaping characters (in the .csv) file. But kept getting login failure errors.

If not for those characters, I bet @vsajip's script would work very well.

Thanks for everyone's help!

nval

I have used the following sequence after partial success with mbsync:

Perform a migration using migmail.py by @vsajip with the relevant .csv, as mentioned
earlier.
After the migration, Thunderbird didn't show any folders on my opalstack
mail account, because by default it "only shows folders you are subscribed to".
Other mail apps may not have this "feature", but for Thunderbird you can do
this: Go to account settings -> server settings -> advanced, and un-check "only show
folders you are subscribed to". Alternatively, you can right-click on the
account name on the left pane, select "Subscribe...", wait for it to find all
your email folders and then subscribe to all of them or as needed.

If, like me, you tried several methods to migrate your emails and you weren't
sure if and when to delete the .mbsync folder before finally using migmail.py,
you may end up with several copies of each email. Here's what you can do:

Go to https://github.com/quentinsf/IMAPdedup and copy imapdedup.py to your
ssh at opalstack. Then run:

python3.6 imapdedup.py -x -s imap1.de.opalstack.com -u your_username -w your_password -l > mailfolders.txt

This produced a list of folders in your mailbox. Now open mailfolders.txt and add double quotes to the start and end of every line. This will help process folder names with spaces and special characters or locales.

Then run:

cat mailfolders.txt | xargs python3.6 imapdedup.py -x -s imap1.de.opalstack.com -u your_username -w your_password

You can add a -n to see a dry run before doing a real run. If you look through the output carefully, you may notice imapdedup recommending that you use -c to catch duplicate messages that may not have a message-id. Go ahead and do that:

cat mailfolders.txt | xargs python3.6 imapdedup.py -x -s imap1.de.opalstack.com -u your_username -w your_password -c

That was it! I ran this sequence for all my team's emails after I had tried other options that had
actually part-worked and it cleand up and completed the migration for us. I
seem to have the same rough number of emails (in the order of a few thousands).
To be honest, I don't really look at emails that have fallen behind the
horizon, but it's nice to know I din't lose any in the move... as far as I know!

saebel

nval After the migration, Thunderbird didn't show any folders on my opalstack
mail account, because by default it "only shows folders you are subscribed to".
Other mail apps may not have this "feature", but for Thunderbird you can do
this: Go to account settings -> server settings -> advanced, and un-check "only show
folders you are subscribed to". Alternatively, you can right-click on the
account name on the left pane, select "Subscribe...", wait for it to find all
your email folders and then subscribe to all of them or as needed.

THANK YOU SO MUCH FOR POSTING THIS! I thought my subfolders hadn't migrated.

christer

Hi all,
I have tracked this discussion from the beginning, but kept myself from trying any of the suggested techniques, hoping that there will be an "official" way to migrate the mailboxes from Webfaction. But with my Webfaction account coming to an end in 2 weeks I begin to worry. Should I still wait or go for it now?

sean

christer we hope to have the email migration tool ready within the few days.

ngkabra

Thanks to everybody who posted their experiences here.

I successfully used imapsync to move a large mailbox (40k messages, 14GB) from webfaction to opalstack. It took 70 minutes and 547MB memory.

I used @aa11's commandline and it worked as expected.

Before that I wasted a bunch of time trying to get mbsync to work, but it kept using up all my memory, and getting killed. Finally found @vsajip's comment about the relative memory use of mbsync vs imapsync.

Anyway, @sean I do think you should update the official guidelines to at least mention the potential memory problems with mbsync and some basic instructions for imapsync usage.

« Previous Page Next Page »