Migrated email duplicated?

kylotan

I copied over my mailboxes from Webfaction using mbsync, switched my domain to Opalstack, changed the credentials in my email client, polled for new email - and it seems to be pulling down an extra copy of every email 😐

Anyone know if I did something wrong? This is using Thunderbird, if the client matters.

sean

kylotan if you log into our webmail are the messages duplicated there as well?

andrebr

@kylotan I had the same problem in an email box.

I solved the problem by installing this extension on Thunderbird.

https://addons.thunderbird.net/pt-BR/thunderbird/addon/removedupes/?src=userprofile

Set the email box and it will remove duplicates

kylotan

The messages are only duplicated in Thunderbird, it seems. Webmail, and the mail client on my iPhone (when I finally got it to accept the changed password, a battle in itself!) both just show 1 copy of the mail.

And to be clear, it's only an extra copy of migrated emails that ended up duplicated locally - new incoming emails seem to work as expected.

sean

kylotan ok, sounds like a local issue with TB. Can you try compacting the folders with the duplicated messages to see if that clears it up? Also maybe clear the TB cache? (Settings > General then scroll down to "Network & Disk Space")

vsajip

I found that if mbysnc is killed due to out-of-memory problems and re-run later, it can duplicate emails (which do show up in webmail, too). It does warn about this with a "lost track of XXX messages" indication. Anyway, I now seem to have imapsync working, and that hasn't caused any out-of-memory problems.

kylotan

It legitimately had just made extra copies of every email, failing to recognise that they all existed locally already. I had to install the extension linked above to remove duplicates.

I just tried changing the login details for a second mailbox and the same thing happened there. I suspect Thunderbird may not be capable of working out that the emails are the same, once the server has changed. 😐

sean

kylotan . I suspect Thunderbird may not be capable of working out that the emails are the same, once the server has changed.

That sound like it might be the case, surprising though :\

kylotan

A reply from their support forum says:

"Tbird still had its locally stored copies downloaded from the old IMAP server and is reusing the same mbox files with the new server. You can avoid this next time by deleting mbox files so that Thunderbird can recreate them with messages from the new server. "

I don't think this makes sense though, because somehow it has enough information to know not to download the same email twice from the old server, but not enough to know not to download the email again from a new host. I'll inquire further.

kylotan

Okay, follow up - first, I'm not entirely convinced the (well-meaning) respondent on the TB forum is correct, because he suggested that TB has no way of knowing whether a local email matches a remote email (which would make IMAP sync impossible, so can't really be true).

Second, I just migrated another mailbox via mbsync and this one looks fine in Thunderbird - which suggests that it is a mbsync issue, quite possibly related to the "lost track" error that @vsajip mentioned, and which I have seen myself on previous migrations.

So, I'm either going to try via imapsync instead next time, or might try and do future migration via creating new Thunderbird accounts and dragging the emails across. Unless a supported migration tool appears first. 🙂

elena

The webapp migration tool is fabulous. I hope the email migration tool appears soon 🙂
If we want to opt-out tsoHost migration, we have to let Webfaction know before October 31st and the email migration worries me a lot.

Thank you very much for your help, Klynton, Sean and John.