Massive numbers of email dupes going back to 12/14/2020

hedronist

Hi --

tl;dr: a new-to-me thing happened causing receipt of hundreds of dupe messages going back over a month.

My wife asked me to look at a bunch of email that had just arrived (this was approx. 19:59 PDT 1/31/2021). It was all dupes -- hundreds of them going back to 12/15/2020. Note: I checked her Inbox using webmail and the oldest message that shows is 1/17/2021. This time disparity makes things even weirder.

I checked my own email -- same thing, more than 100 dupes. My earliest one was from 12/14/2020, but delivered at 19:50PDT 1/31/2021.

She is running Thunderbird 78.7.0, I'm on TB 78.6.0, both on latest Win10.

Is anyone else seeing something like this?

sean

hedronist do you see this in the webmail system at https://webmail.opalstacked.com/ or only in your mail client?

hedronist

It only is seen in our POP3 clients.

Note: I'm writing this while my coffee is brewing, so ...

Here's a WAG about what might have happened: Back a few weeks (6? 8?) OS had a problem with their mail servers. Basically they went on/off line over a period of a few hours. My guess is that you fortified the system by doing some sort of hot-standby and/or multiple masters working in round-robin mode (or whatever).

My guess is that whatever token/timestamp is used by POP3 clients to tell the server what mail they already have received got lost/corrupted/duplicated/something. When we connect to your server(s) we never know which one we're going to get. So if this tiny, but critical, piece of user state is not shared (I know, "shared nothing" is the Holy Grail) then the "new" server says, "Here! Look at all of the back mail I have for you!" And it's off to Dupe City we go.

Anyway, that's my Pre-Columbian Wild Ass Guess of the morning. :-)

sean

hedronist we're not doing anything that fancy.

Last night we migrated the US IMAP service to new hardware like so:

Restored the most recent backup of the US IMAP server to the new machine.
Pointed the US IMAP hostname at the new server.
Ran a final rsync to catch up with mail that had arrived on the old server in the time since the backup that we used in step 1.

I'm not sure why this would have triggered a re-download of POP mail that had already been downloaded but we'll look into it.

hedronist

Now, slightly caffeinated, I can remember to add that in comparing raw message source between the originals and the dupes, in all cases there were only 2 lines different, lines 1 & 3 from the top. Times are PDT (UTC -0800).

Original:
From - Tue Dec 15 11:38:30 2020
X-UIDL: 000002c35f9dc849

Dupe:
From - Sun Jan 31 19:50:01 2021
X-UIDL: 0000061c5f9dc849

The "From -" is added, I believe, by TB. I don't know who is generating the X-UIDL, but it appears to be a breadcrumb (if not a smoking gun). ref: https://emailheaders.net/thunderbird.html. It's about 2/3 of the way down that page.

The original timestamps for all the dupes occurred in the 12/14/2020 - 1/31/2021 time frame, and were (re-)delivered between ₁₉:50PDT and 20:10PDT 1/31/2021.

That's all I've got for now. If this is simply a one-off event from the server switch, then we'll just have to spend some time dupe hunting.

Peter

Ps. If you like I can send you a zip of a few original/dupe pairs of raw sources, but they all exhibit exactly the behavior shown above.

sean

hedronist thanks for the info, no need to send anything else. Pretty sure the UIDL difference is the issue. We'll try to correct it on our end if it's not too late.

mick-p1982

I have something like this too. I just started trying to use my email here as my regular mail address, and the 270 emails I deleted yesterday are now back, and some I wrote/sent yesterday are duplicated in my Sent folder as of today. (Edited: I'm just using your Roundcube webmail client.)