Gmail forwarding problems

sean

Hi all, just want to let you know that we're aware that some customers who forward mail to Gmail are having certain incoming messages rejected with an error similar to this:

The MAIL FROM domain [xxxxxxx.com] has an SPF record with a hard fail policy (-all) but it fails to pass SPF checks with the ip: [XX.XX.XX.XX]. To best protect our users from spam and phishing, the message has been blocked.

When this happens, the message is not forwarded and the sender will receive a bounce message. There will be no notification of the failure for the recipient (you) unless the sender contacts them through some other channel.

What's happening here is that the sending domain's SPF rules are set to disallow mail for the domain to be sent by servers not listed in their SPF record (in this case, Opalstack's forwarding servers).

In the past, Google would usually weight messages like this with a higher spam score but still allow them to be forwarded. However, in the past few months, we've seen a steady increase of these "SPF hard fail" rejections.

To solve this we're planning to implement SRS forwarding and/or ARC signing to help Google recognize that our forwarding servers are intermediaries. Like all system changes, this takes time, planning, and careful testing to get it right. We don't have an ETA for this yet, but this one of our highest priorities and we hope to have it ready soon.

In the interim, the best workaround we've found is to disable forwarding and instead deliver to a local mailbox here at Opalstack, and then configure your Opalstack mailbox as a POP3 account in Gmail.

If the sending domain is a domain that you own, then of course you have the option of updating your domain's SPF record to include Opalstack's SPF. To do so, add the following to the content of your existing SPF record: include:spf.opalstack.com

If you have any questions or concerns about this issue then drop a comment here or email the support team.

houdinihound

There is one issue for some people with this workaround in that Gmail decides (using an algorithm based on when last there was actually mail in the POP3 account) how often to check your POP3 account - you can't specify a frequency. It can vary from 2 minutes to 65 minutes before Gmail checks your mailbox again.

You can see the last 5 retrievals in the 'view history' option in gmail settings after configuring your POP account on Gmail.

A possible workaround I've come across is to write a script that sends a dummy email to your mailbox every 10 minutes say, with a filter in Gmail to delete it as soon as it is fetched. It seems this tricks their algo into fetching every 5 minutes (half the time of the last encountered message).

tych0

houdinihound Or you can simply get into the habit of clicking the "Refresh" button on your inbox to fetch POP accounts immediately. (On this borrowed image, you can see the little circle-arrow just in the header
It's not ideal but it helps when you're expecting an email like a TFA request and don't want to wait...

sean

Last week we rolled out ARC signing for forwarded messages. This should improve deliverability for SPF hard fails.

houdinihound

tych0 True, and possibly useful on desktop/browser environments. Unfortunately it appears that if you refresh on the mobile gmail app it doesn't force the gmail server to check your POP account at that point - it just grabs what gmail already has.

sean

Unfortunately we have not seen a notable decline in SPF hard fail rejections from Google after rolling out ARC signatures and seals on our forwarding setup.

ARC is designed to solve the issue of SPF for forwarding intermediaries, for example the ARC spec page explains:

When an email sender or Internet domain owner uses email authentication to make it easier to detect fraudsters sending messages that impersonate their domain, some services like mailing lists or account forwarding may cause legitimate messages to not pass those mechanisms, and such messages might not be delivered. These services may be referred to as intermediaries because they receive a message, potentially make some changes to it, and then send it on to one or more other destinations. This kind of email traffic may be referred to as an indirect mailflow.

ARC preserves email authentication results across subsequent intermediaries (“hops”) that may modify the message, and thus would cause email authentication measures to fail to verify when that message reaches its final destination. But if an ARC chain were present and validated, a receiver who would otherwise discard the messages might choose to evaluate the ARC results and make an exception, allowing legitimate messages from these indirect mailflows to be delivered.

So in this case it seems Google is choosing to ignore the ARC results and reject some messages with the stated reason being the SPF hard failure. I say "some" because the behavior is inconsistent, for example I created a test domain with a SPF hard fail policy and then sent a message to a forwarding address on that domain. Google acknowledged the SPF failure but still delivered the message so for that one it seems they respected the ARC headers.

The next step will likely be to implement SRS forwarding which rewrites sender address on the forwarding server to allow it to pass the SPF check, but this change is more involved than ARC and will take more time to test and implement (if we decide to).

For now, our recommendation remains that if you must use Gmail to check your Opalstack mail then you use POP3 retrieval from an Opalstack mailbox instead of forwarding directly to Gmail.

houdinihound

sean Will the potential SRS implementation be in addition to ARC or to replace ARC?

aa11

sean For now, our recommendation remains that if you must use Gmail to check your Opalstack mail then you use POP3 retrieval from an Opalstack mailbox instead of forwarding directly to Gmail.

I'm considering implementing this.

I already have a mailbox setup and there were times where incoming email (to a single address) was directed to both:

A forwarding @gmail.com address
Delivered to the OS my_os_mailbox

(I did this usually for testing or when I needed to recieve emails that were being bounced because of this issue.)

So the Inbox for my_os_mailbox currently has a lot of accumulated messages.

Obviously I don't want those downloaded by Gmail during the initial POP Download sync (since they would be duplicates, but maybe not the kind that Gmail would deduplicate?).

Can you confirm that, for this case, this is a good ** Switch to Gmail POP Download procedure**, which would have the least number of lost and duplicate emails?

Procedure (draft)

Prepare the POP Account connection in Gmail (but don't enable)
Sign in to webmail.xx.opalstack.com
Empty the Inbox (via the Empty command in the Gear icon at the bottom left of the left column) or move all of the emails in the Inbox to a folder other than then Inbox.
Enable the POP account connection in Gmail (ASAP).

A few days (hours?) later remove the forward: user@my_OS_hosted_domain.com -> @gmail.com

I'm assuming Gmail will figure out the duplicates that arrive as a result of both the forward and POP account access. But maybe not?

Or is this overly complicate and really I should just use the Enable POP for mail that arrives from now on option when setting up POP download?

Via: https://www.lifewire.com/what-are-the-gmail-pop3-settings-1170853

Thanks.

sean

houdinihound it will most likely be in addition to ARC.

houdinihound

sean great!

jphillippe

Hi there! Just to let the thread know that the issue still remains.

As always, thanks for the good work!

peter

aa11 The steps in your procedure look good to me. I believe Gmail will handle the duplicates automatically because the respective emails would have the same "Message-ID" header. Choosing the Enable POP for mail that arrives from now on should also work. 🙂

aa11

aa11 Or is this overly complicate and really I should just use the Enable POP for mail that arrives from now on option when setting up POP download?

Via: https://www.lifewire.com/what-are-the-gmail-pop3-settings-1170853

This part of what I wrote above is inaccurate ❌.

That POP Download feature is so non-Gmail clients can access email, in Gmail, via POP3.

✅ The correct setting, at Gmail, is Check emails from other accounts under Accounts and Import.

It's here where you will add the OS email address, server, username, password, port (995).

More information:
https://support.google.com/mail/answer/21289

I recently set this up and everything seems to be working as expected.

I set both forward to @gmail.com and deliver to os mailbox in the OS control panel. I did this because I was concerned about the possible delivery delay (maybe up to 1h) associated with Gmail accessing the OS mailbox via POP. (In my quick testing though, that wasn't really an issue. Gmail seemed to be checking that box very quickly. But I haven't tested this much.)

This way:

Emails are forwarded to Gmail and arrive there almost immediately.
Whenever Gmail checks, via POP, the same email is downloaded, but is found to be a duplicate and doesn't seem to clutter the Gmail inbox. 👍🏻 (I'm assuming it's discarded somehow.)
In the case where the forward fails (because of the fundamental issue this thread is about), the email should be downloaded by Gmail, via POP, eventually.

That seems to be working as expected.

(Although, there haven't yet been any emails that are expected to be non-deliverable/bounced. I'll update this thread if that happens and something unexpected occurs.)

aa11

peter I believe Gmail will handle the duplicates automatically because the respective emails would have the same "Message-ID" header.

Yea, I was thinking the same, but didn't for sure if it did.

peter Choosing the Enable POP for mail that arrives from now on should also work. 🙂

Maybe I'll do a little of both.

Quick side-question about the Roundcube UI:
How do you select all messages, across all pages, in a directory? (So they can be moved to another directory)

This always trips me up. Either figuring out how to do it or validating that's what's happening because of a lack of indication in the UI.

houdinihound I did something similar and Gmail did recognize duplicates and ignored them as one would expect.

Thank you so much for the first-hand report.

houdinihound

I did something similar and Gmail did recognize duplicates and ignored them as one would expect.

sean

aa11 How do you select all messages, across all pages, in a directory?

You can select all items in a folder by choosing the "All" option from the "Select" menu:

aa11

sean You can select all items in a folder by choosing the "All" option from the "Select" menu:

Ahh!
That. Now I remember.

That menu is at the bottom in the Larry by FLINT / Büro für Gestaltung, Switzerland theme.

I wanted to attach a screenshot, but I couldn't figure out how. The one I linked to above just used Markdown (to reference an existing image hosted on another server) via the Add an Image button.

@sean It looks like the screenshot you used was uploaded directly to the forum:

https://community.opalstack.com/assets/files/2023-06-16/1686929542-453530-image.png

How did you do that?

https://duckduckgo.com/?q=flarum+upload+image&ia=web
https://next.flarum.org/extension/fof/upload

Thanks.

sean

aa11 How did you do that?

Forum uploads are for admins-only for now.

Please keep further questions on this thread on-topic, thanks.