Topics

moderated From address re-writing


 

Hi All,


There seem to be four cases where we need to re-write the From address (and possibly Reply-To/CC) in a message:

1) When DMARC says we have to, we re-write the From line in messages *from* that domain.
2) When someone is on a @gmail.com domain, we need to re-write the From line in messages that they originate that we send back to them.
3) When someone is using Gmail on a different domain, we need to re-write the From line in messages that they originate that we send back to them (same as #2 except we can't use a simple @gmail.com check).
4) The From line in all messages sent *to* (many/most) Exchange servers.

We currently do 1 and 2. From the other topic, here are the proposed new settings:

- Per user setting to have all From lines re-written in messages to themselves.
- Per group setting to have all From lines re-written in messages sent within the group.

Shal proposed the group setting be in the default sub settings page. But I don't know how that would work. None of those existing settings have a corresponding user setting, which makes it easy to just use them when creating a subscription. If this was a default sub setting, how would we know when to use the default sub setting vs the user's setting? 

I also propose a third, system wide, maintained by me list of domains that are known to run Exchange/corresponding to #4 above. This list of domains would over-rule the per-user and per-group setting in forcing From line re-writing of messages sent to those domains. I'd ask people with knowledge of these domains to send them to me.

Thoughts?

Thanks,
Mark


 

I don't know any of the general answers, but I would like to reiterate that currently, the bcc on a Send Message to member is not being handled properly in gmail and still (or rather, again, since after the "From" rewriting for the other messages sent back to the same member) results in the spam warning.


On Fri, Feb 22, 2019 at 1:44 PM Mark Fletcher <markf@corp.groups.io> wrote:
Hi All,


There seem to be four cases where we need to re-write the From address (and possibly Reply-To/CC) in a message:

1) When DMARC says we have to, we re-write the From line in messages *from* that domain.
2) When someone is on a @gmail.com domain, we need to re-write the From line in messages that they originate that we send back to them.
3) When someone is using Gmail on a different domain, we need to re-write the From line in messages that they originate that we send back to them (same as #2 except we can't use a simple @gmail.com check).
4) The From line in all messages sent *to* (many/most) Exchange servers.

We currently do 1 and 2. From the other topic, here are the proposed new settings:

- Per user setting to have all From lines re-written in messages to themselves.
- Per group setting to have all From lines re-written in messages sent within the group.

Shal proposed the group setting be in the default sub settings page. But I don't know how that would work. None of those existing settings have a corresponding user setting, which makes it easy to just use them when creating a subscription. If this was a default sub setting, how would we know when to use the default sub setting vs the user's setting? 

I also propose a third, system wide, maintained by me list of domains that are known to run Exchange/corresponding to #4 above. This list of domains would over-rule the per-user and per-group setting in forcing From line re-writing of messages sent to those domains. I'd ask people with knowledge of these domains to send them to me.

Thoughts?

Thanks,
Mark


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


 

Me having a Yahoo address, my From line has been rewritten from the beginning, and I'm happy with that. If I suddenly switched domains to one that didn't need rewriting, but the option were there to enable it anyway, I would likely take that option. I like that recipients can easily distinguish using the From line if I posted or am directly mailing them.

Tight groups of members, which are likely to all be on one particular domain, might prefer it the other way, where the From line reflects the member's email address. If I have a team of 20 from whatever.org, and I want to auto-filter all mail from whatever.org in some way, having to worry about a groups.io domain complicates things. It's not something that couldn't be worked around, just more complicated. I don't know if anyone actually does this; I just like to think up situations where something would be harder.

JohnF


 

Mark,

Shal proposed the group setting be in the default sub settings page. But I don't know how that would work. None of those existing settings have a corresponding user setting, which makes it easy to just use them when creating a subscription.

Not true.

I proposed that it work like the Time Zone, Time Display, and Date Display controls on the Default Sub Settings. Those correspond to Account (not Subscription) controls.

If this was a default sub setting, how would we know when to use the default sub setting vs the user's setting? 

The Account control always takes precedence, the Default Sub Setting only applies when a new account is created.

Now we could, and probably should, consider splitting these Account-related controls into a separate "Default Acct. Settings" tab in the group's settings page, for clarity on what they apply to and how. Or at least put section boxes around the defaults for new Subscriptions versus those for new Accounts.

I also propose a third, system wide, maintained by me list of domains that are known to run Exchange/corresponding to #4 above.

I think my idea would handle your #2, #3, and #4.

In the case of #2 I'd have the "gmail.com" test, when true, take precedence over the group's Def Sub Setting when creating a new account. Likewise in the case of #4 when a domain matches your internal list could take precedence.

Granted that does mean that the group's Default Sub Setting for this becomes more of a hint than a control. If this seems too mysterious, you could have a sentence under the control (in the user's Account page) that says "This feature turned on because your email address is at gmail.com" or "... is handled by an Exchange server", or "... group XYZ chose that default for you".

In the case of #3 would it be useful for you to keep a system-wide list of domains "known" to be handled by gmail.com? I don't know if those are all essentially "vanity" domains used by one or a very few accounts, or if some of them are larger organizations that outsource their email service and many new accounts would benefit from you having noticing them. Nor am I entirely sure how you would "notice" them other than something creepily Amazon-like ("you have this default because most users of your email domain prefer this setting").

This list of domains would over-rule the per-user and per-group setting in forcing From line re-writing of messages sent to those domains. I'd ask people with knowledge of these domains to send them to me.

Does that mean in those cases you would disable the control in the user's Account page?  That might be appropriate.

Or maybe it would be better to let the user experiment with the consequences of un-setting it. Maybe users who un-set it and keep it that way represent a valuable signal that the list might be in error, or needing some form of refinement.

Shal


 

John,


Me having a Yahoo address, my From line has been rewritten from the beginning, and I'm happy with that. 

You're talking about case #1 in Mark's message. I don't think he's proposing any changes to that case.

If I suddenly switched domains to one that didn't need rewriting, but the option were there to enable it anyway, I would likely take that option.  I like that recipients can easily distinguish using the From line if I posted or am directly mailing them.

I don't think you need that option to get that effect. Depending on the other members' email interfaces of course. Many of them (including Gmail at least) mark list messages as "via groups.io" even when the From address is not re-written.

Tight groups of members, which are likely to all be on one particular domain, might prefer it the other way, where the From line reflects the member's email address.  If I have a team of 20 from whatever.org, and I want to auto-filter all mail from whatever.org in some way, having to worry about a groups.io domain complicates things.

What happens in that case depends on where "whatever.org" buys its email service. If the email service they hired (or if they run their own and it) publishes the DMARC reject policy in its DNS entries then the From address rewriting is pretty much required on any list message from a user of that domain. That would be an all-or-none situation for the team from whatever.org.
Usually the better filter choice is to use the group's subject tag. That could catch off-list messages from team members if the message started as a reply to a list message, but it is a judgement call whether the user would have wanted that to happen.

Best is to filter based on the List-Id header field, but sadly I suspect that relatively few people use an interface that can do that, or would know how to do that if they could.

Shal


 

I wrote:


I think my idea would handle your #2, #3, and #4.

Or not.

#4 applies to all messages to that account, not just the user's own messages. If exposed as a user Account control it would probably need to be a separate control from the one for one's own messages. Likewise in the group's Default Sub. Settings.

So, yeah, it is a little less slick than I was thinking.

Shal
P.S. My apologies to you Windows nerds out there for accidentally speaking the devil's name. You know who you are because you heard it go "clang" in your mind when you read it. The rest of you are safe.


KWKloeber
 

Thoughts?

Thanks,
Mark

Yeah.  Thought for the day (or early morning as it were.) Remember when we were lied to about all this “going paperless” stuff was going to make our lives so much simpler and easier”?
How’s that workin out for Mark?
😳🥺😱😉😜🤪🥺🤯
Ken

Sent from my phone


 

Hi All,

I had a thought (it happens occasionally, and yeah, I'm surprised too)...

On Fri, Feb 22, 2019 at 1:43 PM Mark Fletcher <markf@corp.groups.io> wrote:

3) When someone is using Gmail on a different domain, we need to re-write the From line in messages that they originate that we send back to them (same as #2 except we can't use a simple @gmail.com check).
4) The From line in all messages sent *to* (many/most) Exchange servers.


Both 3 and 4 are at the domain level. Why bother with new group/user settings, when two lists of domains would do a potentially better job? What if I add a section to the help, with an area for someone to input their domain to be added to one of the two lists? Seems like that'd be less complicated/confusing than new settings and would potentially help more people.

Thanks,
Mark 


Ken Schweizer
 

Hi Mark,

 

Would that be a global setting and one which anyone can change? Or is it a setting you would "moderate"?

 

Ken

 

"And if any man shall take away from the words of the book of this prophecy, God shall take away his part out of the book of life, and out of the holy city, and from the things which are written in this book." God

 

From: main@beta.groups.io [mailto:main@beta.groups.io] On Behalf Of Mark Fletcher
Sent: Saturday, February 23, 2019 11:38 PM
To: beta@groups.io
Subject: Re: [beta] From address re-writing

 

Hi All,

 

I had a thought (it happens occasionally, and yeah, I'm surprised too)...

 

On Fri, Feb 22, 2019 at 1:43 PM Mark Fletcher <markf@corp.groups.io> wrote:

 

3) When someone is using Gmail on a different domain, we need to re-write the From line in messages that they originate that we send back to them (same as #2 except we can't use a simple @gmail.com check).

4) The From line in all messages sent *to* (many/most) Exchange servers.

 

 

Both 3 and 4 are at the domain level. Why bother with new group/user settings, when two lists of domains would do a potentially better job? What if I add a section to the help, with an area for someone to input their domain to be added to one of the two lists? Seems like that'd be less complicated/confusing than new settings and would potentially help more people.

 

Thanks,

Mark 


 

Mark,

Both 3 and 4 are at the domain level. Why bother with new group/user
settings, when two lists of domains would do a potentially better job?
I believe the original impetus for #4 was from a group owner who wanted to be able to set that function for his group members because they were predominantly exchange users. I suggested making it an account setting (with group default) because those members would then benefit in their other subscriptions.

That said, managed domain lists are probably the better solution.

It would be nice to "surface" that fact to the affected users somehow (perhaps a text note in their Account pages) so that there is a ready explanation for the "why are addresses munged at my xyz address but not my abc address?". Group admins (and GMF) could then tell them to go read that place.

What if I add a section to the help, with an area for someone to input
their domain to be added to one of the two lists?
Crowd-sourcing those designations seems like a good idea, especially for #3, but I do share the concern that Ken seems to be alluding to: naive or malicious users could cause havoc by adding inappropriate domains to the list.

If the help form simply nominates domains, and you have to vet them, that's safer. A bit more work for you, but maybe less work overall than having to cull the list from support requests or other sources.

Ordinarily the help form doesn't require being logged in - and that's a good thing. But maybe if the user happens to be logged in, and if their domain isn't yet on your list of known domains, you could display a section that explains about the possible need for munging the address and provides a checkbox for the user to nominate their domain.

I'd take the nomination/confirmation in two steps. That is, begin munging addresses to that user immediately upon nomination. That way the user can observe the effect for themselves, and possibly un-check the nomination. On your own schedule you can review the nominations and determine whether any of them should apply site-wide.

Seems like that'd be less complicated/confusing than new settings and
would potentially help more people.
Agreed. Most group owners shouldn't really need to know all about the quirks of their members' email services.

Even the members themselves shouldn't - at least for major mailbox providers. Users with custom domains may be considered an outlier in terms of their knowledge of their email service.

Shal


 

Hi All,

I've just pushed out the following changes:

- I have two internal lists of domains: a list of domains hosted by Gmail, and a list of domains hosted by Exchange.
- For domains hosted by Gmail, we now re-rewrite the From lines of messages we send back to the original sender.
- For domains hosted by Exchange, we now re-write the From lines of all messages sent to those domains.
- In the Accounts Prefs page, we now display one of the following messages to applicable members:

        Your email is hosted by Gmail, so we will re-write the From line for the copies of your group messages that we send back to you.
        Your email is hosted by Exchange, so we will re-write the From line for all group messages sent to you.

- There are two new help sections in the FAQ for people who are seeing spoofing warnings or are not receiving email to their Exchange hosted accounts.

At least for now, additions to the domain lists are handled manually by me.

Please let me know if you have any questions or wording improvement suggestions.

Thanks,
Mark


 

On Tue, Feb 26, 2019 at 09:28 AM, Mark Fletcher wrote:
- For domains hosted by Gmail, we now re-rewrite the From lines of messages we send back to the original sender.
Mark,

As I mentioned before, paradoxically, after you started the From address rewriting for gmail, the "bcc me" on member messages using "send message" started displaying the spam-warning behavior. This did not happen before you started the From rewriting - the behavior seemed to hop from group messages only to "bcc me" messages only. So my question, especially since you mention "group messages" in the documentation below, is whether whatever you did this time will fix the "bcc me" spam warning problem as well.

Thanks.
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


 

On Tue, Feb 26, 2019 at 9:47 AM J_Catlady <j.olivia.catlady@...> wrote:

As I mentioned before, paradoxically, after you started the From address rewriting for gmail, the "bcc me" on member messages using "send message" started displaying the spam-warning behavior. This did not happen before you started the From rewriting - the behavior seemed to hop from group messages only to "bcc me" messages only. So my question, especially since you mention "group messages" in the documentation below, is whether whatever you did this time will fix the "bcc me" spam warning problem as well.

We should now be re-writing the From lines for post to sub messages.

Thanks,
Mark 


 

Mark,


- In the Accounts Prefs page, we now display one of the following messages to applicable members:

I think it should be on the Accounts Login page, just under the Email field and Change Email button. It isn't a Preference, it is a consequence of their chosen email domain. I think it deserves that much prominence.

Where it is feels a little weird, since it is only tangentially related to the option to have Message-ID's re-written. Unless you're planning for there to be a related control; in which case it needs a label different from "My Posts", at least for the Exchange message.

 
Please let me know if you have any questions or wording improvement suggestions.

I'm not sure how to avoid writing a novel there, but the lack of any explanation of motive seems wrong. Maybe a "Learn more" link to the related FAQ on the help page?

Shal


 

On Tue, Feb 26, 2019 at 02:28 PM, Mark Fletcher wrote:
We should now be re-writing the From lines for post to sub messages.
Great! Thanks.
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


 

On Tue, Feb 26, 2019 at 3:48 PM Shal Farley <shals2nd@...> wrote:

- In the Accounts Prefs page, we now display one of the following messages to applicable members:

I think it should be on the Accounts Login page, just under the Email field and Change Email button. It isn't a Preference, it is a consequence of their chosen email domain. I think it deserves that much prominence.

Done.
 

I'm not sure how to avoid writing a novel there, but the lack of any explanation of motive seems wrong. Maybe a "Learn more" link to the related FAQ on the help page?

Good idea. Done.

Thanks,
Mark 


 

On Tue, Feb 26, 2019 at 02:28 PM, Mark Fletcher wrote:
We should now be re-writing the From lines for post to sub messages.
Mark,

Bcc's on moderator "send message to member" emails in gmail are now fine. However, bcc's on PMs have the spam warning. Was the change supposed to fix that? If not, I can live with it, but a fix would be great!
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


 

On Thu, Feb 28, 2019 at 8:10 AM J_Catlady <j.olivia.catlady@...> wrote:

Bcc's on moderator "send message to member" emails in gmail are now fine. However, bcc's on PMs have the spam warning. Was the change supposed to fix that? If not, I can live with it, but a fix would be great!
 
If you're referring to BCCing yourself on a private reply to a message, that should be fixed now.

Thanks,
Mark 


 

Mark, yes, that’s what I was referring to. Thanks!


On Feb 28, 2019, at 10:52 AM, Mark Fletcher <markf@corp.groups.io> wrote:

On Thu, Feb 28, 2019 at 8:10 AM J_Catlady <j.olivia.catlady@...> wrote:

Bcc's on moderator "send message to member" emails in gmail are now fine. However, bcc's on PMs have the spam warning. Was the change supposed to fix that? If not, I can live with it, but a fix would be great!
 
If you're referring to BCCing yourself on a private reply to a message, that should be fixed now.

Thanks,
Mark 

--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu