Date   

locked Spam filter

 

Hi All,

Based on the recent spammer incident and some conversations about some evolving email standards, it's clear that I need to implement a spam filter sooner rather than later. I am unfamiliar with how Y! Group's spam filter interacts with groups; can someone clue me in? And are there any issues with their implementation (the spam filter went in after the acquisition/after I left)?

Thanks,
Mark


locked Re: Temporarily disabled group creation

 

On Thu, Sep 29, 2016 at 9:25 PM, Shal Farley <shals2nd@...> wrote:

> I am not told how they were marked (I wish I was!).

Alas.

Maybe histogram the delays from message sent to report from each service; perhaps you can infer automatic expiration from the spam folder if there's a substantial bump.

Maybe, but anecdotally, it seems that some services send out FBL messages in batches. We'll see bunches of FBL messages from more than one person all at the same time.

 
The upheaval in Yahoo Groups over DMARC happened because some benighted services (looking at you Yahoo Mail and AOL) naively obeyed the p=reject policy. I suppose because, you know, no one uses email lists... Gmail was apparently smarter out of the gate: more list messages went to spam folders as a result of senders setting p=reject, but from what I read they didn't blindly reject them.

Yeah. I have a lot of respect for Google, for many reasons, but a big one is because Brandon Long works there. He was an eGroups engineer back in the day, one of the best engineers I've ever worked with. After the Yahoo acquisition, he ended up at Google, where he built Google Groups. Then Brandon went on to Gmail, where (I think) he's been ever since. A good guy.

Anyways, based on conversations on the ARC mailing list, it's clear that I need to implement a spam filter. I'll start a new thread to talk about that.

Thanks,
Mark


locked Re: Problem with content outside of the 'charset' #bug

 

I've noticed that since this fix (removing HTML from message summaries), Groups.io now displays the same "div..blockquote.." etc. junk that Yahoo started doing about a year ago for people posting from iPads (at least).

This is what's displayed in the message list for a particular message:

blockquote, div.yahoo_quoted { margin-left: 0 !important; border-left:1px #715FFA solid !important; padding-left:1ex !important; ba..
Whereas this is how the actual message reads:

Ok, I will do all I can to get those labs on here asap.  Thanks again!
Can something be done about this? It only started recently, and it seems to have started when the HTML started being stripped.

--
J

Messages are the sole opinion of the author. 

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


locked "xyz changed his name to abc via email" tons of log entries - meaning?

 

Mark,

Perhaps this has to do with the multiple-profile and naming issues you're working on (e.g., changed member name is not displayed after a moderator changes it). I see frequent log entries of the form "xyz changed his/her name to abc via email" in all my groups. Yet there is never a name change visible on the member's page. (For example, "Peter Piper changed his name to Peter via email" and the member's name still shows as Peter Piper.") I think that 100% of the time a member joins the group, the first thing that I see as a log entry is a message of this type, EVEN if the member shows no name at all on their page.

This has gone on for at least several months and I'm wondering if you can clarify it, and perhaps the status of the names situation generally.
--
J

Messages are the sole opinion of the author. 

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


locked Re: Temporarily disabled group creation

 

Mark,

What I look for is the one-click unsubscribe link in the
message footer. From that I can figure out the recipient.
Ah, clever, and useful.

I am not told how they were marked (I wish I was!).
Alas.

Maybe histogram the delays from message sent to report from each service; perhaps you can infer automatic expiration from the spam folder if there's a substantial bump.

Yeah. I mean spoofed emails are a fact of life on the Internet and I
think (hope) all the major providers recognize the spoofing. So I
don't think it'll ding our email sending reputation.
Not any more than breaking DMARC for the non-rewritten froms does, probably.

The upheaval in Yahoo Groups over DMARC happened because some benighted services (looking at you Yahoo Mail and AOL) naively obeyed the p=reject policy. I suppose because, you know, no one uses email lists... Gmail was apparently smarter out of the gate: more list messages went to spam folders as a result of senders setting p=reject, but from what I read they didn't blindly reject them.

But hey, the other day we had our first million email day!
Awesome! Congratulations.

Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum


locked Re: Temporarily disabled group creation

 

On Thu, Sep 29, 2016 at 6:17 PM, Shal Farley <shals2nd@...> wrote:


I wonder if ARC defends against this kind of replay. That is, when you put an ARC-seal on an outbound messages, is there something that makes this copy unique (a time stamp, serial number, or anything else that would invalidate a copied ARC-seal. If not that may be a weakness.

That's an excellent question and thanks for thinking of it. I just posted it to the ARC group, so we'll see.
 

Oh, interesting. I was wondering how much detail you get in the FBL reports. Specifically I was wondering if you were told the user and/or the Message-ID's of "marked" messages. And how they got marked (explict user action versus automation). But that was concerning some discussion in GMF@ about other issues with the FBL reports.

The FBL reports differ a little bit by domain. In general, the email providers try to obscure the email address of the person who received the email. What I look for is the one-click unsubscribe link in the message footer. From that I can figure out the recipient. Message-IDs are not obscured, and (I think for all of them), I get the entire message sent to me. I am not told how they were marked (I wish I was!). 

 
But, silly Yahoo, shouldn't the reports be flung back at the 5321.MailFrom? Sending it back to the From strikes me as akin to backscatter, though it was useful to you in this case. Or is it the (copied) Return-Path they're using? Either way...

Yeah, not sure. But still, pretty stupid of them.

 
> But I'm not sure country blocking isn't a great long term solution
> for this.

I'm reasonably sure it isn't.

For the moment I've blocked Morocco, because I need to actually get some work done. Having that guy around was distracting.

 
Well, I think you have some head-scratching yet to do on this problem.

Yeah. I mean spoofed emails are a fact of life on the Internet and I think (hope) all the major providers recognize the spoofing. So I don't think it'll ding our email sending reputation. It sucks though because I get angry support emails from people demanding to be removed from groups they aren't even on.

We're still small fry in the grand scheme of things, which is nice in a way because I'm able to add defenses over time. But makes us still open to being blocked if things go bad. But hey, the other day we had our first million email day!


Mark


locked Re: Temporarily disabled group creation

 

Mark,

I'm watching them try to register new email addresses in real time
and being denied. They are persistent.
That's the way of ants and other automatons.

What they're doing is creating a group, doing a post, grabbing the
resulting email, and sending that out to people themselves. ... And
our SPF record doesn't apply because they're using a different
envelope domain (I did just change our SPF records from softfail to
fail, but if I understand what's happening, that won't help here).
No, but my understanding is that the misalignment of the 5322.From with the 5321.MailFrom will cause DMARC to fail anyway, for those receiving services that check it. Of course, so do the messages legitimately through Groups.io when you don't rewrite the From.

So if I understand everything correctly, the only way to 'fix' this
would be to set DMARC p=reject and re-write *all* From lines in
emails. I'd obviously rather not do that.
I'd rather you not do that too. The only advantage I can see is that all outbound messages could pass DMARC, instead of only those from the services you rewrite.

I wonder if ARC defends against this kind of replay. That is, when you put an ARC-seal on an outbound messages, is there something that makes this copy unique (a time stamp, serial number, or anything else that would invalidate a copied ARC-seal. If not that may be a weakness.

The reason I was able to see one of these emails is that Yahoo has
sent some FBL reports to us from these forged emails (people who
received these messages marked them as spam, and Yahoo thought they
originated from us).
Oh, interesting. I was wondering how much detail you get in the FBL reports. Specifically I was wondering if you were told the user and/or the Message-ID's of "marked" messages. And how they got marked (explict user action versus automation). But that was concerning some discussion in GMF@ about other issues with the FBL reports.

But, silly Yahoo, shouldn't the reports be flung back at the 5321.MailFrom? Sending it back to the From strikes me as akin to backscatter, though it was useful to you in this case. Or is it the (copied) Return-Path they're using? Either way...

But I'm not sure country blocking isn't a great long term solution
for this.
I'm reasonably sure it isn't.

Well, I think you have some head-scratching yet to do on this problem.

--
Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum


locked Re: Temporarily disabled group creation

 

Oh, and I know the spammer is reading beta@. 

Exciting!

Mark

On Thu, Sep 29, 2016 at 3:56 PM, Mark Fletcher <markf@corp.groups.io> wrote:
On Thu, Sep 29, 2016 at 1:29 PM, Shal Farley <shals2nd@...> wrote:
Mark,

> Most of the groups are being created using disposable email
> addresses. I am in the process of blocking those email providers.

Maybe you need a throttle (and internal alert) on the rate of group creation from unknown email providers. Then if you get an alert for a rash of creations from an unknown you can decide whether to white or black list the provider, or leave it gray ("unknown") pending further information. Sort of a poor-man's reputation system.


I think at least for now just checking against a comprehensive list of disposable email providers seems to be helping a lot. I'm watching them try to register new email addresses in real time and being denied. They are persistent.


> Also, most likely related, someone is forging spam emails from
> Groups.io. So I may need to look into some changes to mitigate that.
> I'll keep you posted.

If they're forging the groups.io domain in the From field then maybe DMARC p=reject is your friend.

What they're doing is creating a group, doing a post, grabbing the resulting email, and sending that out to people themselves. So the DKIM sig still authenticates. And our SPF record doesn't apply because they're using a different envelope domain (I did just change our SPF records from softfail to fail, but if I understand what's happening, that won't help here).

So if I understand everything correctly, the only way to 'fix' this would be to set DMARC p=reject and re-write *all* From lines in emails. I'd obviously rather not do that.

The reason I was able to see one of these emails is that Yahoo has sent some FBL reports to us from these forged emails (people who received these messages marked them as spam, and Yahoo thought they originated from us). 

I've also thought about blocking all IP addresses from Morocco. We currently block all IPs from Afghanistan, because of an earlier incident. But I'm not sure country blocking isn't a great long term solution for this.

(Note for the majority of people on beta@ who probably don't understand most of the above: The email forgery isn't a big deal right now and won't affect your groups).

Thanks,
Mark



locked Re: Temporarily disabled group creation

 

On Thu, Sep 29, 2016 at 1:29 PM, Shal Farley <shals2nd@...> wrote:
Mark,

> Most of the groups are being created using disposable email
> addresses. I am in the process of blocking those email providers.

Maybe you need a throttle (and internal alert) on the rate of group creation from unknown email providers. Then if you get an alert for a rash of creations from an unknown you can decide whether to white or black list the provider, or leave it gray ("unknown") pending further information. Sort of a poor-man's reputation system.


I think at least for now just checking against a comprehensive list of disposable email providers seems to be helping a lot. I'm watching them try to register new email addresses in real time and being denied. They are persistent.


> Also, most likely related, someone is forging spam emails from
> Groups.io. So I may need to look into some changes to mitigate that.
> I'll keep you posted.

If they're forging the groups.io domain in the From field then maybe DMARC p=reject is your friend.

What they're doing is creating a group, doing a post, grabbing the resulting email, and sending that out to people themselves. So the DKIM sig still authenticates. And our SPF record doesn't apply because they're using a different envelope domain (I did just change our SPF records from softfail to fail, but if I understand what's happening, that won't help here).

So if I understand everything correctly, the only way to 'fix' this would be to set DMARC p=reject and re-write *all* From lines in emails. I'd obviously rather not do that.

The reason I was able to see one of these emails is that Yahoo has sent some FBL reports to us from these forged emails (people who received these messages marked them as spam, and Yahoo thought they originated from us). 

I've also thought about blocking all IP addresses from Morocco. We currently block all IPs from Afghanistan, because of an earlier incident. But I'm not sure country blocking isn't a great long term solution for this.

(Note for the majority of people on beta@ who probably don't understand most of the above: The email forgery isn't a big deal right now and won't affect your groups).

Thanks,
Mark


locked Re: Temporarily disabled group creation

 

Mark,

Most of the groups are being created using disposable email
addresses. I am in the process of blocking those email providers.
Maybe you need a throttle (and internal alert) on the rate of group creation from unknown email providers. Then if you get an alert for a rash of creations from an unknown you can decide whether to white or black list the provider, or leave it gray ("unknown") pending further information. Sort of a poor-man's reputation system.

Also, most likely related, someone is forging spam emails from
Groups.io. So I may need to look into some changes to mitigate that.
I'll keep you posted.
If they're forging the groups.io domain in the From field then maybe DMARC p=reject is your friend.

If they're forging a "subscriber's" domain, as if it is a message that passed through a group, I'm not sure. The message would already fail DMARC, but that puts the disposition policy in the hands of the domain that was forged.

Shal

--
Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum


locked Re: Temporarily disabled group creation

 

On Thu, Sep 29, 2016 at 7:00 AM, J_Catlady <j.olivia.catlady@...> wrote:
On Wed, Sep 28, 2016 at 11:00 pm, Christopher Hallsworth wrote:
Good idea, never thought of that one.

That's why Mark makes the big bucks. ;) 


Heh. :-)

The system is working, in that I've deleted 128 groups just so far this morning. Seems to be a determined spammer from Morocco. 

Most of the groups are being created using disposable email addresses. I am in the process of blocking those email providers.

Also, most likely related, someone is forging spam emails from Groups.io. So I may need to look into some changes to mitigate that. I'll keep you posted.

Thanks,
Mark


locked Re: Problem with content outside of the 'charset' #bug

 

Hi Mark,

Thanks very much for getting back to me! Yes, I'm talking about the summary.

Your code works as designed: garbage in garbage out. However, some groups, like mine, have a need to support multiple languages. The use case is for people communicating in several languages, e.g., Japanese students studying Russian. Some mail clients, like outlook.live.com are uni-lingual.

Tests on my test group indicate that the plain text portion contains characters that, when displayed as utf-8 (the charset that I used to send the message), show up correctly in the body of the message but not in the summary. That indicates to me that you are doing something different to the characters when you display the summary than when you display the message contents. In this case, there is nothing wrong with the characters themselves. The only garbage is that their charset has been wrongly declared

Thanks again!

David.


locked Re: Temporarily disabled group creation

 

On Wed, Sep 28, 2016 at 11:00 pm, Christopher Hallsworth wrote:
Good idea, never thought of that one.

That's why Mark makes the big bucks. ;) 
--
J

Messages are the sole opinion of the author. 

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


locked Re: Temporarily disabled group creation

christopher hallsworth <challsworth2@...>
 

Mark

Good idea, never thought of that one.

On 28 Sep 2016, at 22:35, Mark Fletcher <markf@corp.groups.io> wrote:

Hi All,

Group creation has been turned back on. Now, when a (listed) group is created, it won't actually appear in the directory or search until I approve it. This is mentioned on the page displayed after the group is created. Also, when a group's name or description is changed, I'm notified as well. In those instances, the changes are made instantly to the directory and search, like before. But since I'm notified, I can retroactively delete the group if it was a spam group in hiding, so to speak.

Thanks,
Mark

On Wed, Sep 28, 2016 at 1:04 PM, HR Tech via Groups.io <m.conway11=yahoo.com@groups.io> wrote:
This is likely a bad idea but I'll share anyway. What about a credit card transaction like other sites use to verify identity? Does nextdoor do that? Or paypal when you set up an account and link it to a bank account? Do they take $1 and then reimburse it...? It's been a while since i needed to set one up.

Not ideal, and no idea if it would help.

Maria





locked Re: Problem with content outside of the 'charset' #bug

 

Hi David,

On Wed, Sep 28, 2016 at 5:33 PM, David Andrews <jdabud@...> wrote:

The good news is that the content of the message is displayed much better than it would have been by Yahoo!Groups. Thank you! But the special characters don't show up correctly in the message listing.

I assume you're referring to the topic summary, the line displayed in the Topics view under each topic subject? For that line, we want plain text, because any sort of HTML formatting would screw up the display. So, we first look for a plain text part of the email, which most emails have. If we find one, we just pull out the summary from that. If there is no plain text part, we use the HTML part and strip it of formatting before we pull the summary out.

With your email, it does have a plain text part, but if you look at it, live.com screwed up the characters, and subbed in a bunch of question marks. We dutifully pull those out for the summary line. So, not our fault. But it does have me wondering if we should always just use the HTML part of the message if there is one.

Thanks,
Mark


locked Re: Problem with content outside of the 'charset' #bug

 

On Wed, Sep 28, 2016 at 05:33 pm, David Andrews wrote:
the special characters don't show up correctly in the message listing

Coincidentally, I noticed the same behavior a day or two ago, where the message snippet in the message list didn't preserve formatting. After reporting it to Mark, I found out that HTML is stripped from message snippets, for some reason having to do with spam filters. 
--
J

Messages are the sole opinion of the author. 

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


locked Problem with content outside of the 'charset' #bug

 


道德經: 道可道,非常道。

Ñ ñ Ā ā Ī ī Ŋ ŋ Ū ū Ḍ ḍ Ḷ ḷ Ṁ ṁ Ṃ ṃ Ṅ ṅ Ṇ ṇ Ṭ ṭ


The characters above are the first few characters of the Dao De Jing (Tao Te Ching) followed by characters used for the Romanization of classical Indian languages. The message was sent by outlook.live.com Webmail. If it works as in my other tests, will be sent out in the inappropriate charset, iso-8859-1.


The good news is that the content of the message is displayed much better than it would have been by Yahoo!Groups. Thank you! But the special characters don't show up correctly in the message listing.


Thanks,


David.


locked Re: Temporarily disabled group creation

Steph Mathews <smathews@...>
 

Okay, thank you Mark, I appreciate it.  Steph

Sent: Wednesday, September 28, 2016 4:51 PM
Subject: Re: [beta] Temporarily disabled group creation

Hi Steph,

On Wed, Sep 28, 2016 at 2:47 PM, Steph Mathews <smathews@...> wrote:
Hi Mark,
I have a question about this.
If our group was created and has been listed before hand.  Is there anything we the group owners need to do to get our group re-approved?  Just wondering.  Steph


Sorry, should have made this clear. No, nothing needs to be done for existing groups. 

Thanks,
Mark 


locked Re: Temporarily disabled group creation

 

Hi Steph,

On Wed, Sep 28, 2016 at 2:47 PM, Steph Mathews <smathews@...> wrote:
Hi Mark,
I have a question about this.
If our group was created and has been listed before hand.  Is there anything we the group owners need to do to get our group re-approved?  Just wondering.  Steph


Sorry, should have made this clear. No, nothing needs to be done for existing groups. 

Thanks,
Mark 


locked Re: Temporarily disabled group creation

Steph Mathews <smathews@...>
 

Hi Mark,
I have a question about this.
If our group was created and has been listed before hand.  Is there anything we the group owners need to do to get our group re-approved?  Just wondering.  Steph

Sent: Wednesday, September 28, 2016 4:35 PM
Subject: Re: [beta] Temporarily disabled group creation

Hi All,

Group creation has been turned back on. Now, when a (listed) group is created, it won't actually appear in the directory or search until I approve it. This is mentioned on the page displayed after the group is created. Also, when a group's name or description is changed, I'm notified as well. In those instances, the changes are made instantly to the directory and search, like before. But since I'm notified, I can retroactively delete the group if it was a spam group in hiding, so to speak.

Thanks,
Mark

On Wed, Sep 28, 2016 at 1:04 PM, HR Tech via Groups.io <m.conway11@...> wrote:

This is likely a bad idea but I'll share anyway. What about a credit card transaction like other sites use to verify identity? Does nextdoor do that? Or paypal when you set up an account and link it to a bank account? Do they take $1 and then reimburse it...? It's been a while since i needed to set one up.

Not ideal, and no idea if it would help.

Maria


18701 - 18720 of 29708