Date   

locked Re: tweak top-posting algorithm to allow explicit, intentional, changed, partial quotes #suggestion

 

J,

I think that if possible, the top-posting algorithm should be tweaked
so that the quote is not put into ellipses if either
...
I'd add "(d) is not contained in the In-Reply-To message."

Or perhaps replace (a) through (d) with just: "Does not contain the entirety of the In-Reply-To message." That would be pretty strict, in the sense of covering up the quote only in a true top-post (where the respondent didn't trim the quote at all). My argument in favor of this would be that if the respondent did do some trimming, then he/she probably relied on the partial quote being seen for context.

IIRC I had earlier suggested that the algorithms for hiding or eliding quotes be based (where possible) on comparison to the quoted message. In most cases the quoted message will be available to the algorithm (in the group's Messages).

There are complications with that idea, of course. Comparing only the text is non-trivial in HTML messages, and even in plain-text messages one has to cope with the changes made by the quote itself (inserting > or other quote characters, or leading spaces, or re-wrapping paragraphs, etc.). Still, I'm not sure this is any worse than trying to determine what is and what is not a quote by applying heuristics solely to the current message.

Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum


locked tweak wording for confirmation of deleting a hashtag #suggestion

 

Currently, confirmation of deleting a hashtag reads: "Are you sure you wish to delete this hashtag and aliases pointing to it?" I deleted a tag this morning and after clicking "yes," had a momentary, delayed-reaction panic that all its aliases would also be deleted. I think what is meant is not "aliases pointing to it" but "pointers to it from aliases." 

--
J

Messages are the sole opinion of the author. 

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


locked tweak top-posting algorithm to allow explicit, intentional, changed, partial quotes #suggestion

 

I think the current algorithm that sweeps quotations into ellipses within an presumed top-post needs to be tweaked. In a recent response to someone, I quoted part of anold post by a third-party member in order to make a specific point. I indented, put quotes around the quote, and bolded sections of it. Yet when my response showed up onlist, the quote was gone, swept into ellipses. The person I was addressing didn't even see it; she only saw the link I'd also included. She thanked me for "the link."

I know that people "should" (as was pointed out when I complained about the top-posting algorithm prior to this) know that they can click on the ellipses. But some people don't, because they might think it's simply the prior message in the thread. Many times, it is not. And if a link is included, all they may notice is the link.

I think that if possible, the top-posting algorithm should be tweaked so that the quote is not put into ellipses if either 

(a) quotation marks are put around it (which, after all, makes it differ from the original); and/or

(b) parts of it are bolded (ditto); and/or

(c) it is indented.

All of these show that the quote was an intentional act and that it is not simply a top-posting situation.

The overzealousness (IMO) of the top-posting algorithm has been a minor but frustrating and continuing annoyance to me. So if any of this can be accomplished, I for one would greatly appreciate it.

Thanks!

--
J

Messages are the sole opinion of the author. 

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


locked Re: Temporarily disabled group creation

 

Mark,

Maybe, but anecdotally, it seems that some services send out FBL
messages in batches. We'll see bunches of FBL messages from more than
one person all at the same time.
I was thinking they might be batched daily, but I thought that would still provide enough resolution I think to distinguish a 30-day auto delete. If the batches are less frequent, or the auto-delete prompter then that could scratch that idea.

Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum


locked Re: Spam filter

 

On Fri, Sep 30, 2016 at 10:23 pm, Shal Farley wrote:
it has been a long time since I've noticed it catch anything. Others have reported that it catches only false positives, and misses actual spam.

Shal, thanks for providing my giggles for the evening. Still literally laughing out loud over this. :-) 
--
J

Messages are the sole opinion of the author. 

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


locked Re: Spam filter

 

Mark,

I am unfamiliar with how Y! Group's spam filter interacts with
groups; can someone clue me in?
I think a lot of people would say "What spam filter?" -- it has been a long time since I've noticed it catch anything. Others have reported that it catches only false positives, and misses actual spam.

But I think the more accurate statement is that for the last several years most (all?) spam is being caught (rejected or dropped) during the server transaction or at any rate well ahead of the forlorn content filter of Y!Groups itself. And I suspect that forlorn content filter has had very little attention over the course of time. It may be little more than a Bayesian discriminator, if even that sophisticated.

Effectiveness aside, the UI for it is similar to what one would expect in an email UI: each group has a Spam folder (or Pending Spam list) and messages diverted there can be examined by moderators with the primary options to "approve" or "delete". That's one difference versus a typical email UI: the approve choice posts the message; it doesn't merely move the message from the spam list to the pending list.
I think the other pending list operations (reject with reply to sender and edit) are also available.

The other side of the coin is in the regular Pending Message list, where the moderator has the option to "Delete as spam". That's also a little different than a normal email UI because it immediately deletes the message, not just move it to the Spam list.

During the 2013 "neo" redesign I railed at them to better coordinate the Pending Message UI and the Pending Spam UI. And neither of those had much in common with the UI for the message archive itself. Instead it was as if they handed loose descriptions of "messages in a list" to distinct teams - the list and message viewing controls are weirdly and confusingly different. That may have gotten better over time, but the paucity of messages in the Spam list means I haven't seen much of that UI in long time.

Moderators have a notification checkbox for whether they wish to be notified of Pending Spam.

There is (or was) separately some kind of filtering on messages sent to the -owner address. That filter had no UI at all: no way to know what false positives had been lost nor to inform the filter of false negatives. As with the primary spam filter, I've not seen any evidence that it is still in operation in quite some time.

Shal


On 9/30/2016 8:54 PM, Mark Fletcher wrote:
Hi All,

Based on the recent spammer incident and some conversations about some
evolving email standards, it's clear that I need to implement a spam
filter sooner rather than later. I am unfamiliar with how Y! Group's
spam filter interacts with groups; can someone clue me in? And are there
any issues with their implementation (the spam filter went in after the
acquisition/after I left)?

Thanks,
Mark
--
Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum


locked Site updates #changelog

 

Changes to the site this week:

  • INTERNAL: A lot of work on changes needed for the Enterprise version.
  • CHANGE: Blocking email addresses from many disposable email address providers.
  • CHANGE: New groups must be approved before they are listed in the directory and appear in search, to hopefully prevent spammers.
  • BUGFIX: Fix for deleting table columns in IE.
  • CHANGE: In the Messages/Expanded Messages view, moved Msg # box to the right to prevent confusion with the Search button.
  • CHANGE: Emailed login links no longer expire after the first time they're clicked.
  • CHANGE: When viewing owner messages, highlight the Members entry in the sidebar.
  • CHANGE: Group home page wording changes.
  • CHANGE: Pulled Groups.io CSS out into its own file.
  • BUGFIX: Sub group direct add sorting wasn't working.

Have a good weekend, everybody!

Mark


locked Spam filter

 

Hi All,

Based on the recent spammer incident and some conversations about some evolving email standards, it's clear that I need to implement a spam filter sooner rather than later. I am unfamiliar with how Y! Group's spam filter interacts with groups; can someone clue me in? And are there any issues with their implementation (the spam filter went in after the acquisition/after I left)?

Thanks,
Mark


locked Re: Temporarily disabled group creation

 

On Thu, Sep 29, 2016 at 9:25 PM, Shal Farley <shals2nd@...> wrote:

> I am not told how they were marked (I wish I was!).

Alas.

Maybe histogram the delays from message sent to report from each service; perhaps you can infer automatic expiration from the spam folder if there's a substantial bump.

Maybe, but anecdotally, it seems that some services send out FBL messages in batches. We'll see bunches of FBL messages from more than one person all at the same time.

 
The upheaval in Yahoo Groups over DMARC happened because some benighted services (looking at you Yahoo Mail and AOL) naively obeyed the p=reject policy. I suppose because, you know, no one uses email lists... Gmail was apparently smarter out of the gate: more list messages went to spam folders as a result of senders setting p=reject, but from what I read they didn't blindly reject them.

Yeah. I have a lot of respect for Google, for many reasons, but a big one is because Brandon Long works there. He was an eGroups engineer back in the day, one of the best engineers I've ever worked with. After the Yahoo acquisition, he ended up at Google, where he built Google Groups. Then Brandon went on to Gmail, where (I think) he's been ever since. A good guy.

Anyways, based on conversations on the ARC mailing list, it's clear that I need to implement a spam filter. I'll start a new thread to talk about that.

Thanks,
Mark


locked Re: Problem with content outside of the 'charset' #bug

 

I've noticed that since this fix (removing HTML from message summaries), Groups.io now displays the same "div..blockquote.." etc. junk that Yahoo started doing about a year ago for people posting from iPads (at least).

This is what's displayed in the message list for a particular message:

blockquote, div.yahoo_quoted { margin-left: 0 !important; border-left:1px #715FFA solid !important; padding-left:1ex !important; ba..
Whereas this is how the actual message reads:

Ok, I will do all I can to get those labs on here asap.  Thanks again!
Can something be done about this? It only started recently, and it seems to have started when the HTML started being stripped.

--
J

Messages are the sole opinion of the author. 

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


locked "xyz changed his name to abc via email" tons of log entries - meaning?

 

Mark,

Perhaps this has to do with the multiple-profile and naming issues you're working on (e.g., changed member name is not displayed after a moderator changes it). I see frequent log entries of the form "xyz changed his/her name to abc via email" in all my groups. Yet there is never a name change visible on the member's page. (For example, "Peter Piper changed his name to Peter via email" and the member's name still shows as Peter Piper.") I think that 100% of the time a member joins the group, the first thing that I see as a log entry is a message of this type, EVEN if the member shows no name at all on their page.

This has gone on for at least several months and I'm wondering if you can clarify it, and perhaps the status of the names situation generally.
--
J

Messages are the sole opinion of the author. 

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


locked Re: Temporarily disabled group creation

 

Mark,

What I look for is the one-click unsubscribe link in the
message footer. From that I can figure out the recipient.
Ah, clever, and useful.

I am not told how they were marked (I wish I was!).
Alas.

Maybe histogram the delays from message sent to report from each service; perhaps you can infer automatic expiration from the spam folder if there's a substantial bump.

Yeah. I mean spoofed emails are a fact of life on the Internet and I
think (hope) all the major providers recognize the spoofing. So I
don't think it'll ding our email sending reputation.
Not any more than breaking DMARC for the non-rewritten froms does, probably.

The upheaval in Yahoo Groups over DMARC happened because some benighted services (looking at you Yahoo Mail and AOL) naively obeyed the p=reject policy. I suppose because, you know, no one uses email lists... Gmail was apparently smarter out of the gate: more list messages went to spam folders as a result of senders setting p=reject, but from what I read they didn't blindly reject them.

But hey, the other day we had our first million email day!
Awesome! Congratulations.

Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum


locked Re: Temporarily disabled group creation

 

On Thu, Sep 29, 2016 at 6:17 PM, Shal Farley <shals2nd@...> wrote:


I wonder if ARC defends against this kind of replay. That is, when you put an ARC-seal on an outbound messages, is there something that makes this copy unique (a time stamp, serial number, or anything else that would invalidate a copied ARC-seal. If not that may be a weakness.

That's an excellent question and thanks for thinking of it. I just posted it to the ARC group, so we'll see.
 

Oh, interesting. I was wondering how much detail you get in the FBL reports. Specifically I was wondering if you were told the user and/or the Message-ID's of "marked" messages. And how they got marked (explict user action versus automation). But that was concerning some discussion in GMF@ about other issues with the FBL reports.

The FBL reports differ a little bit by domain. In general, the email providers try to obscure the email address of the person who received the email. What I look for is the one-click unsubscribe link in the message footer. From that I can figure out the recipient. Message-IDs are not obscured, and (I think for all of them), I get the entire message sent to me. I am not told how they were marked (I wish I was!). 

 
But, silly Yahoo, shouldn't the reports be flung back at the 5321.MailFrom? Sending it back to the From strikes me as akin to backscatter, though it was useful to you in this case. Or is it the (copied) Return-Path they're using? Either way...

Yeah, not sure. But still, pretty stupid of them.

 
> But I'm not sure country blocking isn't a great long term solution
> for this.

I'm reasonably sure it isn't.

For the moment I've blocked Morocco, because I need to actually get some work done. Having that guy around was distracting.

 
Well, I think you have some head-scratching yet to do on this problem.

Yeah. I mean spoofed emails are a fact of life on the Internet and I think (hope) all the major providers recognize the spoofing. So I don't think it'll ding our email sending reputation. It sucks though because I get angry support emails from people demanding to be removed from groups they aren't even on.

We're still small fry in the grand scheme of things, which is nice in a way because I'm able to add defenses over time. But makes us still open to being blocked if things go bad. But hey, the other day we had our first million email day!


Mark


locked Re: Temporarily disabled group creation

 

Mark,

I'm watching them try to register new email addresses in real time
and being denied. They are persistent.
That's the way of ants and other automatons.

What they're doing is creating a group, doing a post, grabbing the
resulting email, and sending that out to people themselves. ... And
our SPF record doesn't apply because they're using a different
envelope domain (I did just change our SPF records from softfail to
fail, but if I understand what's happening, that won't help here).
No, but my understanding is that the misalignment of the 5322.From with the 5321.MailFrom will cause DMARC to fail anyway, for those receiving services that check it. Of course, so do the messages legitimately through Groups.io when you don't rewrite the From.

So if I understand everything correctly, the only way to 'fix' this
would be to set DMARC p=reject and re-write *all* From lines in
emails. I'd obviously rather not do that.
I'd rather you not do that too. The only advantage I can see is that all outbound messages could pass DMARC, instead of only those from the services you rewrite.

I wonder if ARC defends against this kind of replay. That is, when you put an ARC-seal on an outbound messages, is there something that makes this copy unique (a time stamp, serial number, or anything else that would invalidate a copied ARC-seal. If not that may be a weakness.

The reason I was able to see one of these emails is that Yahoo has
sent some FBL reports to us from these forged emails (people who
received these messages marked them as spam, and Yahoo thought they
originated from us).
Oh, interesting. I was wondering how much detail you get in the FBL reports. Specifically I was wondering if you were told the user and/or the Message-ID's of "marked" messages. And how they got marked (explict user action versus automation). But that was concerning some discussion in GMF@ about other issues with the FBL reports.

But, silly Yahoo, shouldn't the reports be flung back at the 5321.MailFrom? Sending it back to the From strikes me as akin to backscatter, though it was useful to you in this case. Or is it the (copied) Return-Path they're using? Either way...

But I'm not sure country blocking isn't a great long term solution
for this.
I'm reasonably sure it isn't.

Well, I think you have some head-scratching yet to do on this problem.

--
Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum


locked Re: Temporarily disabled group creation

 

Oh, and I know the spammer is reading beta@. 

Exciting!

Mark

On Thu, Sep 29, 2016 at 3:56 PM, Mark Fletcher <markf@corp.groups.io> wrote:
On Thu, Sep 29, 2016 at 1:29 PM, Shal Farley <shals2nd@...> wrote:
Mark,

> Most of the groups are being created using disposable email
> addresses. I am in the process of blocking those email providers.

Maybe you need a throttle (and internal alert) on the rate of group creation from unknown email providers. Then if you get an alert for a rash of creations from an unknown you can decide whether to white or black list the provider, or leave it gray ("unknown") pending further information. Sort of a poor-man's reputation system.


I think at least for now just checking against a comprehensive list of disposable email providers seems to be helping a lot. I'm watching them try to register new email addresses in real time and being denied. They are persistent.


> Also, most likely related, someone is forging spam emails from
> Groups.io. So I may need to look into some changes to mitigate that.
> I'll keep you posted.

If they're forging the groups.io domain in the From field then maybe DMARC p=reject is your friend.

What they're doing is creating a group, doing a post, grabbing the resulting email, and sending that out to people themselves. So the DKIM sig still authenticates. And our SPF record doesn't apply because they're using a different envelope domain (I did just change our SPF records from softfail to fail, but if I understand what's happening, that won't help here).

So if I understand everything correctly, the only way to 'fix' this would be to set DMARC p=reject and re-write *all* From lines in emails. I'd obviously rather not do that.

The reason I was able to see one of these emails is that Yahoo has sent some FBL reports to us from these forged emails (people who received these messages marked them as spam, and Yahoo thought they originated from us). 

I've also thought about blocking all IP addresses from Morocco. We currently block all IPs from Afghanistan, because of an earlier incident. But I'm not sure country blocking isn't a great long term solution for this.

(Note for the majority of people on beta@ who probably don't understand most of the above: The email forgery isn't a big deal right now and won't affect your groups).

Thanks,
Mark



locked Re: Temporarily disabled group creation

 

On Thu, Sep 29, 2016 at 1:29 PM, Shal Farley <shals2nd@...> wrote:
Mark,

> Most of the groups are being created using disposable email
> addresses. I am in the process of blocking those email providers.

Maybe you need a throttle (and internal alert) on the rate of group creation from unknown email providers. Then if you get an alert for a rash of creations from an unknown you can decide whether to white or black list the provider, or leave it gray ("unknown") pending further information. Sort of a poor-man's reputation system.


I think at least for now just checking against a comprehensive list of disposable email providers seems to be helping a lot. I'm watching them try to register new email addresses in real time and being denied. They are persistent.


> Also, most likely related, someone is forging spam emails from
> Groups.io. So I may need to look into some changes to mitigate that.
> I'll keep you posted.

If they're forging the groups.io domain in the From field then maybe DMARC p=reject is your friend.

What they're doing is creating a group, doing a post, grabbing the resulting email, and sending that out to people themselves. So the DKIM sig still authenticates. And our SPF record doesn't apply because they're using a different envelope domain (I did just change our SPF records from softfail to fail, but if I understand what's happening, that won't help here).

So if I understand everything correctly, the only way to 'fix' this would be to set DMARC p=reject and re-write *all* From lines in emails. I'd obviously rather not do that.

The reason I was able to see one of these emails is that Yahoo has sent some FBL reports to us from these forged emails (people who received these messages marked them as spam, and Yahoo thought they originated from us). 

I've also thought about blocking all IP addresses from Morocco. We currently block all IPs from Afghanistan, because of an earlier incident. But I'm not sure country blocking isn't a great long term solution for this.

(Note for the majority of people on beta@ who probably don't understand most of the above: The email forgery isn't a big deal right now and won't affect your groups).

Thanks,
Mark


locked Re: Temporarily disabled group creation

 

Mark,

Most of the groups are being created using disposable email
addresses. I am in the process of blocking those email providers.
Maybe you need a throttle (and internal alert) on the rate of group creation from unknown email providers. Then if you get an alert for a rash of creations from an unknown you can decide whether to white or black list the provider, or leave it gray ("unknown") pending further information. Sort of a poor-man's reputation system.

Also, most likely related, someone is forging spam emails from
Groups.io. So I may need to look into some changes to mitigate that.
I'll keep you posted.
If they're forging the groups.io domain in the From field then maybe DMARC p=reject is your friend.

If they're forging a "subscriber's" domain, as if it is a message that passed through a group, I'm not sure. The message would already fail DMARC, but that puts the disposition policy in the hands of the domain that was forged.

Shal

--
Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum


locked Re: Temporarily disabled group creation

 

On Thu, Sep 29, 2016 at 7:00 AM, J_Catlady <j.olivia.catlady@...> wrote:
On Wed, Sep 28, 2016 at 11:00 pm, Christopher Hallsworth wrote:
Good idea, never thought of that one.

That's why Mark makes the big bucks. ;) 


Heh. :-)

The system is working, in that I've deleted 128 groups just so far this morning. Seems to be a determined spammer from Morocco. 

Most of the groups are being created using disposable email addresses. I am in the process of blocking those email providers.

Also, most likely related, someone is forging spam emails from Groups.io. So I may need to look into some changes to mitigate that. I'll keep you posted.

Thanks,
Mark


locked Re: Problem with content outside of the 'charset' #bug

 

Hi Mark,

Thanks very much for getting back to me! Yes, I'm talking about the summary.

Your code works as designed: garbage in garbage out. However, some groups, like mine, have a need to support multiple languages. The use case is for people communicating in several languages, e.g., Japanese students studying Russian. Some mail clients, like outlook.live.com are uni-lingual.

Tests on my test group indicate that the plain text portion contains characters that, when displayed as utf-8 (the charset that I used to send the message), show up correctly in the body of the message but not in the summary. That indicates to me that you are doing something different to the characters when you display the summary than when you display the message contents. In this case, there is nothing wrong with the characters themselves. The only garbage is that their charset has been wrongly declared

Thanks again!

David.


locked Re: Temporarily disabled group creation

 

On Wed, Sep 28, 2016 at 11:00 pm, Christopher Hallsworth wrote:
Good idea, never thought of that one.

That's why Mark makes the big bucks. ;) 
--
J

Messages are the sole opinion of the author. 

I wish I could shut up, but I can't, and I won't. - Desmond Tutu

17381 - 17400 of 28395