Topics

moderated What is the algorithm for the display order of groups after entering a keyword search? #misc


 

The bad examples are more frequent than the good ones. In a search on "IBD," three groups that just mention the term tangentially in their descriptions display first. (One of them uses the term simply to say that the group does NOT deal with IBD.) The only group that is *specifically* about IBD (and has "IBD" in its title) comes up last. In this case, there are only four groups total. But in other cases there can be a page or more before the actual relevant group even shows up.
https://groups.io/search?q=ibd

--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


Duane
 

On Sat, Nov 21, 2020 at 02:41 PM, J_Catlady wrote:
There is currently NO relationship between the search results order and the actual search term.
I believe there is.  It's the order that Elasticsearch found them in the (listed) group-name/description database.

Duane


Glenn Glazer
 

On Sat, 11/21 13:43, Duane wrote:
On Sat, Nov 21, 2020 at 02:41 PM, J_Catlady wrote:
There is currently NO relationship between the search results order and the actual search term.
I believe there is.  It's the order that Elasticsearch found them in the (listed) group-name/description database.

Duane

But that order is effectively random.

https://stackoverflow.com/questions/23919296/how-is-elastic-search-sorting-when-no-sort-option-specified-and-no-search-query

Best,

Glenn

--
PG&E Delenda Est


 

On Sat, Nov 21, 2020 at 01:43 PM, Duane wrote:
I believe there is.  It's the order that Elasticsearch found them
Which, as you yourself have said repeatedly, is nothing more than reverse alpha with lowercase names first. This has zero relationship to the search term. The set of results is related to the search term. The display does not.
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


 

On Sat, Nov 21, 2020 at 01:58 PM, J_Catlady wrote:
The display does not
Should read "the display order does not"
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


 

On Sat, Nov 21, 2020 at 01:52 PM, Glenn Glazer wrote:

But that order is effectively random.
If so, then groups.io apparently does a bit of post-processing to put the results in reverse alpha order with lowercase names first. Either way, the display order totally unrelated to the search term. An extremely simple fix would be simply to prioritize the groups with the search term IN the name (especailly since groups.io itself calls the search "By Name"). If groups.io is already post-processing the results of elastisearch, it may as well do it in a way that makes a nonzero amount of sense.
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


Glenn Glazer
 

On Sat, 11/21 14:02, J_Catlady wrote:
On Sat, Nov 21, 2020 at 01:52 PM, Glenn Glazer wrote:

But that order is effectively random.
If so, then groups.io apparently does a bit of post-processing to put the results in reverse alpha order with lowercase names first. Either way, the display order totally unrelated to the search term. An extremely simple fix would be simply to prioritize the groups with the search term IN the name (especailly since groups.io itself calls the search "By Name"). If groups.io is already post-processing the results of elastisearch, it may as well do it in a way that makes a nonzero amount of sense.

I've never seen the code, I can't speak to what it does or doesn't do.

Best,

Glenn

--
PG&E Delenda Est


Duane
 

On Sat, Nov 21, 2020 at 03:52 PM, Glenn Glazer wrote:
But that order is effectively random.
Not completely.  It has to be searching the (listed) group-name/description database which is apparently stored in ASCII order.  Since the results page shows By Name, then it's doing a bottom up search.  Anyone with a bit of computer programming experience knows that in ASCII all of the upper case characters come before all of the lower case characters which is the reverse of the order the results are listed - bottom up.

Duane


Glenn Glazer
 

On Sat, 11/21 14:18, Duane wrote:
On Sat, Nov 21, 2020 at 03:52 PM, Glenn Glazer wrote:
But that order is effectively random.
Not completely.  It has to be searching the (listed) group-name/description database which is apparently stored in ASCII order.  Since the results page shows By Name, then it's doing a bottom up search.  Anyone with a bit of computer programming experience knows that in ASCII all of the upper case characters come before all of the lower case characters which is the reverse of the order the results are listed - bottom up.

Duane

If the search is specified without a sort order, then elasticsearch does not promise an ordering to the result set as that SO link pointed out. As J_Catlady pointed out, the code could very easily be doing post-processing on the result set. What we see are the final results, none of us have the ability to determine which part of the code is doing what.

I'm not arguing that the end effect is desirable. I definitely agree there is room for improvement. I'm simply pointing out that the relationship between the search results order and the actual search term are not necessarily due to Elasticsearch, as you asserted.

Best,

Glenn

--
PG&E Delenda Est


Duane
 

On Sat, Nov 21, 2020 at 04:24 PM, Glenn Glazer wrote:
I'm simply pointing out that the relationship between the search results order and the actual search term are not necessarily due to Elasticsearch, as you asserted.
True, it's totally dependent on what order the database is saved.  Elasticsearch will list them in the order they're retrieved from the search, since all results will have a score of 1, unless some processing is done.

Duane


 

It doesn't seem to matter in what order elastisearch returns the results, because as Duane himself and others have said, and as we have seen with our own eyes, the display order is turned into strictly alphabetical order (actually reverse alphabetical) with lower case names coming first. 

"I believe the default is descending lower case, followed by descending upper case, according to group name.  For example:
water
balloon
Water
Balloon"

https://beta.groups.io/g/main/message/26868

It is clear that the display order bears no relationship to the search term and I find that problematic. How Duane can argue first that the order is strictly alphabetical, and later that it has something to do with the search term, is a mystery to me. Unless, Duane, you changed your mind about the alpha order? I have played around with these searches for a couple days now and have seen nothing to the contrary. It seems to me that you were right in your very first response here. I found it unbelievable at first, but it does seem to be the case that groups.io pays no attention to the search term itself when displaying the results.

--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


 

On Sat, Nov 21, 2020 at 02:24 PM, Glenn Glazer wrote:
the relationship between the search results order and the actual search term are not necessarily due to Elasticsearch, as you asserted.
And it seems that it in fact is not.
Of course, there's one person who could answer this in 2 seconds, and that is the person I addressed in my original message in this thread. :)
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


Donald Hellen
 

Could someone word this well enough to make this a suggestion? I'll
chime in with my support.

It's not a bug as such as it was set up this crazy way.

Donald

On Sat, 21 Nov 2020 14:37:51 -0800, "J_Catlady"
<j.olivia.catlady@gmail.com> wrote:

It doesn't seem to matter in what order elastisearch returns the results, because as Duane himself and others have said, and as we have seen with our own eyes, the display order is turned into strictly alphabetical order (actually reverse alphabetical) with lower case names coming first.

----------------------------------------------------
Some ham radio groups you may be interested in:
https://groups.io/g/ICOM https://groups.io/g/Ham-Antennas
https://groups.io/g/HamRadioHelp https://groups.io/g/Baofeng
https://groups.io/g/CHIRP https://rf-amplifiers.groups.io/g/main


 

On Sat, Nov 21, 2020 at 03:31 PM, Donald Hellen wrote:
Could someone word this well enough to make this a suggestion? I'll
chime in with my support.
Sure. But I'm not sure we should start a new topic for that. Here's what I would say to sum it all up.

Currently, results of a "By Name" group search are displayed in an order that does not relate to the search term, namely: reverse alpha order according to the group name, with lowercase names first. There is no prioritization of (a) groups whose names contain the search term over groups whose descriptions  only (rather than the name) contain the search term; (b) any criterion relating to the search term vis-a-vis the group name (e.g., the search term appears earlier in the group name or description than in others); or (c) any other criterion such as activity level, recency, popularity, etc. Once the user selects "By Name," those other criteria go away.

This results in some odd, unexpected, and often unhelpful behavior to both users looking for groups dealing with a particular subject, and group owners who see their groups that deal with the subject of the term display below, or even pages after, groups who don't really deal with the subject of the search term.

Examples:

- If the search term is "bananas," a group called "jungle" whose description includes the sentence "This group does not deal with bananas" will display before a group whose name is AllAboutBananas.

- A group with 10 members and no activity for the past three years will display above a highly active group simply because the smaller, inactive group's name comes later in alphabetic order than the active group, or is lowercase instead of capitalized.

My suggestions, in increasing order of complexity and usefulness:

(1) Simplest quick fix for now: Remove the text "By Name" as a possible search criterion (because the results are not "By Name" relative to the search term anyway) and simply let users enter a term. The search is currently *not* "by name." Only the results are "by name" in the display order.

(2) At least change the reverse alpha order display to alpha order and drop the criterion about upper or lowercase because it's meaningless.

(3) Prioritize groups whose names actually contain the search term over groups whose names do not contain the term but whose descriptions do.

(4) Best solution: once a user enters a search term, let them ask for a sort by "most popular," "most active," etc., just as appears on the left now. Essentially, they could click on one of those criteria only, or a search term only, or both.
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


Duane
 

On Sat, Nov 21, 2020 at 04:37 PM, J_Catlady wrote:
How Duane can argue first that the order is strictly alphabetical, and later that it has something to do with the search term, is a mystery to me.
That's not conflicting information.  Taken as a whole it makes perfect sense if you understand how databases and searches work. I stand by both statements.

Duane


 

On Sat, Nov 21, 2020 at 04:27 PM, Duane wrote:
Taken as a whole it makes perfect sense if you understand how databases and searches work.
LOL I spent about 10 years doing query optimazation for a database company. Thanks for the great info.

 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


Duane
 

On Sat, Nov 21, 2020 at 06:19 PM, J_Catlady wrote:
(1) Simplest quick fix for now: Remove the text "By Name" as a possible search criterion (because the results are not "By Name" relative to the search term anyway) and simply let users enter a term. The search is currently *not* "by name." Only the results are "by name" in the display order.
Although the results ARE reverse sorted by name, removing the "(by Name)" designation on the results page may prevent confusion.

(2) At least change the reverse alpha order display to alpha order and drop the criterion about upper or lowercase because it's meaningless.
It's quite possible that the order shown is the default for Elasticsearch on the database being used, so it's not a matter of dropping it, but adding a new parameter.

(4) Best solution: once a user enters a search term, let them ask for a sort by "most popular," "most active," etc., just as appears on the left now. Essentially, they could click on one of those criteria only, or a search term only, or both.
Then the problem once again becomes "What is the default results order?"  I can only speak for myself, but I seldom go beyond the initial search and usually wouldn't bother clicking on additional options.  There's no way to know which result might be the one I'm looking for, so going through the default list is fine with me.  This is where an Advanced Search button (bringing up a box or page) next to the Search box might lead some folks to dig a little deeper.

Duane


 

On Sun, Nov 22, 2020 at 06:42 AM, Duane wrote:
Although the results ARE reverse sorted by name, removing the "(by Name)" designation on the results page may prevent confusion.
See my other thread with the bug report. When you don't enter a search term and click "by name," it's alpha (not reverse alpha) order so that's inconsistent. Probably the search does display the elastisearch results, but that needs to be fixed.

More importantly, I think we've all been misconstruing the way the search page was supposed to work (or looks like it was supposed to work). We've been confusing "by name" with the entering of a search term, possibly because the page jumps to the "by name" sorting criterion the minute you enter a search term. That's part of the bug report.
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


Marv Waschke
 

Here's a little background on how searches like this work. Many of the people here are programmers and will find this boring, but if you are not among the cognoscenti, you may find this explanation helpful.

The sort order of the results of a search and the search itself are different beasts.

The search determines which groups make it into the results, the sort order determines which groups show up on top. Sophisticated search engines like Google have complex search algorithms and equally sophisticated sorting algorithms.

As far as I can see, Mark's search method is quite simple: lump together the group name, description, and tags, then look for matches against the search term. When two or more words separated by white space are in the search box, both words must appear for a match, but they can be separated by other words. If there are quotes around the search term, the term must match exactly. For example, if the search box contains [feline lymphoma] (ignore the square brackets)  a description like "lymphoma is a cancer, which sometimes appears in feline pets" will be included in the results. If the box contains ["feline lymphoma"]  , the previous description would not be included because extra words separate feline and lymphoma, and they appear in the wrong order.

The sort order does not affect what appears in the results, only the order the results appear. The gio group search page has four possible sort orders. These only apply when the search box is empty.

The first, and default, is Most Popular. The group with the most members appears first. The group with the next largest membership appear next, and so on.

Next is Most Active. The group with the most topics appears first.

Newest lists the most recently created group first.

By Name is a bit weird. It lists group names in ascii order, which is familiar to programmers, but strange to the uninitiated. Numbers 0-9 come first. Upper case letter A-Z next. Finally lower case letters a-z.

There is another oddity to the By Name sort. At the bottom of the page, you can select a letter. By selecting A, you get all the group names that start with A. Oddly, the names are then sorted case insensitive, i.e. case is ignored and the sort is what most people would expect of an alphabetized list.

Just thought I would lay out what is happening. I apologize for being so pedantic.
Best, Marv


Glenn Glazer
 

This is what I was getting at with my other post about abstract ideas.

To be even more ;) pedantic, in SQL and similar query languages, SELECT is not the same as ORDER BY, nor are the things selected or ordered.

Best,

Glenn

On Sun, 11/22 11:52, Marv Waschke wrote:
Here's a little background on how searches like this work. Many of the people here are programmers and will find this boring, but if you are not among the cognoscenti, you may find this explanation helpful.

The sort order of the results of a search and the search itself are different beasts.

The search determines which groups make it into the results, the sort order determines which groups show up on top. Sophisticated search engines like Google have complex search algorithms and equally sophisticated sorting algorithms.

As far as I can see, Mark's search method is quite simple: lump together the group name, description, and tags, then look for matches against the search term. When two or more words separated by white space are in the search box, both words must appear for a match, but they can be separated by other words. If there are quotes around the search term, the term must match exactly. For example, if the search box contains [feline lymphoma] (ignore the square brackets)  a description like "lymphoma is a cancer, which sometimes appears in feline pets" will be included in the results. If the box contains ["feline lymphoma"]  , the previous description would not be included because extra words separate feline and lymphoma, and they appear in the wrong order.

The sort order does not affect what appears in the results, only the order the results appear. The gio group search page has four possible sort orders. These only apply when the search box is empty.

The first, and default, is Most Popular. The group with the most members appears first. The group with the next largest membership appear next, and so on.

Next is Most Active. The group with the most topics appears first.

Newest lists the most recently created group first.

By Name is a bit weird. It lists group names in ascii order, which is familiar to programmers, but strange to the uninitiated. Numbers 0-9 come first. Upper case letter A-Z next. Finally lower case letters a-z.

There is another oddity to the By Name sort. At the bottom of the page, you can select a letter. By selecting A, you get all the group names that start with A. Oddly, the names are then sorted case insensitive, i.e. case is ignored and the sort is what most people would expect of an alphabetized list.

Just thought I would lay out what is happening. I apologize for being so pedantic.
Best, Marv


--
PG&E Delenda Est