[jira] Created: (SOLR-264) Support 'random' sort order

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ryan McKinley updated SOLR-264:
-------------------------------

    Attachment: RandomSortField.java

wow.  

Here is an updated version that uses the field name hash for the seed.  I added a few comments and I have seen it run and give random results.  

It gets configured with:
<dynamicField name="rand*" type="random" indexed="true" stored="false"/>

then you get nicely repeatably random results for:

 *  http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_1234%20desc
 *  http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_2345%20desc
 *  http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_ABDC%20desc
 *  http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_21%20desc

thanks yonik!

> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley updated SOLR-264:
------------------------------

    Attachment: RandomSortField.java

If we want the comparator to be transitive, a subtraction doesn't work with negative numbers in there because of overflow.  We could either do explicit comparisons, or only use positive ints.  I chose positive ints because it might be a nicer range for ValueSource (function query).

> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506344 ]

Yonik Seeley commented on SOLR-264:
-----------------------------------

Might be nice to check cache statistics to see that the same seed results in a hit.
same with function query:  _val_(rand_1234)

> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506347 ]

Hoss Man commented on SOLR-264:
-------------------------------

i was kind of wondering about the negative overflow (isn't that underflow) but i was trusting you that it worked ... isn't there still a potential problem with the transitive property if  i.doc + seed causes positive integer overflow?

(i guess it's okay because the overflow is the same regardless of whether it's i or j, but in the previous case the overflow happend *after*the hash call.

FWIW: i still think newComparator should be something like...

        public ScoreDocComparator newComparator(IndexReader reader, String fieldname) throws IOException {
          return new RandomComparator(seed ^ reader.getVersion());
        }

...so that people who want orderings that are randomized per each <commit/> can just use...

  <field name="random" type="random" />

With the attachment as is, most changes to the index (ie: add a few documents, delete a few documents) won't have a significant impact on the "random" order because they won't change the majority of the docIds... you have to change the "seed" to see any noticable effects.

> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506352 ]

Yonik Seeley commented on SOLR-264:
-----------------------------------

> i guess it's okay because the overflow is the same regardless of whether it's i or j, but in the previous case the overflow happend *after*the hash call

Yep, no bits of randomness are lost, so having an intermediate value over/underflow is fine.

Including the reader version is an interesting idea, and would probably lead to less confusion (a user might otherwise be tricked into thinking that docs always sort the same relative to each other across a commit).


> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506353 ]

Yonik Seeley commented on SOLR-264:
-----------------------------------

LOL... I'm obviously too tired for more coding tonight.  Could the next person to modify this patch please change
  return key >>> 31;  // only positive numbers
    to
  return key >>> 1;

> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ryan McKinley updated SOLR-264:
-------------------------------

    Attachment: RandomSortField.java

changed to use:
 "return key >>> 1;"
and
 seed ^ reader.getVersion()

actually it needs to be:
 (int)(seed^reader.getVersion())

the long->int overflow wrapping should not be a problem.

> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506959 ]

Koji Sekiguchi commented on SOLR-264:
-------------------------------------

If I post http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_1234%20desc  several times, Solr returns different response every time. This looks good.
But I'm curious how to avoid queryResultCache, while QueryResultKey(Q,F,S) is always same.

Maybe I'm wrong, but:

public SortComparatorSource getFactory() {
  :
  return new SortComparatorSource() {
    :
    public boolean equals(Object o) {
      return (o instanceof RandomSort) && getField().equals(((RandomSort) o).getField());
    }
  };
}

doesn't the equals method use SortComparatorSource, instead of RandomSort? If so, this always return false, then cause to avoid queryResultCache...?

> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Koji Sekiguchi updated SOLR-264:
--------------------------------

    Attachment: SOLR-264-RandomSortField-2.patch

> public SortComparatorSource getFactory() {
>   :
>   return new SortComparatorSource() {
>     :
>    public boolean equals(Object o) {
>      return (o instanceof RandomSort) && getField().equals(((RandomSort) o).getField());
>    }
>  };
>}

The equals() method should care of SortComparatorSource, instead of RandomSort...?

> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortField-2.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Koji Sekiguchi updated SOLR-264:
--------------------------------

    Comment: was deleted

> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortField-2.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Issue Comment Edited: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12507090 ]

Koji Sekiguchi edited comment on SOLR-264 at 6/23/07 12:41 PM:
---------------------------------------------------------------

> public SortComparatorSource getFactory() {
>   :
>   return new SortComparatorSource() {
>     :
>    public boolean equals(Object o) {
>      return (o instanceof RandomSort) && getField().equals(((RandomSort) o).getField());
>    }
>  };
>}

The equals() method should care of SortComparatorSource, instead of RandomSort...? The patch uses SortComparatorSource in equals() method.


 was:
> public SortComparatorSource getFactory() {
>   :
>   return new SortComparatorSource() {
>     :
>    public boolean equals(Object o) {
>      return (o instanceof RandomSort) && getField().equals(((RandomSort) o).getField());
>    }
>  };
>}

The equals() method should care of SortComparatorSource, instead of RandomSort...?

> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortField-2.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Koji Sekiguchi updated SOLR-264:
--------------------------------

    Attachment: SOLR-264-RandomSortField-2.patch

Some non-ASCII characters were slipped into the previous patch. I'd like to upload new one.
regards,


> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortField-2.patch, SOLR-264-RandomSortField-2.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508045 ]

Hoss Man commented on SOLR-264:
-------------------------------

Good catch Koji, i applied your patch as well as fixed up a few other things i noticed (ValueSource wasn't using IndexReader in the seed, confusing "seed" terminology used because of getSeed(field), etc...)

> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortField-2.patch, SOLR-264-RandomSortField-2.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (SOLR-264) Support 'random' sort order

Clark Perkins (Jira)
In reply to this post by Clark Perkins (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man resolved SOLR-264.
---------------------------

       Resolution: Fixed
    Fix Version/s: 1.3

this was all committed a little while ago and seems to be working.


> Support 'random' sort order
> ---------------------------
>
>                 Key: SOLR-264
>                 URL: https://issues.apache.org/jira/browse/SOLR-264
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Ryan McKinley
>            Priority: Minor
>             Fix For: 1.3
>
>         Attachments: RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, RandomSortField.java, SOLR-264-RandomSortField-2.patch, SOLR-264-RandomSortField-2.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch, SOLR-264-RandomSortOrder.patch
>
>
> Support querying for random documents:
>   http://localhost:8983/solr/select/?q=*:*&fl=sku&sort=random%20desc

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

12