[jira] Created: (SOLR-2068) Search Grouping: collapse by string specialization

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (SOLR-2068) Search Grouping: collapse by string specialization

JIRA jira@apache.org
Search Grouping: collapse by string specialization
--------------------------------------------------

                 Key: SOLR-2068
                 URL: https://issues.apache.org/jira/browse/SOLR-2068
             Project: Solr
          Issue Type: Sub-task
            Reporter: Yonik Seeley


Create specialized implementations for collapsing by an indexed string field.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-2068) Search Grouping: collapse by string specialization

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926290#action_12926290 ]

Yonik Seeley commented on SOLR-2068:
------------------------------------

Going back over my old notes on how to efficiently do a string field per-segment:

Phase1:
 - Basically, hash based on ord (or a direct index lookup if the # of ords is small enough).  We don't look up the value of the string at this point.
 - When a segment changes, we need to convert the ords from the old segment to the new segment (i.e. look up it's value in the old segment, and find the ord of that in the new segment).
   - if the group value is not found in the new segment, the remove it from the hash.  Keep it in the ordered map since it can still be pushed out by other insertions.

Phase 2:
 - at the start of each segment, look up the ords for the values and hash the group based on that ord (or leave it out of the hash if it didn't exist in that segment).

Martijn's optimization in SOLR-2205 probably made Phase1 less important (except if there are very few unique groups), so perhaps we should start with Phase2 first.


> Search Grouping: collapse by string specialization
> --------------------------------------------------
>
>                 Key: SOLR-2068
>                 URL: https://issues.apache.org/jira/browse/SOLR-2068
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Yonik Seeley
>
> Create specialized implementations for collapsing by an indexed string field.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-2068) Search Grouping: collapse by string specialization

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/SOLR-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated SOLR-2068:
----------------------------------------

    Attachment: SOLR-2068.patch

The improvements extracted from SOLR-2205

> Search Grouping: collapse by string specialization
> --------------------------------------------------
>
>                 Key: SOLR-2068
>                 URL: https://issues.apache.org/jira/browse/SOLR-2068
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Yonik Seeley
>         Attachments: SOLR-2068.patch
>
>
> Create specialized implementations for collapsing by an indexed string field.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-2068) Search Grouping: collapse by string specialization

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926340#action_12926340 ]

Martijn van Groningen commented on SOLR-2068:
---------------------------------------------

bq.  we would be better off using an OpenBitSet - or even better, a sparse set since the number of elements
I get the OpenBitSet part, but what actual class do you mean with a sparse set? (Not a primitive array I assume)

> Search Grouping: collapse by string specialization
> --------------------------------------------------
>
>                 Key: SOLR-2068
>                 URL: https://issues.apache.org/jira/browse/SOLR-2068
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Yonik Seeley
>         Attachments: SOLR-2068.patch
>
>
> Create specialized implementations for collapsing by an indexed string field.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Issue Comment Edited: (SOLR-2068) Search Grouping: collapse by string specialization

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926338#action_12926338 ]

Martijn van Groningen edited comment on SOLR-2068 at 10/29/10 12:50 PM:
------------------------------------------------------------------------

The improvements extracted from SOLR-2205 dedicated the the phase 2 improvements.

      was (Author: martijn):
    The improvements extracted from SOLR-2205
 

> Search Grouping: collapse by string specialization
> --------------------------------------------------
>
>                 Key: SOLR-2068
>                 URL: https://issues.apache.org/jira/browse/SOLR-2068
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Yonik Seeley
>         Attachments: SOLR-2068.patch
>
>
> Create specialized implementations for collapsing by an indexed string field.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-2068) Search Grouping: collapse by string specialization

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/SOLR-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated SOLR-2068:
----------------------------------------

    Attachment: SOLR-2068.patch

* Updated the patch to the latest trunk
* Changed the boolean array to a OpenBitSet
* Made the Phase2GroupCollector per-segment friendly

> Search Grouping: collapse by string specialization
> --------------------------------------------------
>
>                 Key: SOLR-2068
>                 URL: https://issues.apache.org/jira/browse/SOLR-2068
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Yonik Seeley
>         Attachments: SOLR-2068.patch, SOLR-2068.patch
>
>
> Create specialized implementations for collapsing by an indexed string field.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-2068) Search Grouping: collapse by string specialization

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927700#action_12927700 ]

Yonik Seeley commented on SOLR-2068:
------------------------------------

The ord in one segment isn't equivalent to the ord in another segment... so I don't think the currently attached patch will work.

> Search Grouping: collapse by string specialization
> --------------------------------------------------
>
>                 Key: SOLR-2068
>                 URL: https://issues.apache.org/jira/browse/SOLR-2068
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Yonik Seeley
>         Attachments: SOLR-2068.patch, SOLR-2068.patch
>
>
> Create specialized implementations for collapsing by an indexed string field.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-2068) Search Grouping: collapse by string specialization

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/SOLR-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley updated SOLR-2068:
-------------------------------

    Attachment: SOLR-2068.patch

Here's a draft patch (completely untested) that implements the phase2 specialization:

bq.  at the start of each segment, look up the ords for the values and hash the group based on that ord (or leave it out of the hash if it didn't exist in that segment).


> Search Grouping: collapse by string specialization
> --------------------------------------------------
>
>                 Key: SOLR-2068
>                 URL: https://issues.apache.org/jira/browse/SOLR-2068
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Yonik Seeley
>         Attachments: SOLR-2068.patch, SOLR-2068.patch, SOLR-2068.patch
>
>
> Create specialized implementations for collapsing by an indexed string field.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-2068) Search Grouping: collapse by string specialization

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927893#action_12927893 ]

Martijn van Groningen commented on SOLR-2068:
---------------------------------------------

bq. The ord in one segment isn't equivalent to the ord in another segment... so I don't think the currently attached patch will work.
Yes, your right. I didn't notice that since the index I used for testing was optimized and there was only one subreader...

> Search Grouping: collapse by string specialization
> --------------------------------------------------
>
>                 Key: SOLR-2068
>                 URL: https://issues.apache.org/jira/browse/SOLR-2068
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Yonik Seeley
>         Attachments: SOLR-2068.patch, SOLR-2068.patch, SOLR-2068.patch
>
>
> Create specialized implementations for collapsing by an indexed string field.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]