[jira] Created: (LUCENE-2514) Change Term to use bytes

classic Classic list List threaded Threaded
61 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882341#action_12882341 ]

Robert Muir commented on LUCENE-2514:
-------------------------------------

{quote}
I also removed a Java 1.6 interface override - the Generics Policeman gives a ticket! I don't understand where those come from, Java 1.6 should also fail to compile as the ant build uses -source 1.5...?
{quote}

Sorry, i am on a mac right now and i dont think i configured it correctly... (though ant test never complained... this is wierd).
Normallly my IDE does not generate this... but at the same time it is something we should fix the build for, as i think Eclipse
will generate these by default if configured for Java 6, which solr uses.

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882342#action_12882342 ]

Uwe Schindler commented on LUCENE-2514:
---------------------------------------

bq. mainly I am searching on uses of Term.text() and such to ensure I do not introduce performance regressions.

I would temporary remove all String methods from Term and try to compile core. If this works you should find all perf regressions in both directions.

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2514:
--------------------------------

    Attachment: LUCENE-2514.patch

attached is an updated patch, with uwe's changes, plus some additional conversions like TermsFilter and FieldCacheTermsFilter

The range ones are a bit tricky, mainly because they work with collators with makes no sense with byte[]. but if collator is null then byte[] makes sense.

the collator stuff is silly in a way, if we switch collation to byte[] it will use less ram than even the original String in lucene 3.x, and sort much faster.

one option might be to split the collating range stuff into its own classes or something, i think its a bit confusing how collation is mixed in with 'binary' order... it tricks you into thinking the 'default' is UCA or default locale or something, but is neither.

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882347#action_12882347 ]

Robert Muir commented on LUCENE-2514:
-------------------------------------

bq. I would temporary remove all String methods from Term and try to compile core. If this works you should find all perf regressions in both directions.

i did this, but there is a lot of unrelated stuff that uses Term string methods and is perfectly fine. especially many tests and things like queryparser.

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882352#action_12882352 ]

Uwe Schindler commented on LUCENE-2514:
---------------------------------------

{quote}
bq. I would temporary remove all String methods from Term and try to compile core. If this works you should find all perf regressions in both directions.

i did this, but there is a lot of unrelated stuff that uses Term string methods and is perfectly fine. especially many tests and things like queryparser.
{quote}

Yeah tests can use string methods, i meant only core classes should compile without Term's String methods.

bq. one option might be to split the collating range stuff into its own classes or something, i think its a bit confusing how collation is mixed in with 'binary' order... it tricks you into thinking the 'default' is UCA or default locale or something, but is neither.

I was always thinking about factoring out collation stuff from TermRangeQuery. I would like to have a pure TermRangeQuery without any collations things and maybe a CollationTermRangeQuery or whatever...

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882364#action_12882364 ]

Robert Muir commented on LUCENE-2514:
-------------------------------------

{quote}
Yeah tests can use string methods, i meant only core classes should compile without Term's String methods.
{quote}

Ok, i will have another look at this (ignoring tests and queryparser and such). Maybe i will find something interesting.

{quote}
I was always thinking about factoring out collation stuff from TermRangeQuery. I would like to have a pure TermRangeQuery without any collations things and maybe a CollationTermRangeQuery or whatever...
{quote}

yeah, i think this would be better in the future too. but for now, i can work with what we have. it just means lots of things like String lowerBound; /* only used when collator != null */ and such

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler updated LUCENE-2514:
----------------------------------

    Attachment: LUCENE-2514-MTQPagedBytes.patch

Here an improvement of the MTQ only patch:

The auto MTQ rewrite mode now collects all terms into a PagedBytes until cutoff. This maybe better memory-wise, not sure if this is really needed. For me it was just some usage training :-) and the number of objects is lower, especially for large cutoff numbers.

Mike?

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2514:
--------------------------------

    Attachment: LUCENE-2514.patch

{quote}
We also need to fix FieldCache/TermRangeQuery, since they now take separate String upper/lower. We could just add corresponding BytesRef methods?
{quote}

This actually makes the api ugly and problematic for ctors, because of open-ended values (null). it would be nice to avoid requiring users to cast here...
{noformat}
    [javac] /../eclipse/workspace/solrcene-spell/lucene/src/test/org/apache/lucene/search/TestTermRangeQuery.java:100: reference to TermRangeQuery is ambiguous, both method TermRangeQuery(java.lang.String,java.lang.String,java.lang.String,boolean,boolean) in org.apache.lucene.search.TermRangeQuery and method TermRangeQuery(java.lang.String,org.apache.lucene.util.BytesRef,org.apache.lucene.util.BytesRef,boolean,boolean) in org.apache.lucene.search.TermRangeQuery match
    [javac]     TermRangeQuery query = new TermRangeQuery("content", null, null, true, true);
{noformat}

attached is my updates to TermRangeQuery etc before realizing this. it also includes updated FieldCacheRangeQuery complete with generics violations.

maybe when the policeman wakes up he will have ideas. i don't want the api to be ugly.


> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler updated LUCENE-2514:
----------------------------------

    Attachment: LUCENE-2514-MTQPagedBytes.patch

Improved version of PagedBytes MTQ cut off collector. It adds a method to PagedBytes.Reader to sequentially read all BytesRefs without a separate offset array.

Mike, if you are fine with that, we should add this to the global patch for this issue.

We should maybe also fix PagedBytes.freeze(boolean), as the parameter is currently unused. For the use case here, reallocating the last block is not really needed, it can stay as is. Maybe we should readd support for this parameter.

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler updated LUCENE-2514:
----------------------------------

    Attachment: LUCENE-2514-MTQPagedBytes.patch

This patch only adds a correct trim to PagesBytes.freeze(boolean). The impl was missing, but is important here for performance.

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Issue Comment Edited: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882422#action_12882422 ]

Robert Muir edited comment on LUCENE-2514 at 6/25/10 10:09 AM:
---------------------------------------------------------------

edit: i added newlines and reformatted so the issue is readable. sorry.

{quote}
We also need to fix FieldCache/TermRangeQuery, since they now take separate String upper/lower. We could just add corresponding BytesRef methods?
{quote}

This actually makes the api ugly and problematic for ctors, because of open-ended values (null). it would be nice to avoid requiring users to cast here...
{noformat}
reference to TermRangeQuery is ambiguous, both method
TermRangeQuery(java.lang.String,java.lang.String,java.lang.String,boolean,boolean)
in org.apache.lucene.search.TermRangeQuery and method
TermRangeQuery(java.lang.String,org.apache.lucene.util.BytesRef,
org.apache.lucene.util.BytesRef,boolean,boolean)
in org.apache.lucene.search.TermRangeQuery match
    [javac]     TermRangeQuery query = new TermRangeQuery("content", null, null, true, true);
{noformat}

attached is my updates to TermRangeQuery etc before realizing this. it also includes updated FieldCacheRangeQuery complete with generics violations.

maybe when the policeman wakes up he will have ideas. i don't want the api to be ugly.


      was (Author: rcmuir):
    {quote}
We also need to fix FieldCache/TermRangeQuery, since they now take separate String upper/lower. We could just add corresponding BytesRef methods?
{quote}

This actually makes the api ugly and problematic for ctors, because of open-ended values (null). it would be nice to avoid requiring users to cast here...
{noformat}
    [javac] /../eclipse/workspace/solrcene-spell/lucene/src/test/org/apache/lucene/search/TestTermRangeQuery.java:100: reference to TermRangeQuery is ambiguous, both method TermRangeQuery(java.lang.String,java.lang.String,java.lang.String,boolean,boolean) in org.apache.lucene.search.TermRangeQuery and method TermRangeQuery(java.lang.String,org.apache.lucene.util.BytesRef,org.apache.lucene.util.BytesRef,boolean,boolean) in org.apache.lucene.search.TermRangeQuery match
    [javac]     TermRangeQuery query = new TermRangeQuery("content", null, null, true, true);
{noformat}

attached is my updates to TermRangeQuery etc before realizing this. it also includes updated FieldCacheRangeQuery complete with generics violations.

maybe when the policeman wakes up he will have ideas. i don't want the api to be ugly.

 

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882588#action_12882588 ]

Robert Muir commented on LUCENE-2514:
-------------------------------------

Uwe, thanks for the MTQ updates. i will merge them with my previous patch (not my latest broken one!)

After looking at RangeQuery/Filter, i think i would prefer to make a subtask to refactor the collation part out.
For example, we could make CollatedRangeQuery.

In my opinion this is really very different from an ordinary binary-ordered range query and is confusing to both
users and the code to be mixed in. I think we should consider fixing the API by splitting these and adding
documentation to the migrate.txt

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882591#action_12882591 ]

Uwe Schindler commented on LUCENE-2514:
---------------------------------------

+1

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2514:
--------------------------------

    Attachment: LUCENE-2514.patch

updated patch:
* dropped the range query/filter changes so we can refactor those separately.
* removed the BytesRef.EMPTY, uwe pointed out this is very dangerous since we cant prevent someone from modifying it.
* tried to optimize preflex codec, removing more conversions and removing scratchbytesref copying... i think more can be done here.
* merged in uwe's latest multitermquery changes.

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2514:
--------------------------------

    Attachment: LUCENE-2514.patch

ok i added a couple more optimizations to preflex, to avoid creating so many strings.

i still don't like all the conversions for dealing with terms: is the problem just the shared prefixes and surrogate dance? I wonder if we can do something tricky to avoid this.


> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler updated LUCENE-2514:
----------------------------------

    Attachment: LUCENE-2514.patch

Converted the rest of TestSurrogates (removed FieldAnText class, now direktly uses Term, as Term now supports BytesRef directly). Test respects verbose param.

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2514:
--------------------------------

    Attachment: LUCENE-2514.patch

Thanks Uwe. I added some additional changes:
* naive switch of MemoryIndex to byte terms (this one can use pagedbytes maybe too)
* removed some more unnecessary byte->string in instantiated, uninvertedfield


> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2514:
--------------------------------

    Attachment: LUCENE-2514.patch

the term vectors api should be in sync with Term, but still uses String.
this patch switches them to bytesref too.


> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler updated LUCENE-2514:
----------------------------------

    Attachment: LUCENE-2514.patch

Updated patch with recent trunk commits (TestTV, TestSurrogates)

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2514) Change Term to use bytes

David Eric Pugh (Jira)
In reply to this post by David Eric Pugh (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885061#action_12885061 ]

Uwe Schindler commented on LUCENE-2514:
---------------------------------------

We should commit this soon and solve the rest separately?

Robert? Mike?

> Change Term to use bytes
> ------------------------
>
>                 Key: LUCENE-2514
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2514
>             Project: Lucene - Java
>          Issue Type: Task
>          Components: Search
>    Affects Versions: 4.0
>            Reporter: Robert Muir
>         Attachments: LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-MTQPagedBytes.patch, LUCENE-2514-surrogates-dance.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch, LUCENE-2514.patch
>
>
> in LUCENE-2426, the sort order was changed to codepoint order.
> unfortunately, Term is still using string internally, and more importantly its compareTo() uses the wrong order [utf-16].
> So MultiTermQuery, etc (especially its priority queues) are currently wrong.
> By changing Term to use bytes, we can also support terms encoded as bytes such as numerics, instead of using
> strange string encodings.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

1234