[jira] Commented: (LUCENE-1585) Allow to control how payloads are merged

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1585) Allow to control how payloads are merged

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867382#action_12867382 ]

Shai Erera commented on LUCENE-1585:
------------------------------------

bq. Could we also use this for the term bytes itsself?

I think you'd want to use the same approach, yes. But I'm not sure I want to reuse the same classes for that purpose, for several reasons:
* The classes have the word Payload all over the place - javadocs, names etc. And for a good reason IMO - that's what they do.
* One is expected to include different PP for different Directory and Term, but to convert the NumericField terms, I don't think one will use different PPs at all? I.e. a single TermsConverter / NumericFieldsTermConverter will be good for whatever Directory + Term?
* The sort of operation you suggest (converting terms) seems to be a one time op -- when I migrate my indexes? PPP on the other hand (at least in my case) will be used whenever I call addIndexes* so that I can process and rewrite the payloads of the incoming indexes.

So while both will do byte[] conversion, I think those are two separate tools. Your should probably exist in a o.a.l.migration package or something, because it will be relevant to index migration only? Or did I misunderstood you?

> Allow to control how payloads are merged
> ----------------------------------------
>
>                 Key: LUCENE-1585
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1585
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Shai Erera
>            Priority: Minor
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-1585_3x.patch, LUCENE-1585_3x.patch, LUCENE-1585_3x.patch, LUCENE-1585_3x.patch, LUCENE-1585_3x.patch, LUCENE-1585_trunk.patch, LUCENE-1585_trunk.patch, LUCENE-1585_trunk.patch
>
>
> Lucene handles backwards-compatibility of its data structures by
> converting them from the old into the new formats during segment
> merging.
> Payloads are simply byte arrays in which users can store arbitrary
> data. Applications that use payloads might want to convert the format
> of their payloads in a similar fashion. Otherwise it's not easily
> possible to ever change the encoding of a payload without reindexing.
> So I propose to introduce a PayloadMerger class that the SegmentMerger
> invokes to merge the payloads from multiple segments. Users can then
> implement their own PayloadMerger to convert payloads from an old into
> a new format.
> In the future we need this kind of flexibility also for column-stride
> fields (LUCENE-1231) and flexible indexing codecs.
> In addition to that it would be nice if users could store version
> information in the segments file. E.g. they could store "in segment _2
> the term a:b uses payloads of format x.y".

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]