join query limitations

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

join query limitations

matthew sporleder
I have hit a bit of a cross-road with our usage of solr where I want
to include some slightly dynamic data.

I want to ask solr to find things like "text query" but only if they
meet some specific criteria.  When I have all of those criteria
indexed, everything works great.  (text contains "apples", in_season=1
,sort by latest)

Now I would like to add a criteria which changes every day -
popularity of a document, specifically.  This appeared to be *the*
canonical use case for external field files but I have 50M documents
(and growing) so a *text* file doesn't fit the bill.

I also looked at using a !join but the limitations of !join, as I
understand them, appear to mean I can't use it for my use case? aka I
can't actually use the data from my traffic-stats core to sort/filter
"text contains" "apples", in_season=1, sort by most traffic, sort by
latest

The last option appears to be updating all of my documents every
single day, possibly using atomic/partial updates, but even those have
a growing list of gotchas: losing stored=false documents is a big one,
caveats I don't quite understand related to copyFields, changes to the
_version_ field (the _version_ field is also a non-indexed, non-stored
single valued docValues field;), etc

Where else can I look?  The last time we attempted something like this
we ended up rebuilding the index from scratch each day and shuffling
it out, which was really pretty nasty.

Thanks,
Matt
Reply | Threaded
Open this post in threaded view
|

Re: join query limitations

Erick Erickson
Have you seen “In-place updates”?

See:
https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html

Then use the field as part of a function query. Since it’s non-indexed, you
won’t be searching on it. That said, you can do a lot with function queries
to satisfy use-cases.

Best.
Erick

> On Sep 14, 2020, at 3:12 PM, matthew sporleder <[hidden email]> wrote:
>
> I have hit a bit of a cross-road with our usage of solr where I want
> to include some slightly dynamic data.
>
> I want to ask solr to find things like "text query" but only if they
> meet some specific criteria.  When I have all of those criteria
> indexed, everything works great.  (text contains "apples", in_season=1
> ,sort by latest)
>
> Now I would like to add a criteria which changes every day -
> popularity of a document, specifically.  This appeared to be *the*
> canonical use case for external field files but I have 50M documents
> (and growing) so a *text* file doesn't fit the bill.
>
> I also looked at using a !join but the limitations of !join, as I
> understand them, appear to mean I can't use it for my use case? aka I
> can't actually use the data from my traffic-stats core to sort/filter
> "text contains" "apples", in_season=1, sort by most traffic, sort by
> latest
>
> The last option appears to be updating all of my documents every
> single day, possibly using atomic/partial updates, but even those have
> a growing list of gotchas: losing stored=false documents is a big one,
> caveats I don't quite understand related to copyFields, changes to the
> _version_ field (the _version_ field is also a non-indexed, non-stored
> single valued docValues field;), etc
>
> Where else can I look?  The last time we attempted something like this
> we ended up rebuilding the index from scratch each day and shuffling
> it out, which was really pretty nasty.
>
> Thanks,
> Matt

Reply | Threaded
Open this post in threaded view
|

Re: join query limitations

matthew sporleder
Yes but "the _version_ field is also a non-indexed, non-stored single
valued docValues field;"  <- is that a problem?

My schema has this:
  <!-- to use updateLog: _version_field must exist in schema, using
       indexed="true" stored="true" and multiValued="false"
  -->
  <field name="_version_" type="long" indexed="true" stored="true"/>

I don't know if I use the updateLog or not.  How can I find out?

I think that would work for me as I could just make a dynamic fild like:
<dynamicField name="*_atomici" type="int" indexed="false"
stored="false" multiValued="false" required="false" docValues="true"
/>

---
Yes it is just for functions, sorting, and boosting

On Mon, Sep 14, 2020 at 4:51 PM Erick Erickson <[hidden email]> wrote:

>
> Have you seen “In-place updates”?
>
> See:
> https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html
>
> Then use the field as part of a function query. Since it’s non-indexed, you
> won’t be searching on it. That said, you can do a lot with function queries
> to satisfy use-cases.
>
> Best.
> Erick
>
> > On Sep 14, 2020, at 3:12 PM, matthew sporleder <[hidden email]> wrote:
> >
> > I have hit a bit of a cross-road with our usage of solr where I want
> > to include some slightly dynamic data.
> >
> > I want to ask solr to find things like "text query" but only if they
> > meet some specific criteria.  When I have all of those criteria
> > indexed, everything works great.  (text contains "apples", in_season=1
> > ,sort by latest)
> >
> > Now I would like to add a criteria which changes every day -
> > popularity of a document, specifically.  This appeared to be *the*
> > canonical use case for external field files but I have 50M documents
> > (and growing) so a *text* file doesn't fit the bill.
> >
> > I also looked at using a !join but the limitations of !join, as I
> > understand them, appear to mean I can't use it for my use case? aka I
> > can't actually use the data from my traffic-stats core to sort/filter
> > "text contains" "apples", in_season=1, sort by most traffic, sort by
> > latest
> >
> > The last option appears to be updating all of my documents every
> > single day, possibly using atomic/partial updates, but even those have
> > a growing list of gotchas: losing stored=false documents is a big one,
> > caveats I don't quite understand related to copyFields, changes to the
> > _version_ field (the _version_ field is also a non-indexed, non-stored
> > single valued docValues field;), etc
> >
> > Where else can I look?  The last time we attempted something like this
> > we ended up rebuilding the index from scratch each day and shuffling
> > it out, which was really pretty nasty.
> >
> > Thanks,
> > Matt
>
Reply | Threaded
Open this post in threaded view
|

Re: join query limitations

Erick Erickson
What version of Solr are you using? ‘cause 8x has this definition for _version_

<!-- doc values are enabled by default for primitive types such as long so we don't index the version field  -->
 <field name="_version_" type="plong" indexed="false" stored="false"/>

and I find no text like you’re seeing in any schema file in 8x….

So with a prior version, “try it and see”? See: https://issues.apache.org/jira/browse/SOLR-9449 and linked JIRAs,
the _version_ can be indexed=“false” since 6.3 at least if it’s docValues=“true". It’s not clear to me that it needed
to be indexed=“true” even before that, but no guarantees.

updateLog will be defined in solrconfig.xml, but unless you’re on a very old version of Solr it doesn’t matter
‘cause you don’t need to have indexed=“true”. Updatelog is not necessary if you’re not running SolrCloud...

I strongly urge you to completely remove all your indexes (perhaps create a new collection) and re-index
from scratch if you change the definition. You might be able to get away with deleting all the docs then
re-indexing, but just re-indexing all the docs without starting fresh can have “interesting” results.

Best,
Erick

> On Sep 14, 2020, at 5:16 PM, matthew sporleder <[hidden email]> wrote:
>
> Yes but "the _version_ field is also a non-indexed, non-stored single
> valued docValues field;"  <- is that a problem?
>
> My schema has this:
>  <!-- to use updateLog: _version_field must exist in schema, using
>       indexed="true" stored="true" and multiValued="false"
>  -->
>  <field name="_version_" type="long" indexed="true" stored="true"/>
>
> I don't know if I use the updateLog or not.  How can I find out?
>
> I think that would work for me as I could just make a dynamic fild like:
> <dynamicField name="*_atomici" type="int" indexed="false"
> stored="false" multiValued="false" required="false" docValues="true"
> />
>
> ---
> Yes it is just for functions, sorting, and boosting
>
> On Mon, Sep 14, 2020 at 4:51 PM Erick Erickson <[hidden email]> wrote:
>>
>> Have you seen “In-place updates”?
>>
>> See:
>> https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html
>>
>> Then use the field as part of a function query. Since it’s non-indexed, you
>> won’t be searching on it. That said, you can do a lot with function queries
>> to satisfy use-cases.
>>
>> Best.
>> Erick
>>
>>> On Sep 14, 2020, at 3:12 PM, matthew sporleder <[hidden email]> wrote:
>>>
>>> I have hit a bit of a cross-road with our usage of solr where I want
>>> to include some slightly dynamic data.
>>>
>>> I want to ask solr to find things like "text query" but only if they
>>> meet some specific criteria.  When I have all of those criteria
>>> indexed, everything works great.  (text contains "apples", in_season=1
>>> ,sort by latest)
>>>
>>> Now I would like to add a criteria which changes every day -
>>> popularity of a document, specifically.  This appeared to be *the*
>>> canonical use case for external field files but I have 50M documents
>>> (and growing) so a *text* file doesn't fit the bill.
>>>
>>> I also looked at using a !join but the limitations of !join, as I
>>> understand them, appear to mean I can't use it for my use case? aka I
>>> can't actually use the data from my traffic-stats core to sort/filter
>>> "text contains" "apples", in_season=1, sort by most traffic, sort by
>>> latest
>>>
>>> The last option appears to be updating all of my documents every
>>> single day, possibly using atomic/partial updates, but even those have
>>> a growing list of gotchas: losing stored=false documents is a big one,
>>> caveats I don't quite understand related to copyFields, changes to the
>>> _version_ field (the _version_ field is also a non-indexed, non-stored
>>> single valued docValues field;), etc
>>>
>>> Where else can I look?  The last time we attempted something like this
>>> we ended up rebuilding the index from scratch each day and shuffling
>>> it out, which was really pretty nasty.
>>>
>>> Thanks,
>>> Matt
>>

Reply | Threaded
Open this post in threaded view
|

Re: join query limitations

matthew sporleder
This probably carried forward from a very old version organically.  I
am running 7.7

On Mon, Sep 14, 2020 at 6:25 PM Erick Erickson <[hidden email]> wrote:

>
> What version of Solr are you using? ‘cause 8x has this definition for _version_
>
> <!-- doc values are enabled by default for primitive types such as long so we don't index the version field  -->
>  <field name="_version_" type="plong" indexed="false" stored="false"/>
>
> and I find no text like you’re seeing in any schema file in 8x….
>
> So with a prior version, “try it and see”? See: https://issues.apache.org/jira/browse/SOLR-9449 and linked JIRAs,
> the _version_ can be indexed=“false” since 6.3 at least if it’s docValues=“true". It’s not clear to me that it needed
> to be indexed=“true” even before that, but no guarantees.
>
> updateLog will be defined in solrconfig.xml, but unless you’re on a very old version of Solr it doesn’t matter
> ‘cause you don’t need to have indexed=“true”. Updatelog is not necessary if you’re not running SolrCloud...
>
> I strongly urge you to completely remove all your indexes (perhaps create a new collection) and re-index
> from scratch if you change the definition. You might be able to get away with deleting all the docs then
> re-indexing, but just re-indexing all the docs without starting fresh can have “interesting” results.
>
> Best,
> Erick
>
> > On Sep 14, 2020, at 5:16 PM, matthew sporleder <[hidden email]> wrote:
> >
> > Yes but "the _version_ field is also a non-indexed, non-stored single
> > valued docValues field;"  <- is that a problem?
> >
> > My schema has this:
> >  <!-- to use updateLog: _version_field must exist in schema, using
> >       indexed="true" stored="true" and multiValued="false"
> >  -->
> >  <field name="_version_" type="long" indexed="true" stored="true"/>
> >
> > I don't know if I use the updateLog or not.  How can I find out?
> >
> > I think that would work for me as I could just make a dynamic fild like:
> > <dynamicField name="*_atomici" type="int" indexed="false"
> > stored="false" multiValued="false" required="false" docValues="true"
> > />
> >
> > ---
> > Yes it is just for functions, sorting, and boosting
> >
> > On Mon, Sep 14, 2020 at 4:51 PM Erick Erickson <[hidden email]> wrote:
> >>
> >> Have you seen “In-place updates”?
> >>
> >> See:
> >> https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html
> >>
> >> Then use the field as part of a function query. Since it’s non-indexed, you
> >> won’t be searching on it. That said, you can do a lot with function queries
> >> to satisfy use-cases.
> >>
> >> Best.
> >> Erick
> >>
> >>> On Sep 14, 2020, at 3:12 PM, matthew sporleder <[hidden email]> wrote:
> >>>
> >>> I have hit a bit of a cross-road with our usage of solr where I want
> >>> to include some slightly dynamic data.
> >>>
> >>> I want to ask solr to find things like "text query" but only if they
> >>> meet some specific criteria.  When I have all of those criteria
> >>> indexed, everything works great.  (text contains "apples", in_season=1
> >>> ,sort by latest)
> >>>
> >>> Now I would like to add a criteria which changes every day -
> >>> popularity of a document, specifically.  This appeared to be *the*
> >>> canonical use case for external field files but I have 50M documents
> >>> (and growing) so a *text* file doesn't fit the bill.
> >>>
> >>> I also looked at using a !join but the limitations of !join, as I
> >>> understand them, appear to mean I can't use it for my use case? aka I
> >>> can't actually use the data from my traffic-stats core to sort/filter
> >>> "text contains" "apples", in_season=1, sort by most traffic, sort by
> >>> latest
> >>>
> >>> The last option appears to be updating all of my documents every
> >>> single day, possibly using atomic/partial updates, but even those have
> >>> a growing list of gotchas: losing stored=false documents is a big one,
> >>> caveats I don't quite understand related to copyFields, changes to the
> >>> _version_ field (the _version_ field is also a non-indexed, non-stored
> >>> single valued docValues field;), etc
> >>>
> >>> Where else can I look?  The last time we attempted something like this
> >>> we ended up rebuilding the index from scratch each day and shuffling
> >>> it out, which was really pretty nasty.
> >>>
> >>> Thanks,
> >>> Matt
> >>
>