Too many fields to Sort in Solr

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Too many fields to Sort in Solr

adityab
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

Erick Erickson
Seems like a reasonable thing to do. Examine the debug output to insure
that there's no short-circuiting being done as far as ConstantScoreQuery...

Best
Erick


On Tue, Mar 19, 2013 at 7:05 PM, adityab <[hidden email]> wrote:

> Hi All,
>
> I want to validate my approach by the experts, just to make sure i am on
> doing anything wrong.
>
> #Docs in Solr : 25M
> Solr Versin : 4.2
>
> Our requirement is to list top download document based on user country.
> So we have a dynamic field "*numdownload.**" which is evaluate as
> *numdownloads.<countryId>*
>
> Now as sorting is an expensive and also uses large amount of java heap, I
> planned to use this field in boosting result.
>
> Old Query
> q=*:*&fq=countryId:1&sort=numdownloads.1 desc
>
> which i changed to
>  q={!boost b=numdownloads.1}*:*&fq=countryId:1
>
> Is my approach correct. Any better alternate ?
>
> thanks
> Aditya
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

adityab
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

Erick Erickson
Certainly that will be true for the bare q=*:*, I meant with the boosting
clause added.

Best
Erick


On Sun, Mar 24, 2013 at 7:01 PM, adityab <[hidden email]> wrote:

> thanks Eric. in this query "q=*:*" the Lucene score is always 1
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4050944.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

Joel Bernstein
I pretty sure that the boost function is going load up the fieldCache for
that field, which will have the same memory footprint as sorting.

You may want to try out disk based doc values (new 4.2). Documentation is
sparse on this but you can specify a field as being docValues="true" and
set it to a field type where docValuesFormat="Disk"

example:

   <field name="rec_time" type="date_disk" indexed="true" required="true"
multiValued="false" docValues="true" default="NOW" />


    <fieldType name="date_disk" class="solr.TrieDateField"
precisionStep="0" docValuesFormat="Disk" positionIncrementGap="0"/>



On Mon, Mar 25, 2013 at 11:14 AM, Erick Erickson <[hidden email]>wrote:

> Certainly that will be true for the bare q=*:*, I meant with the boosting
> clause added.
>
> Best
> Erick
>
>
> On Sun, Mar 24, 2013 at 7:01 PM, adityab <[hidden email]> wrote:
>
> > thanks Eric. in this query "q=*:*" the Lucene score is always 1
> >
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4050944.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>



--
Joel Bernstein
Professional Services LucidWorks
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

adityab
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

Joel Bernstein
Hi,

I tested this config on Solr 4.2 this morning and it worked:
<fieldType name="long" class="solr.TrieLongField" precisionStep="0"
docValuesFormat="Disk" positionIncrementGap="0"/>

<field name="YYYYMMDDhh" type="long" indexed="true" stored="true"
required="true" docValues="true" multiValued="false" />

I also loaded data and ran a sort  and looked at the heap with jvisualvm
and the longs were not loaded into the jvm's heap. The sort was also very
fast, although only on 600,000 records.

Possibly you are not on Solr 4.2? Can you post both your filedType
definition and your field definition?

Joel





On Thu, Mar 28, 2013 at 12:57 AM, adityab <[hidden email]> wrote:

> Hi Joel,
> you are correct, boost function populates the field cache. Well i am not
> aware of docValue, so while trying the example you provided i see the error
> when i define the field type
>
> Caused by: org.apache.solr.common.SolrException: FieldType 'dvLong' is
> configured with a docValues format, but the codec does not support it:
> class
> org.apache.solr.core.SolrCore$3
>         at org.apache.solr.core.SolrCore.initCodec(SolrCore.java:854)
>         at org.apache.solr.core.SolrCore.<init>(SolrCore.java:719)
>         ... 13 more
>
> My field defination:
> <fieldType name="dvLong" class="solr.TrieLongField" precisionStep="0"
> positionIncrementGap="0" docValuesFormat="Disk"/>
>
> what am i missing here?
>
> thanks
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4051960.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



--
Joel Bernstein
Professional Services LucidWorks
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

adityab
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

Joel Bernstein
OK, you'll need to re-index. Shutdown, delete the data, re-index.


On Thu, Mar 28, 2013 at 9:12 AM, adityab <[hidden email]> wrote:

> Here is the field type definition. same as what you posted yesterday just a
> different name.
>
> <fieldType name="dvLong" class="solr.TrieLongField" precisionStep="0"
> docValuesFormat="Disk" positionIncrementGap="0"/>
>
> And Field Definition
> <field name="lcontNumOfDownloads" type="dvLong" indexed="true"
> stored="true"
> default="0" docValues="true"/>
>
>
> as soon as i restart the server i see the exception in log. removing the
> *docValuesFormat="Disk"* from the field type i don't see this exception.
>
> 01:49:37,177 ERROR [org.apache.solr.core.CoreContainer]
> (coreLoadExecutor-3-thread-1) Unable to create core: collection1:
> org.apache.solr.common.SolrException: FieldType 'dvLong' is configured with
> a docValues format, but the codec does not support it: class
> org.apache.solr.core.SolrCore$3
>         at org.apache.solr.core.SolrCore.<init>(SolrCore.java:806)
> [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13]
>         at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
> [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13]
>         at
> org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:1021)
> [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13]
>         at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:1051)
> [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13]
>         at
> org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:634)
> [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13]
>         at
> org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629)
> [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13]
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> [rt.jar:1.7.0_09]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> [rt.jar:1.7.0_09]
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [rt.jar:1.7.0_09]
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> [rt.jar:1.7.0_09]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> [rt.jar:1.7.0_09]
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> [rt.jar:1.7.0_09]
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> [rt.jar:1.7.0_09]
>         at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09]
> Caused by: org.apache.solr.common.SolrException: FieldType 'dvLong' is
> configured with a docValues format, but the codec does not support it:
> class
> org.apache.solr.core.SolrCore$3
>         at org.apache.solr.core.SolrCore.initCodec(SolrCore.java:854)
> [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13]
>         at org.apache.solr.core.SolrCore.<init>(SolrCore.java:719)
> [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13]
>         ... 13 more
>
> 01:49:37,202 ERROR [org.apache.solr.core.CoreContainer]
> (coreLoadExecutor-3-thread-1) null:org.apache.solr.common.SolrException:
> Unable to create core: collection1
>         at
> org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1672)
>         at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:1057)
>         at
> org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:634)
>         at
> org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.solr.common.SolrException: FieldType 'dvLong' is
> configured with a docValues format, but the codec does not support it:
> class
> org.apache.solr.core.SolrCore$3
>         at org.apache.solr.core.SolrCore.<init>(SolrCore.java:806)
>         at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
>         at
> org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:1021)
>         at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:1051)
>         ... 10 more
> Caused by: org.apache.solr.common.SolrException: FieldType 'dvLong' is
> configured with a docValues format, but the codec does not support it:
> class
> org.apache.solr.core.SolrCore$3
>         at org.apache.solr.core.SolrCore.initCodec(SolrCore.java:854)
>         at org.apache.solr.core.SolrCore.<init>(SolrCore.java:719)
>         ... 13 more
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052036.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



--
Joel Bernstein
Professional Services LucidWorks
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

adityab
This post was updated on .
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

adityab
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

Joel Bernstein
I didn't have to do anything with the codecs to make it work. Checked my
solrconfig.xml and the codecFactory element is not present.  I'm running
the out of the box jetty setup.


On Thu, Mar 28, 2013 at 11:58 AM, adityab <[hidden email]> wrote:

> Update ---
>
> I was able to fix the exception by adding following line in solrconfig.xml
>
> <codecFactory name="CodecFactory" class="solr.SchemaCodecFactory" />
>
> Not sure if its mentioned in any document to have this declared in config
> file.
> I am now re-indexing and data on the master and will perform test to see if
> it works as expected.
>
> thanks for your support.
>
> Aditya
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052091.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



--
Joel Bernstein
Professional Services LucidWorks
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

adityab
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

Joel Bernstein
Not, sure that making changes to the solrconfig.xml is going down the right
path here. There might something else with your setup that's causing this
issue. I'm not sure what it would be though.


On Thu, Mar 28, 2013 at 1:38 PM, adityab <[hidden email]> wrote:

> Wo. that's strange.
>
> I tried toggling with the code factory line in solrconfig.xml (attached in
> this post)
> commenting gives me error where as un-commenting works.
>
> can you please take a look into config and let me know if anything wrong
> there?
>
> thanks
> Aditya
>
>
> solrconfig.xml
> <http://lucene.472066.n3.nabble.com/file/n4052131/solrconfig.xml>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052131.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



--
Joel Bernstein
Professional Services LucidWorks
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

adityab
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

Joel Bernstein
OK, that makes sense. How are DocValues working for you?


On Fri, Mar 29, 2013 at 9:02 AM, adityab <[hidden email]> wrote:

> Hi Joel,
> Might have an answer for this. Initially my servers were on 3.5 and then i
> moved to Solr 4.0. at this time i use the solrconfig.xml that was in the
> example and updated is with parameters i changed in 3.5 for the
> environment.
> there was no "<codecFactory class="solr.SchemaCodecFactory"/>" in the 4.0
> example solrconfig.xml file. We continued to us the same file and updated
> war to 4.1 then 4.2 just by changing the luceneMatchVersion in the existing
> solrconfig.xml file.
>
> I was looking at the 4.2 and comparing it with the one we have and i see
> that the *codefactory* is in the example solrconfig.xml file.
>
>
>   <codecFactory class="solr.SchemaCodecFactory"/>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052374.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



--
Joel Bernstein
Professional Services LucidWorks
Reply | Threaded
Open this post in threaded view
|

Re: Too many fields to Sort in Solr

adityab
CONTENTS DELETED
The author has deleted this message.