Newbie problem ordering results

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Newbie problem ordering results

germanbio
Hello everybody

I have the following (resumed) schema:

....
    <field name="title" type="text" indexed="true" stored="true"
multiValued="true"/>
   <field name="titleorder" type="string" indexed="true" stored="true"
multiValued="true"/>
   <field name="contributor" type="text" indexed="true" stored="true"
multiValued="true"/>
   <field name="contributorfacet" type="textFacetN" indexed="true"
stored="true" multiValued="true"/>
   <field name="contributororder" type="string" indexed="true"
stored="true" multiValued="true"/>
.....
....
<copyField source="title" dest="text" />
<copyField source="title" dest="titleorder" />
<copyField source="contributor" dest="text" />
<copyField source="contributor" dest="contributorfacet" />
<copyField source="contributor" dest="contributororder" />
...

I use for instance contributor for searching, contributorfacet for
faceting and order for ordering results, but when I try to order using
contributororder, Solr says that cannot order by a tokenized
field...(?)

I'm using Solr 1.4 nightly. Is this a bug? I believe that in previous
versions I have this issue working...

Regards and thanks
Germán
Reply | Threaded
Open this post in threaded view
|

Re: Newbie problem ordering results

Avlesh Singh
Can you please post the fieldType definition for the "string" field in your
schema.xml?

Cheers
Avlesh

On Tue, Aug 11, 2009 at 9:52 AM, Germán Biozzoli
<[hidden email]>wrote:

> Hello everybody
>
> I have the following (resumed) schema:
>
> ....
>    <field name="title" type="text" indexed="true" stored="true"
> multiValued="true"/>
>   <field name="titleorder" type="string" indexed="true" stored="true"
> multiValued="true"/>
>   <field name="contributor" type="text" indexed="true" stored="true"
> multiValued="true"/>
>   <field name="contributorfacet" type="textFacetN" indexed="true"
> stored="true" multiValued="true"/>
>   <field name="contributororder" type="string" indexed="true"
> stored="true" multiValued="true"/>
> .....
> ....
> <copyField source="title" dest="text" />
> <copyField source="title" dest="titleorder" />
> <copyField source="contributor" dest="text" />
> <copyField source="contributor" dest="contributorfacet" />
> <copyField source="contributor" dest="contributororder" />
> ...
>
> I use for instance contributor for searching, contributorfacet for
> faceting and order for ordering results, but when I try to order using
> contributororder, Solr says that cannot order by a tokenized
> field...(?)
>
> I'm using Solr 1.4 nightly. Is this a bug? I believe that in previous
> versions I have this issue working...
>
> Regards and thanks
> Germán
>
Reply | Threaded
Open this post in threaded view
|

Re: Newbie problem ordering results

germanbio
Sure

<fieldtype name="string" class="solr.StrField" sortMissingLast="true"
omitNorms="true"/>

The strange thing is that I could sort by another fields that is
defined using string, but not by another defined as some tokenized
field and after that copied as string.

I attach the schema.xml for the case is there another error and the
error log says the following

INFO: UnInverted multi-valued field
{field=date,memSize=70356,tindexSize=40,time=381,phase1=381,nTerms=99,bigTerms=5,termInstances=4330,uses=0}
11/08/2009 12:42:31 org.apache.solr.request.UnInvertedField uninvert
INFO: UnInverted multi-valued field
{field=local_medium,memSize=70088,tindexSize=56,time=10,phase1=10,nTerms=30,bigTerms=2,termInstances=2461,uses=0}
11/08/2009 12:42:31 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select
params={rows=20&wt=json&facet.field=contributorfacet&facet.field=subjectfacet&facet.field=provenance&facet.field=local_state&facet.field=date&facet.field=local_medium&facet.limit=15&q=text:fisica&start=0&facet.mincount=1&fl=id,title,contributor,subject,provenance,date,coverage,publisher,score,local_state,local_url&sort=score+desc&facet=true}
hits=312 status=0 QTime=4963
11/08/2009 12:51:46 org.apache.solr.core.SolrCore execute

*** This is the order by date desc that is working OK and it's defined as string

INFO: [] webapp=/solr path=/select
params={rows=20&wt=json&q=text:fisica&start=0&sort=date+desc&fl=id,title,contributor,subject,provenance,date,coverage,publisher,score,local_state,local_url}
hits=312 status=0 QTime=174


11/08/2009 12:52:38 org.apache.solr.common.SolrException log
GRAVE: java.lang.RuntimeException: there are more terms than documents
in field "contributororder", but it's impossible to sort on tokenized
fields
        at org.apache.lucene.search.FieldCacheImpl$8.createValue(FieldCacheImpl.java:518)
        at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:81)
        at org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:491)
        at org.apache.solr.search.MissingLastOrdComparator.setNextReader(MissingStringLastComparatorSource.java:181)
        at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:92)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:242)
        at org.apache.lucene.search.Searcher.search(Searcher.java:173)
        at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988)
        at

Thanks a lot
German


On Tue, Aug 11, 2009 at 1:56 AM, Avlesh Singh<[hidden email]> wrote:

> Can you please post the fieldType definition for the "string" field in your
> schema.xml?
>
> Cheers
> Avlesh
>
> On Tue, Aug 11, 2009 at 9:52 AM, Germán Biozzoli
> <[hidden email]>wrote:
>
>> Hello everybody
>>
>> I have the following (resumed) schema:
>>
>> ....
>>    <field name="title" type="text" indexed="true" stored="true"
>> multiValued="true"/>
>>   <field name="titleorder" type="string" indexed="true" stored="true"
>> multiValued="true"/>
>>   <field name="contributor" type="text" indexed="true" stored="true"
>> multiValued="true"/>
>>   <field name="contributorfacet" type="textFacetN" indexed="true"
>> stored="true" multiValued="true"/>
>>   <field name="contributororder" type="string" indexed="true"
>> stored="true" multiValued="true"/>
>> .....
>> ....
>> <copyField source="title" dest="text" />
>> <copyField source="title" dest="titleorder" />
>> <copyField source="contributor" dest="text" />
>> <copyField source="contributor" dest="contributorfacet" />
>> <copyField source="contributor" dest="contributororder" />
>> ...
>>
>> I use for instance contributor for searching, contributorfacet for
>> faceting and order for ordering results, but when I try to order using
>> contributororder, Solr says that cannot order by a tokenized
>> field...(?)
>>
>> I'm using Solr 1.4 nightly. Is this a bug? I believe that in previous
>> versions I have this issue working...
>>
>> Regards and thanks
>> Germán
>>
>

schema.xml (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Newbie problem ordering results

Avlesh Singh
Ahhh, I should have seen this first. Your "contributororder" field is
multi-valued, you cannot sort on that field.

However the RuntimeException that Solr throws has a misleading error message
- "... but it's impossible to sort on tokenized fields". The field in this
case is untokenized.

Cheers
Avlesh

On Tue, Aug 11, 2009 at 8:33 PM, Germán Biozzoli
<[hidden email]>wrote:

> Sure
>
> <fieldtype name="string" class="solr.StrField" sortMissingLast="true"
> omitNorms="true"/>
>
> The strange thing is that I could sort by another fields that is
> defined using string, but not by another defined as some tokenized
> field and after that copied as string.
>
> I attach the schema.xml for the case is there another error and the
> error log says the following
>
> INFO: UnInverted multi-valued field
>
> {field=date,memSize=70356,tindexSize=40,time=381,phase1=381,nTerms=99,bigTerms=5,termInstances=4330,uses=0}
> 11/08/2009 12:42:31 org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field
>
> {field=local_medium,memSize=70088,tindexSize=56,time=10,phase1=10,nTerms=30,bigTerms=2,termInstances=2461,uses=0}
> 11/08/2009 12:42:31 org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/select
>
> params={rows=20&wt=json&facet.field=contributorfacet&facet.field=subjectfacet&facet.field=provenance&facet.field=local_state&facet.field=date&facet.field=local_medium&facet.limit=15&q=text:fisica&start=0&facet.mincount=1&fl=id,title,contributor,subject,provenance,date,coverage,publisher,score,local_state,local_url&sort=score+desc&facet=true}
> hits=312 status=0 QTime=4963
> 11/08/2009 12:51:46 org.apache.solr.core.SolrCore execute
>
> *** This is the order by date desc that is working OK and it's defined as
> string
>
> INFO: [] webapp=/solr path=/select
>
> params={rows=20&wt=json&q=text:fisica&start=0&sort=date+desc&fl=id,title,contributor,subject,provenance,date,coverage,publisher,score,local_state,local_url}
> hits=312 status=0 QTime=174
>
>
> 11/08/2009 12:52:38 org.apache.solr.common.SolrException log
> GRAVE: java.lang.RuntimeException: there are more terms than documents
> in field "contributororder", but it's impossible to sort on tokenized
> fields
>        at
> org.apache.lucene.search.FieldCacheImpl$8.createValue(FieldCacheImpl.java:518)
>        at
> org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:81)
>        at
> org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:491)
>        at
> org.apache.solr.search.MissingLastOrdComparator.setNextReader(MissingStringLastComparatorSource.java:181)
>        at
> org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:92)
>        at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:242)
>        at org.apache.lucene.search.Searcher.search(Searcher.java:173)
>        at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988)
>        at
>
> Thanks a lot
> German
>
>
> On Tue, Aug 11, 2009 at 1:56 AM, Avlesh Singh<[hidden email]> wrote:
> > Can you please post the fieldType definition for the "string" field in
> your
> > schema.xml?
> >
> > Cheers
> > Avlesh
> >
> > On Tue, Aug 11, 2009 at 9:52 AM, Germán Biozzoli
> > <[hidden email]>wrote:
> >
> >> Hello everybody
> >>
> >> I have the following (resumed) schema:
> >>
> >> ....
> >>    <field name="title" type="text" indexed="true" stored="true"
> >> multiValued="true"/>
> >>   <field name="titleorder" type="string" indexed="true" stored="true"
> >> multiValued="true"/>
> >>   <field name="contributor" type="text" indexed="true" stored="true"
> >> multiValued="true"/>
> >>   <field name="contributorfacet" type="textFacetN" indexed="true"
> >> stored="true" multiValued="true"/>
> >>   <field name="contributororder" type="string" indexed="true"
> >> stored="true" multiValued="true"/>
> >> .....
> >> ....
> >> <copyField source="title" dest="text" />
> >> <copyField source="title" dest="titleorder" />
> >> <copyField source="contributor" dest="text" />
> >> <copyField source="contributor" dest="contributorfacet" />
> >> <copyField source="contributor" dest="contributororder" />
> >> ...
> >>
> >> I use for instance contributor for searching, contributorfacet for
> >> faceting and order for ordering results, but when I try to order using
> >> contributororder, Solr says that cannot order by a tokenized
> >> field...(?)
> >>
> >> I'm using Solr 1.4 nightly. Is this a bug? I believe that in previous
> >> versions I have this issue working...
> >>
> >> Regards and thanks
> >> Germán
> >>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Newbie problem ordering results

Shalin Shekhar Mangar
On Tue, Aug 11, 2009 at 9:56 PM, Avlesh Singh <[hidden email]> wrote:

>
> However the RuntimeException that Solr throws has a misleading error
> message
> - "... but it's impossible to sort on tokenized fields". The field in this
> case is untokenized.
>

The error is thrown by Lucene. Actually, multi valued fields are not very
different from tokenized fields. Multiple values are indexed with their
respective token positions differing by the positionIncrementGap value as
specified in schema. The same is done for tokenized fields but the
difference in positions of the tokens depends on the analysis chain.

--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: Newbie problem ordering results

Avlesh Singh
>
> The error is thrown by Lucene. Actually, multi valued fields are not very
> different from tokenized fields. Multiple values are indexed with their
> respective token positions differing by the positionIncrementGap value as
> specified in schema.
>

I truly understand that. But I guess, that is where solr comes in. The
schema definition can be used to convert the message into a more relevant
one. A proper error message might have done the good here and conveyed the
"reality" to end user.

Cheers
Avlesh

On Wed, Aug 12, 2009 at 11:59 AM, Shalin Shekhar Mangar <
[hidden email]> wrote:

> On Tue, Aug 11, 2009 at 9:56 PM, Avlesh Singh <[hidden email]> wrote:
>
> >
> > However the RuntimeException that Solr throws has a misleading error
> > message
> > - "... but it's impossible to sort on tokenized fields". The field in
> this
> > case is untokenized.
> >
>
> The error is thrown by Lucene. Actually, multi valued fields are not very
> different from tokenized fields. Multiple values are indexed with their
> respective token positions differing by the positionIncrementGap value as
> specified in schema. The same is done for tokenized fields but the
> difference in positions of the tokens depends on the analysis chain.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>