Upgrade SOLR version - facets perfomance regression

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgrade SOLR version - facets perfomance regression

SOLR4189
Before few months we upgraded our SOLR in production from 4.10.1 to 5.4.1. And at once we noticed perfomance regressions. After searchings in internet we found SOLR-8096 issue. So we had to downgrade SOLR version to 4.10.1.
We want to be in the latest Solr version (5 or 6), but 
  1. Facets perfomance is very important for us.
  2. We can't use new Facet API, because our client application uses blacklight-4.5.0 (without option for upgrade).

What can we do in this situations? If the truth that new Facet API have to return perfomance of SOLR-4.10.1? Maybe does some convertor from new api to old api exist?
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

Alessandro Benedetti
Hi Solr,
I admit the issue you mentioned has not been transparently solved, and indeed you would need to explicitly use the method=uif to get 4.10.1 behavior.

This is valid if you were using  fc/fcs approaches with high cardinality fields.

In the case you facet method is enum ( Term Enumeration), the issue has been transparently solved (
https://issues.apache.org/jira/browse/SOLR-9176 )

Cheers
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

Bhawna Asnani
Hi,
I am experiencing a similar issue. We have tried method uif but that didn't
help much. There is still some performance degradation.
Perhaps some underlying changes in the lucene version its using.

Will switching to JSON facet API help in this case? We have 5 nodes/single
shard in our production setup.

On Tue, Jan 24, 2017 at 4:34 AM, alessandro.benedetti <[hidden email]
> wrote:

> Hi Solr,
> I admit the issue you mentioned has not been transparently solved, and
> indeed you would need to explicitly use the method=uif to get 4.10.1
> behavior.
>
> This is valid if you were using  fc/fcs approaches with high cardinality
> fields.
>
> In the case you facet method is enum ( Term Enumeration), the issue has
> been
> transparently solved (
> https://issues.apache.org/jira/browse/SOLR-9176 )
>
> Cheers
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Upgrade-SOLR-version-facets-perfomance-
> regression-tp4315027p4315512.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

Billnbell
Are you using docvalues ? Try that it might help.

Bill Bell
Sent from mobile


> On Jan 26, 2017, at 10:38 AM, Bhawna Asnani <[hidden email]> wrote:
>
> Hi,
> I am experiencing a similar issue. We have tried method uif but that didn't
> help much. There is still some performance degradation.
> Perhaps some underlying changes in the lucene version its using.
>
> Will switching to JSON facet API help in this case? We have 5 nodes/single
> shard in our production setup.
>
> On Tue, Jan 24, 2017 at 4:34 AM, alessandro.benedetti <[hidden email]
>> wrote:
>
>> Hi Solr,
>> I admit the issue you mentioned has not been transparently solved, and
>> indeed you would need to explicitly use the method=uif to get 4.10.1
>> behavior.
>>
>> This is valid if you were using  fc/fcs approaches with high cardinality
>> fields.
>>
>> In the case you facet method is enum ( Term Enumeration), the issue has
>> been
>> transparently solved (
>> https://issues.apache.org/jira/browse/SOLR-9176 )
>>
>> Cheers
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.
>> nabble.com/Upgrade-SOLR-version-facets-perfomance-
>> regression-tp4315027p4315512.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

Alessandro Benedetti
Which kind of field are you faceting on ?
Cardinality ?
Field Type ?
Doc Valued ?
Which facet algorithm are you using ?
Which facet parameters ?

Cheers
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

SOLR4189
Method uif: we used it also but it didn't help
Cardinality: high
Field Type: string, tdate
DocValued: yes, for all facet fields
Facet Method: fc (but tried fcs and enum)
Facet Params:
  1. Mincount = 1
  2. Limit = 11
  3. Threads = -1
  4. Query (on tdate field for each query)

My question: if Json Facet Api is good enough and if exists some converter from old facet api to new facet api?
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

Alessandro Benedetti
Hi,
Reading in here : https://issues.apache.org/jira/browse/SOLR-8466
It seems that uif has been introduced in Solr 5.5 ( you were using 5.4.1 , were't you?)
Furthermore, I would recommend to check if your field is/isn't multi-valued ( that could affect as well)
It is weird you don't get any benefit from docValues though...

Cheers
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

SOLR4189

After failing with SOLR-5.4.1, we checked SOLR-5.5.2 also.
Most of our facet fields are multivalued.
I see that Json Facet API is experimental API and I can't find how to use it (I am not speaking about syntax, I need to know how to use it from the point of view of configurations and jars).
I came to the conclusion that inorder to upgrade SOLR, I need to give up facets or to leave in SOLR-4.10.1. Is it right?


alessandro.benedetti wrote
Hi,
Reading in here : https://issues.apache.org/jira/browse/SOLR-8466
It seems that uif has been introduced in Solr 5.5 ( you were using 5.4.1 ,
were't you?)
Furthermore, I would recommend to check if your field is/isn't multi-valued
( that could affect as well)
It is weird you don't get any benefit from docValues though...

Cheers
Quoted from:
http://lucene.472066.n3.nabble.com/Upgrade-SOLR-version-facets-perfomance-regression-tp4315027p4317781.html
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

Alessandro Benedetti
You should not give up, starting from 5.5 uif should be there helping and starting from 6.1 term enum is fixed as well.

There is still the logic of forcing a facet approach depending on your config and request parameters [1].
It is not expected you don't solve your problem with docValues either ...

Using the json Faceting approach anyway will not imply any particular configuration or jars.
it is in Solr out of the box, starting from 5.
It is experimental and you may check yourself this APIs changed and improved a lot in late 5 and 6 :)

Cheers


[1] org/apache/solr/request/SimpleFacets.java:578 : selectFacetMethod
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

SOLR4189
But I can't run Json Facet API. I checked on SOLR-5.4.1.
If I write:
localhost:9001/solr/Test1_shard1_replica1/myHandler/q=*:*&rows=5&fl=*&wt=json&facet=true&facet.field=someField
It works fine. But if I write:
localhost:9001/solr/Test1_shard1_replica1/myHandler/q=*:*&rows=5&fl=*&wt=json&json.facet={field:someField}
It doesn't work.
Are you sure that it is built-in? If it is built-in, why I can't find explanation about it in reference guid?
Thank you for your help.
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

Tom Evans
On Tue, Jan 31, 2017 at 5:49 AM, SOLR4189 <[hidden email]> wrote:
> But I can't run Json Facet API. I checked on SOLR-5.4.1.
> If I write:
> localhost:9001/solr/Test1_shard1_replica1/myHandler/q=*:*&rows=5&fl=*&wt=json&facet=true&facet.field=someField
> It works fine. But if I write:
> localhost:9001/solr/Test1_shard1_replica1/myHandler/q=*:*&rows=5&fl=*&wt=json&json.facet={field:someField}
> It doesn't work.
> Are you sure that it is built-in? If it is built-in, why I can't find
> explanation about it in reference guid?
> Thank you for your help.

You do have to follow the correct syntax:

  json.facet={name_of_facet_in_output:{type:terms, field:name_of_field}}

It is documented in confluence:

https://cwiki.apache.org/confluence/display/solr/Faceted+Search

Also by yonik:

http://yonik.com/json-facet-api/

Cheers

Tom

Cheers

Tom
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

SOLR4189
Tom, I already tried this syntax (and many another different syntax). It still doesn't work. Are you sure that no need to change facet name in request handler to something else?
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

SOLR4189
I noticed if I don't write list of components in request handler it works fine, but if I add something like
<arr name="components">
  <str>query</str>
  <str>facet</str>
</arr> 
Facets don't work...
How can you explian it?
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

Alessandro Benedetti
The reason must to be found in the default list of components :

protected List<String> getDefaultComponents()
  {
    ArrayList<String> names = new ArrayList<>(8);
    names.add( QueryComponent.COMPONENT_NAME );
    names.add( FacetComponent.COMPONENT_NAME );
    names.add( FacetModule.COMPONENT_NAME );
    names.add( MoreLikeThisComponent.COMPONENT_NAME );
    ...
    return names;
  }

Based on the Solr doc[1] you were overriding the list of components, forgetting the component required for Json faceting (facet module).
To have Json facets you don't need only the facet component but also the facet module.

Furthermore what is your reason to override the default search components involved in your request handler ?

Cheers

[1] https://cwiki.apache.org/confluence/display/solr/RequestHandlers+and+SearchComponents+in+SolrConfig
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

SOLR4189
Alessandro, it helped! Thank you.
But I asked which changes do we do in configuration and I think these things must be documented in the reference guide.
About your question, first of all I don't override default componets. Second of all, I add my own components and for many reasons (For example, I checked permissions before each query with my own component).
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

SOLR4189
And still I have a question:
Is there some convertor from the legacy api to the new API?
Or a search component that converts from legacy api to json facet api?

I explained why I need it in my first post.

Thank you
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

Alessandro Benedetti
What I meant is that :
"Components
If you define components, the default components (see above) will not be executed, and first-components and last-components are disallowed:"

Anyway it is documented in the Confluence page.
If you don't override the default components, json facets will be there.

Related the converter, i don't think there's anything out there.
You will need to define the json facets as json.

Cheers
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade SOLR version - facets perfomance regression

SOLR4189
I finished to write FacetConverter, but I have a question:
How do I config facet.threads parameter in Json Facet Api?
 

I didn't find right syntax in the Confluence page.