Code for getting distinct facet counts across shards(Distributed Process).

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Code for getting distinct facet counts across shards(Distributed Process).

Rajinimaski
 In solr 1.4.1, for getting "distinct facet terms count" across shards,



The piece of code added for getting count of distinct facet terms across
distributed process is as followed:





Class: facetcomponent.java

Function: -- finishStage(ResponseBuilder rb)



  for (DistribFieldFacet dff : fi.facets.values()) {

        //just after this line of code

         else { // TODO: log error or throw exception?

         counts = dff.getLexSorted();



            int namedistint = 0;


namedistint=rb.req.getParams().getFieldInt(dff.getKey().toString(),FacetParams.FACET_NAMEDISTINCT,0);

            if (namedistint  == 0)

                        facet_fields.add(dff.getKey(), fieldCounts);



            if (namedistint  == 1)

                        facet_fields.add("numfacetTerms", counts.length);




             if (namedistint  == 2) {

                         NamedList resCount = new NamedList();


                         resCount.add("numfacetTerms", counts.length);


                         resCount.add("counts", fieldCounts);

                        facet_fields.add(dff.getKey(), resCount);

             }




Is this flow correct ?  I have worked with few test cases and it has worked
fine.  but i want to know if there are any bugs that can creep in here?  (My
concern is this piece of code should not effect the rest of logic)




*Code flow with comments for reference:*


 Function : --   finishStage(ResponseBuilder rb)



          //in this for loop ,

         for (DistribFieldFacet dff : fi.facets.values()) {



        //just after this line of code

         else { // TODO: log error or throw exception?

         counts = dff.getLexSorted();



         int namedistint = 0;  //default



//get the value of facet.numterms from the input query


namedistint=rb.req.getParams().getFieldInt(dff.getKey().toString(),FacetParams.FACET_NAMEDISTINCT,0);



// based on the value for  facet.numterms==0 or 1 or 2  , if conditions



//Get only facet field counts

            if (namedistint  == 0)

            {

                        facet_fields.add(dff.getKey(), fieldCounts);


            }



//get only distinct facet term count

            if (namedistint  == 1)

            {

                        facet_fields.add("numfacetTerms", counts.length);


            }



//get facet field count and distinct term count.

             if (namedistint  == 2) {

                         NamedList resCount = new NamedList();


                         resCount.add("numfacetTerms", counts.length);


                         resCount.add("counts", fieldCounts);

                        facet_fields.add(dff.getKey(), resCount);

             }





Regards,

Rajani





On Fri, May 27, 2011 at 1:14 PM, rajini maski <[hidden email]> wrote:

>  No such issues . Successfully integrated with 1.4.1 and it works across
> single index.
>
> for f.2.facet.numFacetTerms=1  parameter it will give the distinct count
> result
>
> for f.2.facet.numFacetTerms=2 parameter  it will give counts as well as
> results for facets.
>
> But this is working only across single index not distributed process. The
> conditions you have added in simple facet.java- "if namedistinct count ==int
> " ( 0, 1 and 2 condtions).. Should it be added in distributed process
> function to enable it work across shards?
>
> Rajani
>
>
>
> On Fri, May 27, 2011 at 12:33 PM, Bill Bell <[hidden email]> wrote:
>
>> I am pretty sure it does not yet support distributed shards..
>>
>> But the patch was written for 4.0... So there might be issues with running
>> it on 1.4.1.
>>
>> On 5/26/11 11:08 PM, "rajini maski" <[hidden email]> wrote:
>>
>> >     The patch solr 2242 for getting count of distinct facet terms
>> doesn't
>> >work for distributedProcess
>> >
>> >(https://issues.apache.org/jira/browse/SOLR-2242)
>> >
>> >The error log says
>> >
>> > HTTP ERROR 500
>> >Problem accessing /solr/select. Reason:
>> >
>> >    For input string: "numFacetTerms"
>> >
>> >java.lang.NumberFormatException: For input string: "numFacetTerms"
>> >at
>>
>> >java.lang.NumberFormatException.forInputString(NumberFormatException.java:
>> >48)
>> >at java.lang.Long.parseLong(Long.java:403)
>> >at java.lang.Long.parseLong(Long.java:461)
>> >at org.apache.solr.schema.TrieField.readableToIndexed(TrieField.java:331)
>> >at org.apache.solr.schema.TrieField.toInternal(TrieField.java:344)
>> >at
>>
>> >org.apache.solr.handler.component.FacetComponent$DistribFieldFacet.add(Fac
>> >etComponent.java:619)
>> >at
>>
>> >org.apache.solr.handler.component.FacetComponent.countFacets(FacetComponen
>> >t.java:265)
>> >at
>>
>> >org.apache.solr.handler.component.FacetComponent.handleResponses(FacetComp
>> >onent.java:235)
>> >at
>>
>> >org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHa
>> >ndler.java:290)
>> >at
>>
>> >org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBas
>> >e.java:131)
>> >at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>> >at
>>
>> >org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java
>> >:338)
>> >at
>>
>> >org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.jav
>> >a:241)
>> >at
>>
>> >org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandl
>> >er.java:1212)
>> >at
>> >org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>> >at
>>
>> >org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216
>> >)
>> >at
>> >org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>> >at
>> >org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> >at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>> >at
>> >org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> >at org.mortbay.jetty.Server.handle(Server.java:326)
>> >at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>> >at
>>
>> >org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnect
>> >ion.java:928)
>> >at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>> >at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>> >at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>> >at
>>
>> >org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:41
>> >0)
>> >at
>>
>> >org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:5
>> >82)
>> >
>> >
>> >The query I passed :
>> >
>> http://localhost:8983/solr/select?q=*:*&facet=true&facet.field=2&facet.fie
>>
>> >ld=648&facet.mincount=1&facet.limit=-1&f.2.facet.numFacetTerms=1&rows=0&sh
>> >ards=localhost:8983/solr,localhost:8985/solrtwo
>> >
>> >Anyone can suggest me the changes i need to make to enable the same
>> >funcionality for shards?
>> >
>> >When i do it across single core.. I get the correct results. I have
>> >applied
>> >the solr 2242 patch in solr1.4.1
>> >
>> >Awaiting for reply
>> >
>> >Regards,
>> >Rajani
>>
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Code for getting distinct facet counts across shards(Distributed Process).

Billnbell
I have coded and tested this and it appears to work.

Are you having any problems?

On 6/9/11 12:35 AM, "rajini maski" <[hidden email]> wrote:

> In solr 1.4.1, for getting "distinct facet terms count" across shards,
>
>
>
>The piece of code added for getting count of distinct facet terms across
>distributed process is as followed:
>
>
>
>
>
>Class: facetcomponent.java
>
>Function: -- finishStage(ResponseBuilder rb)
>
>
>
>  for (DistribFieldFacet dff : fi.facets.values()) {
>
>        //just after this line of code
>
>         else { // TODO: log error or throw exception?
>
>         counts = dff.getLexSorted();
>
>
>
>            int namedistint = 0;
>
>
>namedistint=rb.req.getParams().getFieldInt(dff.getKey().toString(),FacetPa
>rams.FACET_NAMEDISTINCT,0);
>
>            if (namedistint  == 0)
>
>                        facet_fields.add(dff.getKey(), fieldCounts);
>
>
>
>            if (namedistint  == 1)
>
>                        facet_fields.add("numfacetTerms", counts.length);
>
>
>
>
>             if (namedistint  == 2) {
>
>                         NamedList resCount = new NamedList();
>
>
>                         resCount.add("numfacetTerms", counts.length);
>
>
>                         resCount.add("counts", fieldCounts);
>
>                        facet_fields.add(dff.getKey(), resCount);
>
>             }
>
>
>
>
>Is this flow correct ?  I have worked with few test cases and it has
>worked
>fine.  but i want to know if there are any bugs that can creep in here?
>(My
>concern is this piece of code should not effect the rest of logic)
>
>
>
>
>*Code flow with comments for reference:*
>
>
> Function : --   finishStage(ResponseBuilder rb)
>
>
>
>          //in this for loop ,
>
>         for (DistribFieldFacet dff : fi.facets.values()) {
>
>
>
>        //just after this line of code
>
>         else { // TODO: log error or throw exception?
>
>         counts = dff.getLexSorted();
>
>
>
>         int namedistint = 0;  //default
>
>
>
>//get the value of facet.numterms from the input query
>
>
>namedistint=rb.req.getParams().getFieldInt(dff.getKey().toString(),FacetPa
>rams.FACET_NAMEDISTINCT,0);
>
>
>
>// based on the value for  facet.numterms==0 or 1 or 2  , if conditions
>
>
>
>//Get only facet field counts
>
>            if (namedistint  == 0)
>
>            {
>
>                        facet_fields.add(dff.getKey(), fieldCounts);
>
>
>            }
>
>
>
>//get only distinct facet term count
>
>            if (namedistint  == 1)
>
>            {
>
>                        facet_fields.add("numfacetTerms", counts.length);
>
>
>            }
>
>
>
>//get facet field count and distinct term count.
>
>             if (namedistint  == 2) {
>
>                         NamedList resCount = new NamedList();
>
>
>                         resCount.add("numfacetTerms", counts.length);
>
>
>                         resCount.add("counts", fieldCounts);
>
>                        facet_fields.add(dff.getKey(), resCount);
>
>             }
>
>
>
>
>
>Regards,
>
>Rajani
>
>
>
>
>
>On Fri, May 27, 2011 at 1:14 PM, rajini maski <[hidden email]>
>wrote:
>
>>  No such issues . Successfully integrated with 1.4.1 and it works across
>> single index.
>>
>> for f.2.facet.numFacetTerms=1  parameter it will give the distinct count
>> result
>>
>> for f.2.facet.numFacetTerms=2 parameter  it will give counts as well as
>> results for facets.
>>
>> But this is working only across single index not distributed process.
>>The
>> conditions you have added in simple facet.java- "if namedistinct count
>>==int
>> " ( 0, 1 and 2 condtions).. Should it be added in distributed process
>> function to enable it work across shards?
>>
>> Rajani
>>
>>
>>
>> On Fri, May 27, 2011 at 12:33 PM, Bill Bell <[hidden email]> wrote:
>>
>>> I am pretty sure it does not yet support distributed shards..
>>>
>>> But the patch was written for 4.0... So there might be issues with
>>>running
>>> it on 1.4.1.
>>>
>>> On 5/26/11 11:08 PM, "rajini maski" <[hidden email]> wrote:
>>>
>>> >     The patch solr 2242 for getting count of distinct facet terms
>>> doesn't
>>> >work for distributedProcess
>>> >
>>> >(https://issues.apache.org/jira/browse/SOLR-2242)
>>> >
>>> >The error log says
>>> >
>>> > HTTP ERROR 500
>>> >Problem accessing /solr/select. Reason:
>>> >
>>> >    For input string: "numFacetTerms"
>>> >
>>> >java.lang.NumberFormatException: For input string: "numFacetTerms"
>>> >at
>>>
>>>
>>>>java.lang.NumberFormatException.forInputString(NumberFormatException.ja
>>>>va:
>>> >48)
>>> >at java.lang.Long.parseLong(Long.java:403)
>>> >at java.lang.Long.parseLong(Long.java:461)
>>> >at
>>>org.apache.solr.schema.TrieField.readableToIndexed(TrieField.java:331)
>>> >at org.apache.solr.schema.TrieField.toInternal(TrieField.java:344)
>>> >at
>>>
>>>
>>>>org.apache.solr.handler.component.FacetComponent$DistribFieldFacet.add(
>>>>Fac
>>> >etComponent.java:619)
>>> >at
>>>
>>>
>>>>org.apache.solr.handler.component.FacetComponent.countFacets(FacetCompo
>>>>nen
>>> >t.java:265)
>>> >at
>>>
>>>
>>>>org.apache.solr.handler.component.FacetComponent.handleResponses(FacetC
>>>>omp
>>> >onent.java:235)
>>> >at
>>>
>>>
>>>>org.apache.solr.handler.component.SearchHandler.handleRequestBody(Searc
>>>>hHa
>>> >ndler.java:290)
>>> >at
>>>
>>>
>>>>org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandler
>>>>Bas
>>> >e.java:131)
>>> >at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>> >at
>>>
>>>
>>>>org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.j
>>>>ava
>>> >:338)
>>> >at
>>>
>>>
>>>>org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.
>>>>jav
>>> >a:241)
>>> >at
>>>
>>>
>>>>org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHa
>>>>ndl
>>> >er.java:1212)
>>> >at
>>>
>>>>org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399
>>>>)
>>> >at
>>>
>>>
>>>>org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
>>>>216
>>> >)
>>> >at
>>>
>>>>org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182
>>>>)
>>> >at
>>>
>>>>org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766
>>>>)
>>> >at
>>>org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>> >at
>>>
>>>>org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152
>>>>)
>>> >at org.mortbay.jetty.Server.handle(Server.java:326)
>>> >at
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>> >at
>>>
>>>
>>>>org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConn
>>>>ect
>>> >ion.java:928)
>>> >at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>> >at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>> >at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>> >at
>>>
>>>
>>>>org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java
>>>>:41
>>> >0)
>>> >at
>>>
>>>
>>>>org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.jav
>>>>a:5
>>> >82)
>>> >
>>> >
>>> >The query I passed :
>>> >
>>>
>>>http://localhost:8983/solr/select?q=*:*&facet=true&facet.field=2&facet.f
>>>ie
>>>
>>>
>>>>ld=648&facet.mincount=1&facet.limit=-1&f.2.facet.numFacetTerms=1&rows=0
>>>>&sh
>>> >ards=localhost:8983/solr,localhost:8985/solrtwo
>>> >
>>> >Anyone can suggest me the changes i need to make to enable the same
>>> >funcionality for shards?
>>> >
>>> >When i do it across single core.. I get the correct results. I have
>>> >applied
>>> >the solr 2242 patch in solr1.4.1
>>> >
>>> >Awaiting for reply
>>> >
>>> >Regards,
>>> >Rajani
>>>
>>>
>>>
>>