limit on number of values in a filter query?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

limit on number of values in a filter query?

Phillip Farber
I use a filter query (fq) parameter in my requests to limit the select
response to a subset of all document ids.  I'm getting a Solr exception
when the number of values in the fq approaches 1000.   I get the
following response from Solr:

_header=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
... <<several 100's of ids>> ... +851832976+8536687}{}
_buffer=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
... <<several 100's of ids>> ... +851832976+8536687}{}
2008-04-03 10:42:54.005::WARN:  handle failed
java.io.IOException: FULL
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:274)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
        at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
        at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)


The id field is currently a string.  Would it help to make it a long
int? Is there some internal limit on the number of values in a fq or is
there a configuration parameter that will allow me to increase this number?

Thanks,

Phil
Reply | Threaded
Open this post in threaded view
|

Re: limit on number of values in a filter query?

Yonik Seeley-2
1000 filters, wow!

Perhaps you hit a jetty limit on the size of a GET request or
something?  Perhaps try POST?

-Yonik

On Thu, Apr 3, 2008 at 11:03 AM, Phillip Farber <[hidden email]> wrote:

> I use a filter query (fq) parameter in my requests to limit the select
> response to a subset of all document ids.  I'm getting a Solr exception when
> the number of values in the fq approaches 1000.   I get the following
> response from Solr:
>
>
> _header=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
>  ... <<several 100's of ids>> ... +851832976+8536687}{}
>
> _buffer=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
>  ... <<several 100's of ids>> ... +851832976+8536687}{}
>  2008-04-03 10:42:54.005::WARN:  handle failed
>  java.io.IOException: FULL
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:274)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
>         at
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
>         at
> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>
>
>  The id field is currently a string.  Would it help to make it a long int?
> Is there some internal limit on the number of values in a fq or is there a
> configuration parameter that will allow me to increase this number?
>
>  Thanks,
>
>  Phil
>
Reply | Threaded
Open this post in threaded view
|

Re: limit on number of values in a filter query?

Phillip Farber
It is a POST so it's not a jetty limit.

The mysterious "4096" numbers in the string that begins
_header=[6518349,32231686,m=4,g=4096,p=4096,c=4096] ... and
"java.io.IOException: FULL" in the server traceback made me think that
the size of a static buffer is being exceeded. Since the exception is
I/O could it be an I/O buffer limitation?

The fq values represent the document ids in a  collection.  Is there a
workaround to allow longer fq lists?  I guess one way is to limit
collection size but this is not desirable for application reasons.

Phil

Yonik Seeley wrote:

> 1000 filters, wow!
>
> Perhaps you hit a jetty limit on the size of a GET request or
> something?  Perhaps try POST?
>
> -Yonik
>
> On Thu, Apr 3, 2008 at 11:03 AM, Phillip Farber <[hidden email]> wrote:
>> I use a filter query (fq) parameter in my requests to limit the select
>> response to a subset of all document ids.  I'm getting a Solr exception when
>> the number of values in the fq approaches 1000.   I get the following
>> response from Solr:
>>
>>
>> _header=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
>>  ... <<several 100's of ids>> ... +851832976+8536687}{}
>>
>> _buffer=[6518349,32231686,m=4,g=4096,p=4096,c=4096]={/solr/select?q=ship&fl=id,score&fq=id:(883108+1355308+
>>  ... <<several 100's of ids>> ... +851832976+8536687}{}
>>  2008-04-03 10:42:54.005::WARN:  handle failed
>>  java.io.IOException: FULL
>>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:274)
>>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:202)
>>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
>>         at
>> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
>>         at
>> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>>
>>
>>  The id field is currently a string.  Would it help to make it a long int?
>> Is there some internal limit on the number of values in a fq or is there a
>> configuration parameter that will allow me to increase this number?
>>
>>  Thanks,
>>
>>  Phil
>>