How to send an OpenBitSet object from Solr server?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

How to send an OpenBitSet object from Solr server?

Satish Talim
We have a need to query and fetch millions of document ids from a Solr 3.3
index and convert the same to a BitSet. To speed things up, we want to
convert these document ids into OpenBitSet on the server side, put them into
the response object and read the same on the client side.

To achieve this, we wrote our own RequestHandler and overwrote
the handleRequest method. Using this RequestHandler we do get the response
object but when we try to fetch the OpenBitSet we get an error -

Exception in thread "main" java.lang.ClassCastException: java.lang.String
cannot be cast to org.apache.lucene.util.OpenBitSet

The documentation at -
http://lucene.apache.org/solr/api/org/apache/solr/response/SolrQueryResponse.html

says that "Other data types may be added to the SolrQueryResponse, but there
is no guarantee that QueryResponseWriters will be able to deal with
unexpected types."

Is there a work-around wherein I can send an OpenBitSet object?

Satish
Reply | Threaded
Open this post in threaded view
|

Re: How to send an OpenBitSet object from Solr server?

Federico Fissore
Satish Talim, il 30/08/2011 05:42, ha scritto:
[...]
>
> Is there a work-around wherein I can send an OpenBitSet object?
>

JavaBinCodec (used by default by solr) supports writing arrays. you can
"getBits()" from openbitset and throw them into the binary response

federico
Reply | Threaded
Open this post in threaded view
|

Re: How to send an OpenBitSet object from Solr server?

Federico Fissore-2
In reply to this post by Satish Talim
Satish Talim, il 30/08/2011 05:42, ha scritto:
[...]
>
> Is there a work-around wherein I can send an OpenBitSet object?
>

JavaBinCodec (used by default by solr) supports writing arrays. you can
"getBits()" from openbitset and throw them into the binary response

federico
Reply | Threaded
Open this post in threaded view
|

Re: How to send an OpenBitSet object from Solr server?

Satish Talim
In reply to this post by Federico Fissore
But how to throw? As a stream of bits?

Satish

On Tue, Aug 30, 2011 at 5:39 PM, Federico Fissore <[hidden email]>wrote:

> Satish Talim, il 30/08/2011 05:42, ha scritto:
> [...]
>
>
>> Is there a work-around wherein I can send an OpenBitSet object?
>>
>>
> JavaBinCodec (used by default by solr) supports writing arrays. you can
> "getBits()" from openbitset and throw them into the binary response
>
> federico
>
Reply | Threaded
Open this post in threaded view
|

Re: How to send an OpenBitSet object from Solr server?

Federico Fissore-2
Satish Talim, il 30/08/2011 14:22, ha scritto:
> But how to throw? As a stream of bits?
>

getBits() return a long[]
add a long[] part to your response

rb.rsp.add("long_array", obs.getBits())

federico
Reply | Threaded
Open this post in threaded view
|

Re: How to send an OpenBitSet object from Solr server?

Chris Hostetter-3
In reply to this post by Satish Talim

: We have a need to query and fetch millions of document ids from a Solr 3.3
: index and convert the same to a BitSet. To speed things up, we want to
: convert these document ids into OpenBitSet on the server side, put them into
: the response object and read the same on the client side.

This smells like an XY Problem ... what do you intend to do with this
BitSet on the client side?  the lucene doc ids are meaningless outside of
hte server, and for any given doc, the id could change from one request to
the next -- so how would having this data on the clinet be of any use to
you?

https://people.apache.org/~hossman/#xyproblem
XY Problem

Your question appears to be an "XY Problem" ... that is: you are dealing
with "X", you are assuming "Y" will help you, and you are asking about "Y"
without giving more details about the "X" so that we can understand the
full issue.  Perhaps the best solution doesn't involve "Y" at all?
See Also: http://www.perlmonks.org/index.pl?node_id=542341


-Hoss
Reply | Threaded
Open this post in threaded view
|

Re: How to send an OpenBitSet object from Solr server?

Satish Talim
I was not referring to Lucene's doc ids but the doc numbers (unique key)

Satish



On Tue, Aug 30, 2011 at 9:28 PM, Chris Hostetter
<[hidden email]>wrote:

>
> : We have a need to query and fetch millions of document ids from a Solr
> 3.3
> : index and convert the same to a BitSet. To speed things up, we want to
> : convert these document ids into OpenBitSet on the server side, put them
> into
> : the response object and read the same on the client side.
>
> This smells like an XY Problem ... what do you intend to do with this
> BitSet on the client side?  the lucene doc ids are meaningless outside of
> hte server, and for any given doc, the id could change from one request to
> the next -- so how would having this data on the clinet be of any use to
> you?
>
> https://people.apache.org/~hossman/#xyproblem
> XY Problem
>
> Your question appears to be an "XY Problem" ... that is: you are dealing
> with "X", you are assuming "Y" will help you, and you are asking about "Y"
> without giving more details about the "X" so that we can understand the
> full issue.  Perhaps the best solution doesn't involve "Y" at all?
> See Also: http://www.perlmonks.org/index.pl?node_id=542341
>
>
> -Hoss
>
Reply | Threaded
Open this post in threaded view
|

Re: How to send an OpenBitSet object from Solr server?

Chris Hostetter-3

: I was not referring to Lucene's doc ids but the doc numbers (unique key)

Uh ... ok.  this is why i asked you waht you were planning on doing with
the OpenBitSet -- it's just bits, indicating hte offsets of the documents
in the total index.  having access to a copy of that on the lcient side
isn't going to give you any info about the value of the uniqueKey field.

Like i said ... XY Problem ... please tell us what you overall goal is.  
why do you want the uniqueKey values for every matching doc in your remove
client?  What would you do with the OpenBitSet from solr if you had a copy
of it?

: > : We have a need to query and fetch millions of document ids from a Solr
: > 3.3
: > : index and convert the same to a BitSet. To speed things up, we want to
: > : convert these document ids into OpenBitSet on the server side, put them
: > into
: > : the response object and read the same on the client side.
: >
: > This smells like an XY Problem ... what do you intend to do with this
: > BitSet on the client side?  the lucene doc ids are meaningless outside of
: > hte server, and for any given doc, the id could change from one request to
: > the next -- so how would having this data on the clinet be of any use to
: > you?
: >
: > https://people.apache.org/~hossman/#xyproblem
: > XY Problem
: >
: > Your question appears to be an "XY Problem" ... that is: you are dealing
: > with "X", you are assuming "Y" will help you, and you are asking about "Y"
: > without giving more details about the "X" so that we can understand the
: > full issue.  Perhaps the best solution doesn't involve "Y" at all?
: > See Also: http://www.perlmonks.org/index.pl?node_id=542341


-Hoss