How to use bit fields to narrow a search

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to use bit fields to narrow a search

Siegfried Goeschl
Hi folks,

I'm currently evaluating SOLR to implement fulltext search and within 8
hours I have my content imported and able to benchmark the queries ... :-)

As a beginner with Lucence/SOLR I have a problem where to add my
"special features" - little bit overloaded with "Lucene in Action" and
SOLR over the weekend ...

Some background ...

+) I have 4 millions document indexed
+) each document has 3 long variables (stored but not indexed)
representing a 64 bit mask each
+) I have to filter the Hits based on the bit mask using BIT AND with
application supplied parameters

Any suggestions/ideas where to add this processing within SOLR ...

Thanks in advance

Siegfried Goeschl



Reply | Threaded
Open this post in threaded view
|

Re: How to use bit fields to narrow a search

Yonik Seeley-2
On 6/26/07, Siegfried Goeschl <[hidden email]> wrote:

> Hi folks,
>
> I'm currently evaluating SOLR to implement fulltext search and within 8
> hours I have my content imported and able to benchmark the queries ... :-)
>
> As a beginner with Lucence/SOLR I have a problem where to add my
> "special features" - little bit overloaded with "Lucene in Action" and
> SOLR over the weekend ...
>
> Some background ...
>
> +) I have 4 millions document indexed
> +) each document has 3 long variables (stored but not indexed)
> representing a 64 bit mask each
> +) I have to filter the Hits based on the bit mask using BIT AND with
> application supplied parameters
>
> Any suggestions/ideas where to add this processing within SOLR ...

Due to the nature of an inverted index, it could actually be more
efficient to store the bits separately.  You could also then use Solr
w/o any custom java code.

Index a field called bits, which just contains the bit numbers set,
separated by whitespace.
At query time, use filters on the required bit numbers:
q=foo&fq=bits:12&fq=bits:45

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: How to use bit fields to narrow a search

Siegfried Goeschl
Hi Yonik,

looks intersting - I give it a try ....

Cheers,

Siegfried Goeschl

Yonik Seeley wrote:

> On 6/26/07, Siegfried Goeschl <[hidden email]> wrote:
>> Hi folks,
>>
>> I'm currently evaluating SOLR to implement fulltext search and within 8
>> hours I have my content imported and able to benchmark the queries ...
>> :-)
>>
>> As a beginner with Lucence/SOLR I have a problem where to add my
>> "special features" - little bit overloaded with "Lucene in Action" and
>> SOLR over the weekend ...
>>
>> Some background ...
>>
>> +) I have 4 millions document indexed
>> +) each document has 3 long variables (stored but not indexed)
>> representing a 64 bit mask each
>> +) I have to filter the Hits based on the bit mask using BIT AND with
>> application supplied parameters
>>
>> Any suggestions/ideas where to add this processing within SOLR ...
>
> Due to the nature of an inverted index, it could actually be more
> efficient to store the bits separately.  You could also then use Solr
> w/o any custom java code.
>
> Index a field called bits, which just contains the bit numbers set,
> separated by whitespace.
> At query time, use filters on the required bit numbers:
> q=foo&fq=bits:12&fq=bits:45
>
> -Yonik
>
>