Multiple indexes?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple indexes?

Matthew Runo
Hey there-

I was wondering if the following was possible, and, if so, how to set  
it up...

I want to index two different types of data, and have them searchable  
from the same interface.

For example, a group of products, with size, color, price, etc info.
And a group of brands, with brand, genre, brand description, etc info

So, the info does overlap some. But a lot of the fields for each  
"type" don't matter to the other. Is there a way to set up two  
different schema so that both types may be indexed with relative ease?

+--------------------------------------------------------+
  | Matthew Runo
  | Zappos Development
  | [hidden email]
  | 702-943-7833
+--------------------------------------------------------+


Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes?

gonzoprosperity
Why not just store an additional "object_type" field which
differentiates between the actual type of data you are looking for?

So if you're looking for some shoes:

(size:8 AND color:'blue') AND object_type:'shoe'

Or if you're searching on brands

(genre:'skater' AND brand_desc:'skater boy') AND object_type:'brand'

I apologize if I misunderstood your question.

/cody

On 4/19/07, Matthew Runo <[hidden email]> wrote:

> Hey there-
>
> I was wondering if the following was possible, and, if so, how to set
> it up...
>
> I want to index two different types of data, and have them searchable
> from the same interface.
>
> For example, a group of products, with size, color, price, etc info.
> And a group of brands, with brand, genre, brand description, etc info
>
> So, the info does overlap some. But a lot of the fields for each
> "type" don't matter to the other. Is there a way to set up two
> different schema so that both types may be indexed with relative ease?
>
> +--------------------------------------------------------+
>   | Matthew Runo
>   | Zappos Development
>   | [hidden email]
>   | 702-943-7833
> +--------------------------------------------------------+
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes?

Henrib-2
In reply to this post by Matthew Runo
You can not have more than one Solr core per application (to be precise, per class-loader since there are a few statics).
One way is thus to have 2 webapps - when & if indexes do not have the same lifetime/radically different schema/etc.
However, the common wisdom is that you usually dont really need different indexes (I discussed about this last week).

If you really are in desperate need of multiple cores, in the 'Multiple Solr Cores' thread, you'll find (early state) patches that allow just that...

Cheers
Henri

Matthew Runo wrote
Hey there-

I was wondering if the following was possible, and, if so, how to set  
it up...

I want to index two different types of data, and have them searchable  
from the same interface.

For example, a group of products, with size, color, price, etc info.
And a group of brands, with brand, genre, brand description, etc info

So, the info does overlap some. But a lot of the fields for each  
"type" don't matter to the other. Is there a way to set up two  
different schema so that both types may be indexed with relative ease?

+--------------------------------------------------------+
  | Matthew Runo
  | Zappos Development
  | mruno@zappos.com
  | 702-943-7833
+--------------------------------------------------------+

Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes?

Matthew Runo
In reply to this post by gonzoprosperity
Ah hah! This appears to be what I'm interested in doing.

I'll have to read up on object_types.

+--------------------------------------------------------+
  | Matthew Runo
  | Zappos Development
  | [hidden email]
  | 702-943-7833
+--------------------------------------------------------+


On Apr 19, 2007, at 10:04 AM, Cody Caughlan wrote:

> Why not just store an additional "object_type" field which
> differentiates between the actual type of data you are looking for?
>
> So if you're looking for some shoes:
>
> (size:8 AND color:'blue') AND object_type:'shoe'
>
> Or if you're searching on brands
>
> (genre:'skater' AND brand_desc:'skater boy') AND object_type:'brand'
>
> I apologize if I misunderstood your question.
>
> /cody
>
> On 4/19/07, Matthew Runo <[hidden email]> wrote:
>> Hey there-
>>
>> I was wondering if the following was possible, and, if so, how to set
>> it up...
>>
>> I want to index two different types of data, and have them searchable
>> from the same interface.
>>
>> For example, a group of products, with size, color, price, etc info.
>> And a group of brands, with brand, genre, brand description, etc info
>>
>> So, the info does overlap some. But a lot of the fields for each
>> "type" don't matter to the other. Is there a way to set up two
>> different schema so that both types may be indexed with relative  
>> ease?
>>
>> +--------------------------------------------------------+
>>   | Matthew Runo
>>   | Zappos Development
>>   | [hidden email]
>>   | 702-943-7833
>> +--------------------------------------------------------+
>>
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes?

gonzoprosperity
If you're doing this in Ruby, there is an "acts_as_solr" plugin for
Rails which takes exactly this approach to store all different kinds
of Model objects in the same index...I just "took" the idea from
there...

/Cody

On 4/19/07, Matthew Runo <[hidden email]> wrote:

> Ah hah! This appears to be what I'm interested in doing.
>
> I'll have to read up on object_types.
>
> +--------------------------------------------------------+
>   | Matthew Runo
>   | Zappos Development
>   | [hidden email]
>   | 702-943-7833
> +--------------------------------------------------------+
>
>
> On Apr 19, 2007, at 10:04 AM, Cody Caughlan wrote:
>
> > Why not just store an additional "object_type" field which
> > differentiates between the actual type of data you are looking for?
> >
> > So if you're looking for some shoes:
> >
> > (size:8 AND color:'blue') AND object_type:'shoe'
> >
> > Or if you're searching on brands
> >
> > (genre:'skater' AND brand_desc:'skater boy') AND object_type:'brand'
> >
> > I apologize if I misunderstood your question.
> >
> > /cody
> >
> > On 4/19/07, Matthew Runo <[hidden email]> wrote:
> >> Hey there-
> >>
> >> I was wondering if the following was possible, and, if so, how to set
> >> it up...
> >>
> >> I want to index two different types of data, and have them searchable
> >> from the same interface.
> >>
> >> For example, a group of products, with size, color, price, etc info.
> >> And a group of brands, with brand, genre, brand description, etc info
> >>
> >> So, the info does overlap some. But a lot of the fields for each
> >> "type" don't matter to the other. Is there a way to set up two
> >> different schema so that both types may be indexed with relative
> >> ease?
> >>
> >> +--------------------------------------------------------+
> >>   | Matthew Runo
> >>   | Zappos Development
> >>   | [hidden email]
> >>   | 702-943-7833
> >> +--------------------------------------------------------+
> >>
> >>
> >>
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes?

Matthew Runo
I'll actually be doing this in Perl..

any ideas on perl? heh

+--------------------------------------------------------+
  | Matthew Runo
  | Zappos Development
  | [hidden email]
  | 702-943-7833
+--------------------------------------------------------+


On Apr 19, 2007, at 11:59 AM, Cody Caughlan wrote:

> If you're doing this in Ruby, there is an "acts_as_solr" plugin for
> Rails which takes exactly this approach to store all different kinds
> of Model objects in the same index...I just "took" the idea from
> there...
>
> /Cody
>
> On 4/19/07, Matthew Runo <[hidden email]> wrote:
>> Ah hah! This appears to be what I'm interested in doing.
>>
>> I'll have to read up on object_types.
>>
>> +--------------------------------------------------------+
>>   | Matthew Runo
>>   | Zappos Development
>>   | [hidden email]
>>   | 702-943-7833
>> +--------------------------------------------------------+
>>
>>
>> On Apr 19, 2007, at 10:04 AM, Cody Caughlan wrote:
>>
>> > Why not just store an additional "object_type" field which
>> > differentiates between the actual type of data you are looking for?
>> >
>> > So if you're looking for some shoes:
>> >
>> > (size:8 AND color:'blue') AND object_type:'shoe'
>> >
>> > Or if you're searching on brands
>> >
>> > (genre:'skater' AND brand_desc:'skater boy') AND  
>> object_type:'brand'
>> >
>> > I apologize if I misunderstood your question.
>> >
>> > /cody
>> >
>> > On 4/19/07, Matthew Runo <[hidden email]> wrote:
>> >> Hey there-
>> >>
>> >> I was wondering if the following was possible, and, if so, how  
>> to set
>> >> it up...
>> >>
>> >> I want to index two different types of data, and have them  
>> searchable
>> >> from the same interface.
>> >>
>> >> For example, a group of products, with size, color, price, etc  
>> info.
>> >> And a group of brands, with brand, genre, brand description,  
>> etc info
>> >>
>> >> So, the info does overlap some. But a lot of the fields for each
>> >> "type" don't matter to the other. Is there a way to set up two
>> >> different schema so that both types may be indexed with relative
>> >> ease?
>> >>
>> >> +--------------------------------------------------------+
>> >>   | Matthew Runo
>> >>   | Zappos Development
>> >>   | [hidden email]
>> >>   | 702-943-7833
>> >> +--------------------------------------------------------+
>> >>
>> >>
>> >>
>> >
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes?

Erik Hatcher
Matthew,

All that is meant by "object_types" is an additional stored/indexed  
field in the Solr schema that gets added to every document providing  
context of which type it is (shoes or brands).  Then you can limit  
searches to a particular area by just filtering on type:shoes, for  
example.

        Erik

p.s. I could use some new shoes!

On Apr 19, 2007, at 3:17 PM, Matthew Runo wrote:

> I'll actually be doing this in Perl..
>
> any ideas on perl? heh
>
> +--------------------------------------------------------+
>  | Matthew Runo
>  | Zappos Development
>  | [hidden email]
>  | 702-943-7833
> +--------------------------------------------------------+
>
>
> On Apr 19, 2007, at 11:59 AM, Cody Caughlan wrote:
>
>> If you're doing this in Ruby, there is an "acts_as_solr" plugin for
>> Rails which takes exactly this approach to store all different kinds
>> of Model objects in the same index...I just "took" the idea from
>> there...
>>
>> /Cody
>>
>> On 4/19/07, Matthew Runo <[hidden email]> wrote:
>>> Ah hah! This appears to be what I'm interested in doing.
>>>
>>> I'll have to read up on object_types.
>>>
>>> +--------------------------------------------------------+
>>>   | Matthew Runo
>>>   | Zappos Development
>>>   | [hidden email]
>>>   | 702-943-7833
>>> +--------------------------------------------------------+
>>>
>>>
>>> On Apr 19, 2007, at 10:04 AM, Cody Caughlan wrote:
>>>
>>> > Why not just store an additional "object_type" field which
>>> > differentiates between the actual type of data you are looking  
>>> for?
>>> >
>>> > So if you're looking for some shoes:
>>> >
>>> > (size:8 AND color:'blue') AND object_type:'shoe'
>>> >
>>> > Or if you're searching on brands
>>> >
>>> > (genre:'skater' AND brand_desc:'skater boy') AND  
>>> object_type:'brand'
>>> >
>>> > I apologize if I misunderstood your question.
>>> >
>>> > /cody
>>> >
>>> > On 4/19/07, Matthew Runo <[hidden email]> wrote:
>>> >> Hey there-
>>> >>
>>> >> I was wondering if the following was possible, and, if so, how  
>>> to set
>>> >> it up...
>>> >>
>>> >> I want to index two different types of data, and have them  
>>> searchable
>>> >> from the same interface.
>>> >>
>>> >> For example, a group of products, with size, color, price, etc  
>>> info.
>>> >> And a group of brands, with brand, genre, brand description,  
>>> etc info
>>> >>
>>> >> So, the info does overlap some. But a lot of the fields for each
>>> >> "type" don't matter to the other. Is there a way to set up two
>>> >> different schema so that both types may be indexed with relative
>>> >> ease?
>>> >>
>>> >> +--------------------------------------------------------+
>>> >>   | Matthew Runo
>>> >>   | Zappos Development
>>> >>   | [hidden email]
>>> >>   | 702-943-7833
>>> >> +--------------------------------------------------------+
>>> >>
>>> >>
>>> >>
>>> >
>>>
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes?

Matthew Runo
Ah. That makes sense then. I wasn't sure if that was the best way to  
go about things or not. I didn't want to end up with a bunch of  
fields that were not being used all the time  if it would cause a  
degradation in search quality.

+--------------------------------------------------------+
  | Matthew Runo
  | Zappos Development
  | [hidden email]
  | 702-943-7833
+--------------------------------------------------------+


On Apr 19, 2007, at 12:59 PM, Erik Hatcher wrote:

> Matthew,
>
> All that is meant by "object_types" is an additional stored/indexed  
> field in the Solr schema that gets added to every document  
> providing context of which type it is (shoes or brands).  Then you  
> can limit searches to a particular area by just filtering on  
> type:shoes, for example.
>
> Erik
>
> p.s. I could use some new shoes!
>
> On Apr 19, 2007, at 3:17 PM, Matthew Runo wrote:
>
>> I'll actually be doing this in Perl..
>>
>> any ideas on perl? heh
>>
>> +--------------------------------------------------------+
>>  | Matthew Runo
>>  | Zappos Development
>>  | [hidden email]
>>  | 702-943-7833
>> +--------------------------------------------------------+
>>
>>
>> On Apr 19, 2007, at 11:59 AM, Cody Caughlan wrote:
>>
>>> If you're doing this in Ruby, there is an "acts_as_solr" plugin for
>>> Rails which takes exactly this approach to store all different kinds
>>> of Model objects in the same index...I just "took" the idea from
>>> there...
>>>
>>> /Cody
>>>
>>> On 4/19/07, Matthew Runo <[hidden email]> wrote:
>>>> Ah hah! This appears to be what I'm interested in doing.
>>>>
>>>> I'll have to read up on object_types.
>>>>
>>>> +--------------------------------------------------------+
>>>>   | Matthew Runo
>>>>   | Zappos Development
>>>>   | [hidden email]
>>>>   | 702-943-7833
>>>> +--------------------------------------------------------+
>>>>
>>>>
>>>> On Apr 19, 2007, at 10:04 AM, Cody Caughlan wrote:
>>>>
>>>> > Why not just store an additional "object_type" field which
>>>> > differentiates between the actual type of data you are looking  
>>>> for?
>>>> >
>>>> > So if you're looking for some shoes:
>>>> >
>>>> > (size:8 AND color:'blue') AND object_type:'shoe'
>>>> >
>>>> > Or if you're searching on brands
>>>> >
>>>> > (genre:'skater' AND brand_desc:'skater boy') AND  
>>>> object_type:'brand'
>>>> >
>>>> > I apologize if I misunderstood your question.
>>>> >
>>>> > /cody
>>>> >
>>>> > On 4/19/07, Matthew Runo <[hidden email]> wrote:
>>>> >> Hey there-
>>>> >>
>>>> >> I was wondering if the following was possible, and, if so,  
>>>> how to set
>>>> >> it up...
>>>> >>
>>>> >> I want to index two different types of data, and have them  
>>>> searchable
>>>> >> from the same interface.
>>>> >>
>>>> >> For example, a group of products, with size, color, price,  
>>>> etc info.
>>>> >> And a group of brands, with brand, genre, brand description,  
>>>> etc info
>>>> >>
>>>> >> So, the info does overlap some. But a lot of the fields for each
>>>> >> "type" don't matter to the other. Is there a way to set up two
>>>> >> different schema so that both types may be indexed with relative
>>>> >> ease?
>>>> >>
>>>> >> +--------------------------------------------------------+
>>>> >>   | Matthew Runo
>>>> >>   | Zappos Development
>>>> >>   | [hidden email]
>>>> >>   | 702-943-7833
>>>> >> +--------------------------------------------------------+
>>>> >>
>>>> >>
>>>> >>
>>>> >
>>>>
>>>>
>>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes?

Chris Hostetter-3
In reply to this post by gonzoprosperity
: So if you're looking for some shoes:
: (size:8 AND color:'blue') AND object_type:'shoe'

: Or if you're searching on brands
: (genre:'skater' AND brand_desc:'skater boy') AND object_type:'brand'

a slight improvement on this: put your object_type restriction in a
filter query (&fq=object_type:foo) and not in your main query ... that way
it won't affect the scoring, and it will be cached uniquely so the work of
identifying the set of all shows will only be done once per commit (and
likewise for brands)



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Multiple indexes?

Ryan McKinley

As this question comes up so often, i put a new page on the wiki:
  http://wiki.apache.org/solr/MultipleIndexes

We should fill in more details and link it to the front page.


Chris Hostetter wrote:

> : So if you're looking for some shoes:
> : (size:8 AND color:'blue') AND object_type:'shoe'
>
> : Or if you're searching on brands
> : (genre:'skater' AND brand_desc:'skater boy') AND object_type:'brand'
>
> a slight improvement on this: put your object_type restriction in a
> filter query (&fq=object_type:foo) and not in your main query ... that way
> it won't affect the scoring, and it will be cached uniquely so the work of
> identifying the set of all shows will only be done once per commit (and
> likewise for brands)
>
>
>
> -Hoss
>
>