Solr and Multiple Index Partitions

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr and Multiple Index Partitions

Venkatesh Seetharam
Hello there,

Howdy. I was wondering if there is a way to configure one Solr instance to
search multiple Index partitions? I read the wiki and found the entry in
SolrConfig.xml:

<dataDir>/var/data/solr</dataDir>

Can I have mutiple directories? Comma separated?

Any help is greatly appreciated.

--
Thanks,
Venkatesh

"Perfection (in design) is achieved not when there is nothing more to add,
but rather when there is nothing more to take away."
- Antoine de Saint-Exupéry
Reply | Threaded
Open this post in threaded view
|

Re: Solr and Multiple Index Partitions

Ryan McKinley
Solr looks at one index - If you want to look at multiple indexes, you
need multiple solr instances running.  Check the wiki for how to set
that up:

  http://wiki.apache.org/solr/SolrJetty

(the resin and tomcat pages have something similar)



On 3/7/07, Venkatesh Seetharam <[hidden email]> wrote:

> Hello there,
>
> Howdy. I was wondering if there is a way to configure one Solr instance to
> search multiple Index partitions? I read the wiki and found the entry in
> SolrConfig.xml:
>
> <dataDir>/var/data/solr</dataDir>
>
> Can I have mutiple directories? Comma separated?
>
> Any help is greatly appreciated.
>
> --
> Thanks,
> Venkatesh
>
> "Perfection (in design) is achieved not when there is nothing more to add,
> but rather when there is nothing more to take away."
> - Antoine de Saint-Exupéry
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr and Multiple Index Partitions

Venkatesh Seetharam
Thanks Ryan for your inputs. If I'm not using Solr webapp but wrapping Solr
in plain Java, is there any way that I could get Solr to work with multiple
index partitions?

Venkatesh

On 3/7/07, Ryan McKinley <[hidden email]> wrote:

>
> Solr looks at one index - If you want to look at multiple indexes, you
> need multiple solr instances running.  Check the wiki for how to set
> that up:
>
>   http://wiki.apache.org/solr/SolrJetty
>
> (the resin and tomcat pages have something similar)
>
>
>
> On 3/7/07, Venkatesh Seetharam <[hidden email]> wrote:
> > Hello there,
> >
> > Howdy. I was wondering if there is a way to configure one Solr instance
> to
> > search multiple Index partitions? I read the wiki and found the entry in
> > SolrConfig.xml:
> >
> > <dataDir>/var/data/solr</dataDir>
> >
> > Can I have mutiple directories? Comma separated?
> >
> > Any help is greatly appreciated.
> >
> > --
> > Thanks,
> > Venkatesh
> >
> > "Perfection (in design) is achieved not when there is nothing more to
> add,
> > but rather when there is nothing more to take away."
> > - Antoine de Saint-Exupéry
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr and Multiple Index Partitions

Bradley Burke
Venkatesh,

Are you perhapes talking about Federated Searching (
http://wiki.apache.org/solr/FederatedSearch).  I am new to Solr but this was
a feature I was looking far as well.  I do not think its built into Solr at
the momment.

Regards,
Brad


On 3/7/07, Venkatesh Seetharam <[hidden email]> wrote:

>
> Thanks Ryan for your inputs. If I'm not using Solr webapp but wrapping
> Solr
> in plain Java, is there any way that I could get Solr to work with
> multiple
> index partitions?
>
> Venkatesh
>
> On 3/7/07, Ryan McKinley <[hidden email]> wrote:
> >
> > Solr looks at one index - If you want to look at multiple indexes, you
> > need multiple solr instances running.  Check the wiki for how to set
> > that up:
> >
> >   http://wiki.apache.org/solr/SolrJetty
> >
> > (the resin and tomcat pages have something similar)
> >
> >
> >
> > On 3/7/07, Venkatesh Seetharam <[hidden email]> wrote:
> > > Hello there,
> > >
> > > Howdy. I was wondering if there is a way to configure one Solr
> instance
> > to
> > > search multiple Index partitions? I read the wiki and found the entry
> in
> > > SolrConfig.xml:
> > >
> > > <dataDir>/var/data/solr</dataDir>
> > >
> > > Can I have mutiple directories? Comma separated?
> > >
> > > Any help is greatly appreciated.
> > >
> > > --
> > > Thanks,
> > > Venkatesh
> > >
> > > "Perfection (in design) is achieved not when there is nothing more to
> > add,
> > > but rather when there is nothing more to take away."
> > > - Antoine de Saint-Exupéry
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr and Multiple Index Partitions

Ryan McKinley
In reply to this post by Venkatesh Seetharam
As it is now... I don't think so.  SolrCore is a static singleton
class -- without some serious reworking, i think there is only one
instance per jvm.

I think getting rid of the static singleton should go on the long term
TODO list, but that doesn't help you now.

Although it seems weird, just adding a "type" field makes it possible
to put things that seem like they need their own database into the
same index - you select the 'type' you want by adding "+type:mytype"
to your query


On 3/7/07, Venkatesh Seetharam <[hidden email]> wrote:

> Thanks Ryan for your inputs. If I'm not using Solr webapp but wrapping Solr
> in plain Java, is there any way that I could get Solr to work with multiple
> index partitions?
>
> Venkatesh
>
> On 3/7/07, Ryan McKinley <[hidden email]> wrote:
> >
> > Solr looks at one index - If you want to look at multiple indexes, you
> > need multiple solr instances running.  Check the wiki for how to set
> > that up:
> >
> >   http://wiki.apache.org/solr/SolrJetty
> >
> > (the resin and tomcat pages have something similar)
> >
> >
> >
> > On 3/7/07, Venkatesh Seetharam <[hidden email]> wrote:
> > > Hello there,
> > >
> > > Howdy. I was wondering if there is a way to configure one Solr instance
> > to
> > > search multiple Index partitions? I read the wiki and found the entry in
> > > SolrConfig.xml:
> > >
> > > <dataDir>/var/data/solr</dataDir>
> > >
> > > Can I have mutiple directories? Comma separated?
> > >
> > > Any help is greatly appreciated.
> > >
> > > --
> > > Thanks,
> > > Venkatesh
> > >
> > > "Perfection (in design) is achieved not when there is nothing more to
> > add,
> > > but rather when there is nothing more to take away."
> > > - Antoine de Saint-Exupéry
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr and Multiple Index Partitions

Venkatesh Seetharam
In reply to this post by Bradley Burke
Yes, I'm implementing federated search. I do have N partitions of indexes
built and I'd like to have mutiple Solr instances in a cluster each serving
atleast 2 partitions. I was wondering if I could somehow find a way to make
Solr work with atleast 2 partitions. It looks like I need to have smaller
number of partitions.

Thanks,
Venkatesh

On 3/7/07, Bradley Burke <[hidden email]> wrote:

>
> Venkatesh,
>
> Are you perhapes talking about Federated Searching (
> http://wiki.apache.org/solr/FederatedSearch).  I am new to Solr but this
> was
> a feature I was looking far as well.  I do not think its built into Solr
> at
> the momment.
>
> Regards,
> Brad
>
>
> On 3/7/07, Venkatesh Seetharam <[hidden email]> wrote:
> >
> > Thanks Ryan for your inputs. If I'm not using Solr webapp but wrapping
> > Solr
> > in plain Java, is there any way that I could get Solr to work with
> > multiple
> > index partitions?
> >
> > Venkatesh
> >
> > On 3/7/07, Ryan McKinley <[hidden email]> wrote:
> > >
> > > Solr looks at one index - If you want to look at multiple indexes, you
> > > need multiple solr instances running.  Check the wiki for how to set
> > > that up:
> > >
> > >   http://wiki.apache.org/solr/SolrJetty
> > >
> > > (the resin and tomcat pages have something similar)
> > >
> > >
> > >
> > > On 3/7/07, Venkatesh Seetharam <[hidden email]> wrote:
> > > > Hello there,
> > > >
> > > > Howdy. I was wondering if there is a way to configure one Solr
> > instance
> > > to
> > > > search multiple Index partitions? I read the wiki and found the
> entry
> > in
> > > > SolrConfig.xml:
> > > >
> > > > <dataDir>/var/data/solr</dataDir>
> > > >
> > > > Can I have mutiple directories? Comma separated?
> > > >
> > > > Any help is greatly appreciated.
> > > >
> > > > --
> > > > Thanks,
> > > > Venkatesh
> > > >
> > > > "Perfection (in design) is achieved not when there is nothing more
> to
> > > add,
> > > > but rather when there is nothing more to take away."
> > > > - Antoine de Saint-Exupéry
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr and Multiple Index Partitions

Venkatesh Seetharam
In reply to this post by Ryan McKinley
Thanks Ryan for your insight. I do not wish to change Solr.

> Although it seems weird, just adding a "type" field makes it possible
I'm not working with a DB but since we are indexing a huge vault of XML
documents, I use Hadoop + Lucene for indexing which finally generates N
number of partitions, I was thinking to use Solr as the searcher and will
have a broker which merges results from individual Solr searchers.

I do not know if I can ask another question here or start a new thread.

I use a custom Analyzer which extends Lucene's StandardAnalyzer. When I
configured Solr to use this one, It throws an exception
RuntimeException("Can't set positionIncrementGap on custom analyzer " +
analyzer.getClass()).

Do I need to extend a specific Analyzer for it to work with Solr?

Venkatesh

On 3/7/07, Ryan McKinley <[hidden email]> wrote:

>
> As it is now... I don't think so.  SolrCore is a static singleton
> class -- without some serious reworking, i think there is only one
> instance per jvm.
>
> I think getting rid of the static singleton should go on the long term
> TODO list, but that doesn't help you now.
>
> Although it seems weird, just adding a "type" field makes it possible
> to put things that seem like they need their own database into the
> same index - you select the 'type' you want by adding "+type:mytype"
> to your query
>
>
> On 3/7/07, Venkatesh Seetharam <[hidden email]> wrote:
> > Thanks Ryan for your inputs. If I'm not using Solr webapp but wrapping
> Solr
> > in plain Java, is there any way that I could get Solr to work with
> multiple
> > index partitions?
> >
> > Venkatesh
> >
> > On 3/7/07, Ryan McKinley <[hidden email]> wrote:
> > >
> > > Solr looks at one index - If you want to look at multiple indexes, you
> > > need multiple solr instances running.  Check the wiki for how to set
> > > that up:
> > >
> > >   http://wiki.apache.org/solr/SolrJetty
> > >
> > > (the resin and tomcat pages have something similar)
> > >
> > >
> > >
> > > On 3/7/07, Venkatesh Seetharam <[hidden email]> wrote:
> > > > Hello there,
> > > >
> > > > Howdy. I was wondering if there is a way to configure one Solr
> instance
> > > to
> > > > search multiple Index partitions? I read the wiki and found the
> entry in
> > > > SolrConfig.xml:
> > > >
> > > > <dataDir>/var/data/solr</dataDir>
> > > >
> > > > Can I have mutiple directories? Comma separated?
> > > >
> > > > Any help is greatly appreciated.
> > > >
> > > > --
> > > > Thanks,
> > > > Venkatesh
> > > >
> > > > "Perfection (in design) is achieved not when there is nothing more
> to
> > > add,
> > > > but rather when there is nothing more to take away."
> > > > - Antoine de Saint-Exupéry
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

ReSolr and Multiple Index Partitions

Jack L
In reply to this post by Ryan McKinley
Selecting by type will do the job. But I suppose it sacrifice
performance because having multiple document types in the same
index will render a larger index. Is it bad?

--
Best regards,
Jack

Wednesday, March 7, 2007, 2:15:14 PM, you wrote:

> As it is now... I don't think so.  SolrCore is a static singleton
> class -- without some serious reworking, i think there is only one
> instance per jvm.

> I think getting rid of the static singleton should go on the long term
> TODO list, but that doesn't help you now.

> Although it seems weird, just adding a "type" field makes it possible
> to put things that seem like they need their own database into the
> same index - you select the 'type' you want by adding "+type:mytype"
> to your query


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 
Reply | Threaded
Open this post in threaded view
|

Re: ReSolr and Multiple Index Partitions

Erik Hatcher

On Mar 7, 2007, at 9:20 PM, Jack L wrote:
> Selecting by type will do the job. But I suppose it sacrifice
> performance because having multiple document types in the same
> index will render a larger index. Is it bad?

A many documents we talking here?

My hunch is you'll be fine :)

        Erik

Reply | Threaded
Open this post in threaded view
|

Re: Solr and Multiple Index Partitions

Chris Hostetter-3
In reply to this post by Venkatesh Seetharam

: I use a custom Analyzer which extends Lucene's StandardAnalyzer. When I
: configured Solr to use this one, It throws an exception
: RuntimeException("Can't set positionIncrementGap on custom analyzer " +
: analyzer.getClass()).
:
: Do I need to extend a specific Analyzer for it to work with Solr?

you can use any Analyzer you want, but you can't configure a
positionIncrementGap in the schema.xml unless your Analyzer extends
SolrAnalyzer (the concept of a position increment gap is an inherient
property that Lucene Analyzers can specify, but configuring it explicitly
is a Solr concept)




-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Solr and Multiple Index Partitions

Chris Hostetter-3

whoops .. forgot the documentaiton link...

http://wiki.apache.org/solr/SolrPlugins#head-9939da9abe85a79eb30a026e85cc4aec0beac10c

: you can use any Analyzer you want, but you can't configure a
: positionIncrementGap in the schema.xml unless your Analyzer extends
: SolrAnalyzer (the concept of a position increment gap is an inherient
: property that Lucene Analyzers can specify, but configuring it explicitly
: is a Solr concept)


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Solr and Multiple Index Partitions

Venkatesh Seetharam
In reply to this post by Chris Hostetter-3
Thanks Chris for a wonderful explanation. I completely get it now. Thanks
for the handy URL too.

Venkatesh

On 3/8/07, Chris Hostetter <[hidden email]> wrote:

>
>
> : I use a custom Analyzer which extends Lucene's StandardAnalyzer. When I
> : configured Solr to use this one, It throws an exception
> : RuntimeException("Can't set positionIncrementGap on custom analyzer " +
> : analyzer.getClass()).
> :
> : Do I need to extend a specific Analyzer for it to work with Solr?
>
> you can use any Analyzer you want, but you can't configure a
> positionIncrementGap in the schema.xml unless your Analyzer extends
> SolrAnalyzer (the concept of a position increment gap is an inherient
> property that Lucene Analyzers can specify, but configuring it explicitly
> is a Solr concept)
>
>
>
>
> -Hoss
>
>