Two Solr Webapps, one folder for the index data?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Two Solr Webapps, one folder for the index data?

Jörg Kiegeland
I have successfully configured two parallel  Solr webapps , however I
see that all data gets stored in one folder of my Tomcat installation,
namely C:\Tomcat\solr\data\index.

How can I configure that each Solr webapp shall store the data in the
folders I assigned at  <Environment value="individualSolrFolder">, where
already the Solr scheme etc. resides (so that it get stored at
individualSolrFolder/data/index)?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Two Solr Webapps, one folder for the index data?

Patrick O'Leary-2
I actually have a patch for solr config parser which allows you to use context environment variables in the solrconfig.xml
I generally use it for development when I'm working with multiple instances and different data dirs.  I'll add it to jira today if you want it.

P

Jörg Kiegeland wrote:
I have successfully configured two parallel  Solr webapps , however I see that all data gets stored in one folder of my Tomcat installation, namely C:\Tomcat\solr\data\index.

How can I configure that each Solr webapp shall store the data in the folders I assigned at  <Environment value="individualSolrFolder">, where already the Solr scheme etc. resides (so that it get stored at individualSolrFolder/data/index)?

Thanks


--
Patrick O'Leary


You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles.
 Do you understand this? 
And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.
  - Albert Einstein
View Patrick O Leary's LinkedIn profileView Patrick O Leary's profile
Reply | Threaded
Open this post in threaded view
|

Re: Two Solr Webapps, one folder for the index data?

Jörg Kiegeland
>  I actually have a patch for solr config parser which allows you to
> use context environment variables in the solrconfig.xml
> I generally use it for development when I'm working with multiple
> instances and different data dirs.  I'll add it to jira today if you
> want it.
That would be nice.

However I cannot believe that one cannot configure this by
someconfiguration file by now - what if only one index needs to be
backuped and the other index does not need to be backuped because it
carries only redundant information of some other data source, like in my
case. And if all data is put in one folder, you can backup only both
indexes together..

Reply | Threaded
Open this post in threaded view
|

Re: Two Solr Webapps, one folder for the index data?

Mike Klaas
In reply to this post by Patrick O'Leary-2
I use jvm system properties for this; they seem to work well.

-Mike

On 11-Dec-07, at 7:39 AM, patrick o'leary wrote:

> I actually have a patch for solr config parser which allows you to  
> use context environment variables in the solrconfig.xml
> I generally use it for development when I'm working with multiple  
> instances and different data dirs.  I'll add it to jira today if  
> you want it.
>
> P
>
> Jörg Kiegeland wrote:
>> I have successfully configured two parallel  Solr webapps ,  
>> however I see that all data gets stored in one folder of my Tomcat  
>> installation, namely C:\Tomcat\solr\data\index.
>>
>> How can I configure that each Solr webapp shall store the data in  
>> the folders I assigned at  <Environment  
>> value="individualSolrFolder">, where already the Solr scheme etc.  
>> resides (so that it get stored at individualSolrFolder/data/index)?
>>
>> Thanks
>>
>
> --
> Patrick O'Leary You see, wire telegraph is a kind of a very, very  
> long cat. You pull his tail in New York and his head is meowing in  
> Los Angeles. Do you understand this? And radio operates exactly the  
> same way: you send signals here, they receive them there. The only  
> difference is that there is no cat. - Albert Einstein
> <btn_in_20x15.gif>
> View Patrick O Leary's profile

Reply | Threaded
Open this post in threaded view
|

Re: Two Solr Webapps, one folder for the index data?

Patrick O'Leary-2
JVM properties restrict you to a single implementation within a jvm.

For instance if you want multiple instances of solr running with the same schema, with different data
dir's in the one app server. You'll have to have several copies of solrconfig and schema.xml.

By using context environment, I can have multiple contexts like

pjaol:~/tmp/locallucene/solr/tomcat-conf pjaol$ more solr.xml
<Context docBase="/Users/pjaol/tmp/locallucene/solr/solr.war" debug="0" crossContext="true" allowLinking="true">
   <Environment name="solr/home" type="java.lang.String" value="/Users/pjaol/tmp/locallucene/solr" override="true" />
   <Environment name="solr/data/dir" type="java.lang.String" value="/Users/pjaol/tmp/locallucene/solr/partitions/data" override="true"/>
</Context>

pjaol:~/tmp/locallucene/solr/tomcat-conf pjaol$ more solr1.xml
<Context docBase="/Users/pjaol/tmp/locallucene/solr/solr.war" debug="0" crossContext="true" allowLinking="true" >
   <Environment name="solr/home" type="java.lang.String" value="/Users/pjaol/tmp/locallucene/solr" override="true" />
   <Environment name="solr/data/dir" type="java.lang.String" value="/Users/pjaol/tmp/locallucene/solr/partitions/data1" override="true"/>
</Context>

Changing just the solr/data/dir for each instance.

And in my solrconfig.xml
 <dataDir>${env/solr/data/dir:./solr/data}</dataDir>

It certainly makes development & operations easier.

P

Mike Klaas wrote:
I use jvm system properties for this; they seem to work well.

-Mike

On 11-Dec-07, at 7:39 AM, patrick o'leary wrote:

I actually have a patch for solr config parser which allows you to use context environment variables in the solrconfig.xml
I generally use it for development when I'm working with multiple instances and different data dirs.  I'll add it to jira today if you want it.

P

Jörg Kiegeland wrote:
I have successfully configured two parallel  Solr webapps , however I see that all data gets stored in one folder of my Tomcat installation, namely C:\Tomcat\solr\data\index.

How can I configure that each Solr webapp shall store the data in the folders I assigned at  <Environment value="individualSolrFolder">, where already the Solr scheme etc. resides (so that it get stored at individualSolrFolder/data/index)?

Thanks


-- 
Patrick O'Leary You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat. - Albert Einstein
<btn_in_20x15.gif>
View Patrick O Leary's profile


--
Patrick O'Leary

You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles.
 Do you understand this? 
And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.
  - Albert Einstein
View Patrick O Leary's LinkedIn profileView Patrick O Leary's profile
Reply | Threaded
Open this post in threaded view
|

Re: Two Solr Webapps, one folder for the index data?

hossman
In reply to this post by Jörg Kiegeland

: However I cannot believe that one cannot configure this by someconfiguration
: file by now - what if only one index needs to be backuped and the other index

is the option you are looking forthe <dataDir /> in solrconfig.xml ?



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Two Solr Webapps, one folder for the index data?

hossman
In reply to this post by Patrick O'Leary-2

: I actually have a patch for solr config parser which allows you to use
: context environment variables in the solrconfig.xml
: I generally use it for development when I'm working with multiple
: instances and different data dirs.  I'll add it to jira today if you
: want it.

yes please! ... Solr already has system property variable replacement in
solrconfig.xml, and we discussed a while back (on solr-dev i think) adding
code to automatily create system properties on startup for any solr/* JNDI
variables set so the same variable subst code could be reused ... but i
don't think anyone ever opened an issue or created a patch for it.



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Two Solr Webapps, one folder for the index data?

Otis Gospodnetic-2
In reply to this post by Jörg Kiegeland
Maybe I'm confused.  Can't you use the brand-spanking new MultiCore stuff for this, or JNDI, as I just mentioned in the "Re: Solr, Multiple processes running" thread?

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----
From: patrick o'leary <[hidden email]>
To: [hidden email]
Sent: Tuesday, December 11, 2007 2:18:58 PM
Subject: Re: Two Solr Webapps, one folder for the index data?




 
 

JVM properties restrict you to a single implementation within a jvm.



For instance if you want multiple instances of solr running with the
same schema, with different data

dir's in the one app server. You'll have to have several copies of
solrconfig and schema.xml.



By using context environment, I can have multiple contexts like



pjaol:~/tmp/locallucene/solr/tomcat-conf pjaol$ more solr.xml

<Context docBase="/Users/pjaol/tmp/locallucene/solr/solr.war"
debug="0" crossContext="true" allowLinking="true">

   <Environment name="solr/home" type="java.lang.String"
value="/Users/pjaol/tmp/locallucene/solr" override="true" />

   <Environment name="solr/data/dir" type="java.lang.String"
value="/Users/pjaol/tmp/locallucene/solr/partitions/data"
override="true"/>

</Context>



pjaol:~/tmp/locallucene/solr/tomcat-conf pjaol$ more solr1.xml

<Context docBase="/Users/pjaol/tmp/locallucene/solr/solr.war"
debug="0" crossContext="true" allowLinking="true" >

   <Environment name="solr/home" type="java.lang.String"
value="/Users/pjaol/tmp/locallucene/solr" override="true" />

   <Environment name="solr/data/dir" type="java.lang.String"
value="/Users/pjaol/tmp/locallucene/solr/partitions/data1"
override="true"/>

</Context>



Changing just the solr/data/dir for each instance.



And in my solrconfig.xml

 <dataDir>${env/solr/data/dir:./solr/data}</dataDir>



It certainly makes development & operations easier.



P



Mike Klaas wrote:
I use jvm system properties for this; they seem to work
well.
 

 

-Mike
 

 

On 11-Dec-07, at 7:39 AM, patrick o'leary wrote:
 

 

  I actually have a patch for solr config
parser which allows you to use context environment variables in the
solrconfig.xml
   

I generally use it for development when I'm working with multiple
instances and different data dirs.  I'll add it to jira today if you
want it.
   

   

P
   

   

Jörg Kiegeland wrote:
   

    I have successfully configured two
parallel  Solr webapps , however I see that all data gets stored in one
folder of my Tomcat installation, namely C:\Tomcat\solr\data\index.
     

     

How can I configure that each Solr webapp shall store the data in the
folders I assigned at  <Environment
value="individualSolrFolder">, where already the Solr scheme etc.
resides (so that it get stored at individualSolrFolder/data/index)?
     

     

Thanks
     

     

   
   

--

Patrick O'Leary You see, wire telegraph is a kind of a very, very long
cat. You pull his tail in New York and his head is meowing in Los
Angeles. Do you understand this? And radio operates exactly the same
way: you send signals here, they receive them there. The only
difference is that there is no cat. - Albert Einstein
   

<btn_in_20x15.gif>
   

View Patrick O Leary's profile
   

 
 




--

Patrick O'Leary

You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles.
 Do you understand this?
And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.
  - Albert Einstein

View
Patrick O Leary's profile





Reply | Threaded
Open this post in threaded view
|

RE: Two Solr Webapps, one folder for the index data?

Arnone, Anthony
In reply to this post by hossman
I asked a question similar to this back in http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200709.mbox/%3c25E214935D7C0342B91631BC43467D8DDB6F1D@...%3e and didn't really find anyone who was doing this. What I wound up doing was adding a variable to the context.xml file called contextRelativeHome:

<env-entry>
  <env-entry-name>solr/contextRelativeHome</env-entry-name>
  <env-entry-type>java.lang.Boolean</env-entry-type>
  <env-entry-value>true</env-entry-value>
</env-entry>

Which causes the SolrResourceLoader to prepend the context directory to the solr/home variable (the context directory is identified in the SolrDispatchFilter and stored in the global Config). This way, I can have a multiple instances of Solr up and running with the exact same configuration, and their indices contained wholly within their deployment directories.

So since this is a fresh thread, does this seem like a bad way to do it? It would be much easier if I could put context variables directly into the existing solr/home variable, for sure.

Anthony


-----Original Message-----
From: Chris Hostetter [mailto:[hidden email]]
Sent: Tuesday, December 11, 2007 1:09 PM
To: [hidden email]
Subject: Re: Two Solr Webapps, one folder for the index data?


: I actually have a patch for solr config parser which allows you to use
: context environment variables in the solrconfig.xml
: I generally use it for development when I'm working with multiple
: instances and different data dirs.  I'll add it to jira today if you
: want it.

yes please! ... Solr already has system property variable replacement in
solrconfig.xml, and we discussed a while back (on solr-dev i think) adding
code to automatily create system properties on startup for any solr/* JNDI
variables set so the same variable subst code could be reused ... but i
don't think anyone ever opened an issue or created a patch for it.



-Hoss

Reply | Threaded
Open this post in threaded view
|

RE: Two Solr Webapps, one folder for the index data?

hossman

: I asked a question similar to this back in
: http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200709.mbox/%3c25E214935D7C0342B91631BC43467D8DDB6F1D@...%3e 

: SolrDispatchFilter and stored in the global Config). This way, I can
: have a multiple instances of Solr up and running with the exact same
: configuration, and their indices contained wholly within their
: deployment directories.

As i mentioned in that thread (and i don't think you ever replied) this
seems like a really bad idea ... anytime you want to upgrade Solr, your
configs and data all get completely blown away.

I think if people want to reuse the same configs multiple times with
only small varaitions (for things like the dataDir) it makes a lot more
sense to add support for variable substitution based on JNDI variables...

: : I actually have a patch for solr config parser which allows you to use
: : context environment variables in the solrconfig.xml
: : I generally use it for development when I'm working with multiple
: : instances and different data dirs.  I'll add it to jira today if you
: : want it.




-Hoss