Separating cores from Solr home

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Separating cores from Solr home

Tom Evans
Hi all

I'm struggling to configure solr cloud to put the index files and
core.properties in the correct places in SolrCloud 5.5. Let me explain
what I am trying to achieve:

* solr is installed in /opt/solr
* the user who runs solr only has read only access to that tree
* the solr home files - custom libraries, log4j.properties, solr.in.sh
and solr.xml - live in /data/project/solr/releases/<revision>, which
is then the target of a symlink /data/project/solr/releases/current
* releasing a new version of the solr home (eg adding/changing
libraries, changing logging options) is done by checking out a fresh
copy of the solr home, switching the symlink and restarting solr
* the solr core.properties and any data live in /data/project/indexes,
so they are preserved when new solr home is released

Setting core specific dataDir with absolute paths in solrconfig.xml
only gets me part of the way, as the core.properties for each shard is
created inside the solr home.

This is obviously no good, as when releasing a new version of the solr
home, they will no longer be in the current solr home.

Cheers

Tom
Reply | Threaded
Open this post in threaded view
|

Re: Separating cores from Solr home

Tom Evans
Hmm, I've worked around this by setting the directory where the
indexes should live to be the actual solr home, and symlink the files
from the current release in to that directory, but it feels icky.

Any better ideas?

Cheers

Tom

On Thu, Mar 3, 2016 at 11:12 AM, Tom Evans <[hidden email]> wrote:

> Hi all
>
> I'm struggling to configure solr cloud to put the index files and
> core.properties in the correct places in SolrCloud 5.5. Let me explain
> what I am trying to achieve:
>
> * solr is installed in /opt/solr
> * the user who runs solr only has read only access to that tree
> * the solr home files - custom libraries, log4j.properties, solr.in.sh
> and solr.xml - live in /data/project/solr/releases/<revision>, which
> is then the target of a symlink /data/project/solr/releases/current
> * releasing a new version of the solr home (eg adding/changing
> libraries, changing logging options) is done by checking out a fresh
> copy of the solr home, switching the symlink and restarting solr
> * the solr core.properties and any data live in /data/project/indexes,
> so they are preserved when new solr home is released
>
> Setting core specific dataDir with absolute paths in solrconfig.xml
> only gets me part of the way, as the core.properties for each shard is
> created inside the solr home.
>
> This is obviously no good, as when releasing a new version of the solr
> home, they will no longer be in the current solr home.
>
> Cheers
>
> Tom
Reply | Threaded
Open this post in threaded view
|

Re: Separating cores from Solr home

Malcolm Upayavira Holmes
You can add a <lib> entry in your solrconfig to point to an entirely
different location for libraries. Update them, then you just need to
reload cores, not the whole of Solr.

Then, try setting the LOG4J_PROPS envvar before starting Solr, so you
can have your logging configs somewhere else, and solr_home becomes
purely about data alone.

Does that work?

Upayavira

On Thu, Mar 3, 2016, at 11:58 AM, Tom Evans wrote:

> Hmm, I've worked around this by setting the directory where the
> indexes should live to be the actual solr home, and symlink the files
> from the current release in to that directory, but it feels icky.
>
> Any better ideas?
>
> Cheers
>
> Tom
>
> On Thu, Mar 3, 2016 at 11:12 AM, Tom Evans <[hidden email]>
> wrote:
> > Hi all
> >
> > I'm struggling to configure solr cloud to put the index files and
> > core.properties in the correct places in SolrCloud 5.5. Let me explain
> > what I am trying to achieve:
> >
> > * solr is installed in /opt/solr
> > * the user who runs solr only has read only access to that tree
> > * the solr home files - custom libraries, log4j.properties, solr.in.sh
> > and solr.xml - live in /data/project/solr/releases/<revision>, which
> > is then the target of a symlink /data/project/solr/releases/current
> > * releasing a new version of the solr home (eg adding/changing
> > libraries, changing logging options) is done by checking out a fresh
> > copy of the solr home, switching the symlink and restarting solr
> > * the solr core.properties and any data live in /data/project/indexes,
> > so they are preserved when new solr home is released
> >
> > Setting core specific dataDir with absolute paths in solrconfig.xml
> > only gets me part of the way, as the core.properties for each shard is
> > created inside the solr home.
> >
> > This is obviously no good, as when releasing a new version of the solr
> > home, they will no longer be in the current solr home.
> >
> > Cheers
> >
> > Tom
Reply | Threaded
Open this post in threaded view
|

Re: Separating cores from Solr home

Jeff Wartes
In reply to this post by Tom Evans
It’s a bit backwards feeling, but I’ve had luck setting the install dir and solr home, instead of the data dir.

Something like:
-Dsolr.solr.home=/data/solr
-Dsolr.install.dir=/opt/solr


So all of the Solr files are in in /opt/solr and all of the index/core related files end up in /data/solr.




On 3/3/16, 3:58 AM, "Tom Evans" <[hidden email]> wrote:

>Hmm, I've worked around this by setting the directory where the
>indexes should live to be the actual solr home, and symlink the files
>from the current release in to that directory, but it feels icky.
>
>Any better ideas?
>
>Cheers
>
>Tom
>
>On Thu, Mar 3, 2016 at 11:12 AM, Tom Evans <[hidden email]> wrote:
>> Hi all
>>
>> I'm struggling to configure solr cloud to put the index files and
>> core.properties in the correct places in SolrCloud 5.5. Let me explain
>> what I am trying to achieve:
>>
>> * solr is installed in /opt/solr
>> * the user who runs solr only has read only access to that tree
>> * the solr home files - custom libraries, log4j.properties, solr.in.sh
>> and solr.xml - live in /data/project/solr/releases/<revision>, which
>> is then the target of a symlink /data/project/solr/releases/current
>> * releasing a new version of the solr home (eg adding/changing
>> libraries, changing logging options) is done by checking out a fresh
>> copy of the solr home, switching the symlink and restarting solr
>> * the solr core.properties and any data live in /data/project/indexes,
>> so they are preserved when new solr home is released
>>
>> Setting core specific dataDir with absolute paths in solrconfig.xml
>> only gets me part of the way, as the core.properties for each shard is
>> created inside the solr home.
>>
>> This is obviously no good, as when releasing a new version of the solr
>> home, they will no longer be in the current solr home.
>>
>> Cheers
>>
>> Tom
Reply | Threaded
Open this post in threaded view
|

Re: Separating cores from Solr home

Shawn Heisey-2
In reply to this post by Tom Evans
On 3/3/2016 4:12 AM, Tom Evans wrote:

> I'm struggling to configure solr cloud to put the index files and
> core.properties in the correct places in SolrCloud 5.5. Let me explain
> what I am trying to achieve:
>
> * solr is installed in /opt/solr
> * the user who runs solr only has read only access to that tree
> * the solr home files - custom libraries, log4j.properties, solr.in.sh
> and solr.xml - live in /data/project/solr/releases/<revision>, which
> is then the target of a symlink /data/project/solr/releases/current
> * releasing a new version of the solr home (eg adding/changing
> libraries, changing logging options) is done by checking out a fresh
> copy of the solr home, switching the symlink and restarting solr
> * the solr core.properties and any data live in /data/project/indexes,
> so they are preserved when new solr home is released
>
> Setting core specific dataDir with absolute paths in solrconfig.xml
> only gets me part of the way, as the core.properties for each shard is
> created inside the solr home.
>
> This is obviously no good, as when releasing a new version of the solr
> home, they will no longer be in the current solr home.\

Add the coreRootDirectory config element to your solr.xml file, so Solr
does core discovery directly on /data/project/indexes, rather than the
solr home.  The solr.xml and lib directory with your jars will remain in
the solr home.

https://cwiki.apache.org/confluence/display/solr/Format+of+solr.xml#Formatofsolr.xml-Solr.xmlParameters

Because it's easy to end up with different cores sharing a data
directory, changing dataDir should be avoided unless you have good
reason, especially in solrconfig.xml, and absolute paths are a
particularly bad idea.  If you leave dataDir alone, then both
instanceDir and dataDir for each core will be in subdirectories under
/data/project/indexes.

As you probably know, SolrCloud does not use the conf directory in the
core's instanceDir, so aside from properties files (like
core.properties) data is the only information you will have for each
core -- there's no need to have it in a different location than
instanceDir, which is where core.properties lives.

I'm going against the advice I've given you in my own config and setting
dataDir in my core.properties files, but it's a relative path, so if I
move things around, the data moves too.  I'm also very aware of what's
in *every* core.properties file, and take care to ensure that they are
all correct.  I am not running SolrCloud, so each of my cores contains a
conf directory.

If moving dataDir outside of instanceDir is important to you, set it in
core.properties, and use a relative path.  When using a relative path,
dataDir is relative to instanceDir.  It defaults to ${instanceDir}/data.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Separating cores from Solr home

Jan Høydahl / Cominvent
There is also the more clean solution in https://issues.apache.org/jira/browse/SOLR-6671 (not committed).

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 4. mar. 2016 kl. 15.38 skrev Shawn Heisey <[hidden email]>:
>
> On 3/3/2016 4:12 AM, Tom Evans wrote:
>> I'm struggling to configure solr cloud to put the index files and
>> core.properties in the correct places in SolrCloud 5.5. Let me explain
>> what I am trying to achieve:
>>
>> * solr is installed in /opt/solr
>> * the user who runs solr only has read only access to that tree
>> * the solr home files - custom libraries, log4j.properties, solr.in.sh
>> and solr.xml - live in /data/project/solr/releases/<revision>, which
>> is then the target of a symlink /data/project/solr/releases/current
>> * releasing a new version of the solr home (eg adding/changing
>> libraries, changing logging options) is done by checking out a fresh
>> copy of the solr home, switching the symlink and restarting solr
>> * the solr core.properties and any data live in /data/project/indexes,
>> so they are preserved when new solr home is released
>>
>> Setting core specific dataDir with absolute paths in solrconfig.xml
>> only gets me part of the way, as the core.properties for each shard is
>> created inside the solr home.
>>
>> This is obviously no good, as when releasing a new version of the solr
>> home, they will no longer be in the current solr home.\
>
> Add the coreRootDirectory config element to your solr.xml file, so Solr
> does core discovery directly on /data/project/indexes, rather than the
> solr home.  The solr.xml and lib directory with your jars will remain in
> the solr home.
>
> https://cwiki.apache.org/confluence/display/solr/Format+of+solr.xml#Formatofsolr.xml-Solr.xmlParameters
>
> Because it's easy to end up with different cores sharing a data
> directory, changing dataDir should be avoided unless you have good
> reason, especially in solrconfig.xml, and absolute paths are a
> particularly bad idea.  If you leave dataDir alone, then both
> instanceDir and dataDir for each core will be in subdirectories under
> /data/project/indexes.
>
> As you probably know, SolrCloud does not use the conf directory in the
> core's instanceDir, so aside from properties files (like
> core.properties) data is the only information you will have for each
> core -- there's no need to have it in a different location than
> instanceDir, which is where core.properties lives.
>
> I'm going against the advice I've given you in my own config and setting
> dataDir in my core.properties files, but it's a relative path, so if I
> move things around, the data moves too.  I'm also very aware of what's
> in *every* core.properties file, and take care to ensure that they are
> all correct.  I am not running SolrCloud, so each of my cores contains a
> conf directory.
>
> If moving dataDir outside of instanceDir is important to you, set it in
> core.properties, and use a relative path.  When using a relative path,
> dataDir is relative to instanceDir.  It defaults to ${instanceDir}/data.
>
> Thanks,
> Shawn
>