[jira] [Comment Edited] (SOLR-13035) Utilize solr.data.home / solrDataHome in solr.xml to set all writable files in single directory

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Comment Edited] (SOLR-13035) Utilize solr.data.home / solrDataHome in solr.xml to set all writable files in single directory

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-13035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711533#comment-16711533 ]

Amrit Sarkar edited comment on SOLR-13035 at 12/6/18 2:48 PM:
--------------------------------------------------------------

Thanks [~janhoy] for the feedback;

> Looks scary to do SOLR_LOGS_DIR="$(pwd)/$SOLR_LOGS_DIR". It would be better to require the user to configure absolute path here.
Sure. I felt supporting relative path makes it a bit easy for deploying multiple Solr nodes on the machine/server. If  {{$(pwd)/$SOLR_LOGS_DIR}} is not the right way to do; probably {{SOLR_TIP/$SOLR_LOGS_DIR}}. The absolute path is supported as the original design.

>In general, I'd prefer of much of the logic regarding folder resolution etc was done in the common SolrCLI.java so as little logic as possible needs to be duplicated in bin/solr and bin/solr.cmd
Yeah makes sense. I followed the same resolution criteria implemented for SOLR_HOME, will move that logic to SolrCLI.java.

>While I believe solr.xml could be created if not exist, I'm not so sure we should go create SOLR_DATA_HOME if it does not exist?
Sure. Creating SOLR_DATA_HOME if not exist was again minor improvement for end-user, just like $SOLR_LOGS_DIR, though not necessary.

> I agree on the goal of making Solr more container friendly and clearly separate what paths can be made R/O and what paths typically need a separate volume mounted. Can you perhaps create a diagram that clearly shows the various directory paths we have today with their defaults and how they will be changed?
We intend not to change the default respective directory paths but instead makes it easy to run in a container environment.

SOLR_TIP -> solr installation directory
Default state:

                WRITE specific dirs from Solr/Zk node.
SOLR_TIP -----> server ------> solr ------> [data_dir] Index files
SOLR_TIP -----> server ------> solr ------> [instance_dir] Core properties
SOLR_TIP -----> server ------> solr ------> [zoo_data] Embedded ZK data
SOLR_TIP -----> server ------> [logs]

                READ specific contents in the same directory [server/solr]
SOLR_TIP -----> server ------> solr ------> solr.xml [changes requires NODE restart]
SOLR_TIP -----> server ------> solr ------> [configsets] Default config sets

For the below-stated startup command with the current patch; R/W directories will look like following;
{code}
cd $SOLR_TIP
bin/solr start -c -p 8983 -t data -l logs
{code}

                WRITE specific dirs from Solr/Zk node.
SOLR_TIP -----> [data_dir] Index files
SOLR_TIP -----> [instance_dir] Core properties
SOLR_TIP -----> [zoo_data] Embedded ZK data
SOLR_TIP -----> [logs]

                All other respective dirs would be READ specific; and changes in them requires NODE restart.

Adding support for adding solr.xml if not exist will be helpful. That way, as mentioned above pointing SOLR_HOME to an empty directory would be enough to achieve defined R/W directories. Though intention of writing the patch was to seperate directories which are created / modified after node restart. Default {{SOLR_HOME}} can still point to [SOLR_TIP]/server/solr and picks up default solr.xml.

Looking forward to feedback.


was (Author: [hidden email]):
Thanks [~janhoy] for the feedback;

> Looks scary to do SOLR_LOGS_DIR="$(pwd)/$SOLR_LOGS_DIR". It would be better to require the user to configure absolute path here.
Sure. I felt supporting relative path makes it a bit easy for deploying multiple Solr nodes on the machine/server. If  {{$(pwd)/$SOLR_LOGS_DIR}} is not the right way to do; probably {{SOLR_TIP/$SOLR_LOGS_DIR}}. The absolute path is supported as the original design.

>In general, I'd prefer of much of the logic regarding folder resolution etc was done in the common SolrCLI.java so as little logic as possible needs to be duplicated in bin/solr and bin/solr.cmd
Yeah makes sense. I followed the same resolution criteria implemented for SOLR_HOME, will move that logic to SolrCLI.java.

>While I believe solr.xml could be created if not exist, I'm not so sure we should go create SOLR_DATA_HOME if it does not exist?
Sure. Creating SOLR_DATA_HOME if not exist was again minor improvement for end-user, just like $SOLR_LOGS_DIR, though not necessary.

> I agree on the goal of making Solr more container friendly and clearly separate what paths can be made R/O and what paths typically need a separate volume mounted. Can you perhaps create a diagram that clearly shows the various directory paths we have today with their defaults and how they will be changed?
We intend not to change the default respective directory paths but instead makes it easy to run in a container environment.

SOLR_TIP -> solr installation directory
Default state:

                WRITE specific dirs from Solr/Zk node.
SOLR_TIP -----> server ------> solr ------> [data_dir] Index files
         -----> server ------> solr ------> [instance_dir] Core properties
         -----> server ------> solr ------> [zoo_data] Embedded ZK data
         -----> server ------> [logs]

                READ specific contents in the same directory [server/solr]
         -----> server ------> solr ------> solr.xml [changes requires NODE restart]
         -----> server ------> solr ------> [configsets] Default config sets

For the below-stated startup command with the current patch; R/W directories will look like following;
{code}
cd $SOLR_TIP
bin/solr start -c -p 8983 -t data -l logs
{code}

                WRITE specific dirs from Solr/Zk node.
SOLR_TIP -----> [data_dir] Index files
         -----> [instance_dir] Core properties
         -----> [zoo_data] Embedded ZK data
         -----> [logs]

         All other respective dirs would be READ specific; and changes in them requires NODE restart.

Adding support for adding solr.xml if not exist will be helpful. That way, as mentioned above pointing SOLR_HOME to an empty directory would be enough to achieve defined R/W directories. Though intention of writing the patch was to seperate directories which are created / modified after node restart. Default {{SOLR_HOME}} can still point to [SOLR_TIP]/server/solr and picks up default solr.xml.

Looking forward to feedback.

> Utilize solr.data.home / solrDataHome in solr.xml to set all writable files in single directory
> -----------------------------------------------------------------------------------------------
>
>                 Key: SOLR-13035
>                 URL: https://issues.apache.org/jira/browse/SOLR-13035
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public)
>            Reporter: Amrit Sarkar
>            Priority: Major
>         Attachments: SOLR-13035.patch, SOLR-13035.patch
>
>
> {{solr.data.home}} system property or {{solrDataHome}} in _solr.xml_ is already available as per SOLR-6671.
> The writable content in Solr are index files, core properties, and ZK data if embedded zookeeper is started in SolrCloud mode. It would be great if all writable content can come under the same directory to have separate READ-ONLY and WRITE-ONLY directories.
> It can then also solve official docker Solr image issues:
> https://github.com/docker-solr/docker-solr/issues/74
> https://github.com/docker-solr/docker-solr/issues/133



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]