Reverse-engineering existing installation

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Reverse-engineering existing installation

Doug Reeder
The documentation for SOLR is good.  However it is oriented toward setting
up a new installation, with the data model known.

I have inherited an existing installation.  Aspects of the data model I
know, but there's a lot of ways things could have been configured in SOLR,
and for some cases, I don't know what SOLR was supposed to do.

Can you reccomend any documentation on working out the configuration of an
existing installation?
Reply | Threaded
Open this post in threaded view
|

Re: Reverse-engineering existing installation

Alexandre Rafalovitch
My presentation from 2016 may be interesting as I deconstruct a Solr
example, including the tips/commands on how to do so:
https://www.slideshare.net/arafalov/rebuilding-solr-6-examples-layer-by-layer-lucenesolrrevolution-2016

The commands start around the slide 20.

Hope this helps,
    Alex.
P.s. If this (and other's ideas) is not enough, make sure to mention
the Solr version when you come back for additional questions. It may
help to know which files to recommend checking for additional hints.

On Thu, 2 May 2019 at 23:21, Doug Reeder <[hidden email]> wrote:

>
> The documentation for SOLR is good.  However it is oriented toward setting
> up a new installation, with the data model known.
>
> I have inherited an existing installation.  Aspects of the data model I
> know, but there's a lot of ways things could have been configured in SOLR,
> and for some cases, I don't know what SOLR was supposed to do.
>
> Can you reccomend any documentation on working out the configuration of an
> existing installation?
Reply | Threaded
Open this post in threaded view
|

Re: Reverse-engineering existing installation

david.w.smiley@gmail.com
In reply to this post by Doug Reeder
Consider trying to diff configs from a default at the version it was copied
from, if possible. Even better, the configs should be in source control and
then you can browse history with commentary and sometimes links to issue
trackers and code reviews.

Also a big part that you can’t see by staring at configs is what the
queries look like. You should examine the system interacting with Solr to
observe embedded comments/docs for insights.

On Thu, May 2, 2019 at 11:21 PM Doug Reeder <[hidden email]> wrote:

> The documentation for SOLR is good.  However it is oriented toward setting
> up a new installation, with the data model known.
>
> I have inherited an existing installation.  Aspects of the data model I
> know, but there's a lot of ways things could have been configured in SOLR,
> and for some cases, I don't know what SOLR was supposed to do.
>
> Can you reccomend any documentation on working out the configuration of an
> existing installation?
>
--
Sent from Gmail Mobile
Reply | Threaded
Open this post in threaded view
|

Re: Reverse-engineering existing installation

Doug Reeder
Thanks! Alexandre's presentation is helpful in understanding what's not
essential.  David's suggesting of comparing config files is good - I'll
have to see if I can dig up the config files for version 4.2, which we're
currently running.

I'll also look into updating to a supported version. I guess I'll be
reading https://lucene.apache.org/solr/guide/6_6/upgrading-solr.html and
the similar ones for later versions.  Is an upgrade guide for version 4 to
5 still around somewhere?

On Fri, May 3, 2019 at 12:21 AM David Smiley <[hidden email]>
wrote:

> Consider trying to diff configs from a default at the version it was copied
> from, if possible. Even better, the configs should be in source control and
> then you can browse history with commentary and sometimes links to issue
> trackers and code reviews.
>
> Also a big part that you can’t see by staring at configs is what the
> queries look like. You should examine the system interacting with Solr to
> observe embedded comments/docs for insights.
>
> On Thu, May 2, 2019 at 11:21 PM Doug Reeder <[hidden email]>
> wrote:
>
> > The documentation for SOLR is good.  However it is oriented toward
> setting
> > up a new installation, with the data model known.
> >
> > I have inherited an existing installation.  Aspects of the data model I
> > know, but there's a lot of ways things could have been configured in
> SOLR,
> > and for some cases, I don't know what SOLR was supposed to do.
> >
> > Can you reccomend any documentation on working out the configuration of
> an
> > existing installation?
> >
> --
> Sent from Gmail Mobile
>
Reply | Threaded
Open this post in threaded view
|

Re: Reverse-engineering existing installation

Erick Erickson
Doug:

You can pull any version of Solr from Git.

git clone https://gitbox.apache.org/repos/asf/lucene-solr.git some_local_dir

Then git will let you check out any previous branch. 4.2 is from before we switched to Git, co I’m not sure you can go that far back, but 4x is probably close enough for comparing configs.

All that said, and assuming you’re going to either 7x or 8x… I’d just think about starting over. Once you get the old configs and account for

1> any schema changes.
2> any config changes, _especially_ any custom components

Consider starting with a current version of Solr and re-indexing. You’ll absolutely _have_ to re-index _all_ your source material anyway so thinking about going from 4x->5x->6x->7x->8x is futile anyway.

Best,
Erick

> On May 3, 2019, at 12:51 PM, Doug Reeder <[hidden email]> wrote:
>
> Thanks! Alexandre's presentation is helpful in understanding what's not
> essential.  David's suggesting of comparing config files is good - I'll
> have to see if I can dig up the config files for version 4.2, which we're
> currently running.
>
> I'll also look into updating to a supported version. I guess I'll be
> reading https://lucene.apache.org/solr/guide/6_6/upgrading-solr.html and
> the similar ones for later versions.  Is an upgrade guide for version 4 to
> 5 still around somewhere?
>
> On Fri, May 3, 2019 at 12:21 AM David Smiley <[hidden email]>
> wrote:
>
>> Consider trying to diff configs from a default at the version it was copied
>> from, if possible. Even better, the configs should be in source control and
>> then you can browse history with commentary and sometimes links to issue
>> trackers and code reviews.
>>
>> Also a big part that you can’t see by staring at configs is what the
>> queries look like. You should examine the system interacting with Solr to
>> observe embedded comments/docs for insights.
>>
>> On Thu, May 2, 2019 at 11:21 PM Doug Reeder <[hidden email]>
>> wrote:
>>
>>> The documentation for SOLR is good.  However it is oriented toward
>> setting
>>> up a new installation, with the data model known.
>>>
>>> I have inherited an existing installation.  Aspects of the data model I
>>> know, but there's a lot of ways things could have been configured in
>> SOLR,
>>> and for some cases, I don't know what SOLR was supposed to do.
>>>
>>> Can you reccomend any documentation on working out the configuration of
>> an
>>> existing installation?
>>>
>> --
>> Sent from Gmail Mobile
>>

Reply | Threaded
Open this post in threaded view
|

Re: Reverse-engineering existing installation

Shawn Heisey-2
On 5/3/2019 1:44 PM, Erick Erickson wrote:
> Then git will let you check out any previous branch. 4.2 is from before we switched to Git, co I’m not sure you can go that far back, but 4x is probably close enough for comparing configs.

Git has all of Lucene's history, and most of Solr's history, back to
when Lucene and Solr were merged before the 3.1.0 release.  So the 4.x
releases are there:

--------------------
elyograg@smeagol:~/asf/lucene-solr$ git checkout releases/lucene-solr/4.2.1
Checking out files: 100% (13209/13209), done.
Note: checking out 'releases/lucene-solr/4.2.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in
this state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

   git checkout -b <new-branch-name>

HEAD is now at 50c41a3e5c Lucene Java 4.2.1 release.
--------------------

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Reverse-engineering existing installation

Doug Reeder
Thanks! Diffs for solr.xml and zoo.cfg were easy, but it looks like we'll
need to strip the comments before we can get a useful diff of
solrconfig.xml or schema.xml.  Can you recommend tools to normalize XML
files?  XMLStarlet is hosted on SourceForge, which I no longer trust, and
hasn't been updated in years.


On Fri, May 3, 2019 at 4:24 PM Shawn Heisey <[hidden email]> wrote:

> On 5/3/2019 1:44 PM, Erick Erickson wrote:
> > Then git will let you check out any previous branch. 4.2 is from before
> we switched to Git, co I’m not sure you can go that far back, but 4x is
> probably close enough for comparing configs.
>
> Git has all of Lucene's history, and most of Solr's history, back to
> when Lucene and Solr were merged before the 3.1.0 release.  So the 4.x
> releases are there:
>
> --------------------
> elyograg@smeagol:~/asf/lucene-solr$ git checkout
> releases/lucene-solr/4.2.1
> Checking out files: 100% (13209/13209), done.
> Note: checking out 'releases/lucene-solr/4.2.1'.
>
> You are in 'detached HEAD' state. You can look around, make experimental
> changes and commit them, and you can discard any commits you make in
> this state without impacting any branches by performing another checkout.
>
> If you want to create a new branch to retain commits you create, you may
> do so (now or later) by using -b with the checkout command again. Example:
>
>    git checkout -b <new-branch-name>
>
> HEAD is now at 50c41a3e5c Lucene Java 4.2.1 release.
> --------------------
>
> Thanks,
> Shawn
>
Reply | Threaded
Open this post in threaded view
|

Re: Reverse-engineering existing installation

Erick Erickson
Wait. I was recommending you diff the 4.2.1 solrconfig and the solrconfig you’re using. Ditto with the schema. If you’re trying to diff the 7x or 8x ones they’ll be totally different.

But if you are getting massive differences in the yo4.2.1 stock and what you’re using, then whoever set it up made the changes and you’ll probably have to go through them by hand, noting all the differences in the non-commented parts.

Things that are _missing_ from the one you’re using .vs. the stock distro files you can pretty much ignore. They’ll be interesting in that you can delete the equivalent from the new distro, but…

I expect the schema will be the most different, solrconfig usually doesn’t change much.

FWIW,
Erick



> On May 3, 2019, at 7:30 PM, Doug Reeder <[hidden email]> wrote:
>
> Thanks! Diffs for solr.xml and zoo.cfg were easy, but it looks like we'll
> need to strip the comments before we can get a useful diff of
> solrconfig.xml or schema.xml.  Can you recommend tools to normalize XML
> files?  XMLStarlet is hosted on SourceForge, which I no longer trust, and
> hasn't been updated in years.
>
>
> On Fri, May 3, 2019 at 4:24 PM Shawn Heisey <[hidden email]> wrote:
>
>> On 5/3/2019 1:44 PM, Erick Erickson wrote:
>>> Then git will let you check out any previous branch. 4.2 is from before
>> we switched to Git, co I’m not sure you can go that far back, but 4x is
>> probably close enough for comparing configs.
>>
>> Git has all of Lucene's history, and most of Solr's history, back to
>> when Lucene and Solr were merged before the 3.1.0 release.  So the 4.x
>> releases are there:
>>
>> --------------------
>> elyograg@smeagol:~/asf/lucene-solr$ git checkout
>> releases/lucene-solr/4.2.1
>> Checking out files: 100% (13209/13209), done.
>> Note: checking out 'releases/lucene-solr/4.2.1'.
>>
>> You are in 'detached HEAD' state. You can look around, make experimental
>> changes and commit them, and you can discard any commits you make in
>> this state without impacting any branches by performing another checkout.
>>
>> If you want to create a new branch to retain commits you create, you may
>> do so (now or later) by using -b with the checkout command again. Example:
>>
>>   git checkout -b <new-branch-name>
>>
>> HEAD is now at 50c41a3e5c Lucene Java 4.2.1 release.
>> --------------------
>>
>> Thanks,
>> Shawn
>>

Reply | Threaded
Open this post in threaded view
|

Re: Reverse-engineering existing installation

Alexandre Rafalovitch
In reply to this post by Doug Reeder
XMLStarlet still works just fine. So if you want the fast way, that is the
one.

Otherwise, some xml editors can do it (not sure which ones) or you can look
for XSLT or XQuery examples on the web.

XMLStarlet actually just spits out XSLT internally, or even externally if
you ask.

Regards,
     Alex


On Fri, May 3, 2019, 10:30 PM Doug Reeder, <[hidden email]> wrote:

> Thanks! Diffs for solr.xml and zoo.cfg were easy, but it looks like we'll
> need to strip the comments before we can get a useful diff of
> solrconfig.xml or schema.xml.  Can you recommend tools to normalize XML
> files?  XMLStarlet is hosted on SourceForge, which I no longer trust, and
> hasn't been updated in years.
>
>
> On Fri, May 3, 2019 at 4:24 PM Shawn Heisey <[hidden email]> wrote:
>
> > On 5/3/2019 1:44 PM, Erick Erickson wrote:
> > > Then git will let you check out any previous branch. 4.2 is from before
> > we switched to Git, co I’m not sure you can go that far back, but 4x is
> > probably close enough for comparing configs.
> >
> > Git has all of Lucene's history, and most of Solr's history, back to
> > when Lucene and Solr were merged before the 3.1.0 release.  So the 4.x
> > releases are there:
> >
> > --------------------
> > elyograg@smeagol:~/asf/lucene-solr$ git checkout
> > releases/lucene-solr/4.2.1
> > Checking out files: 100% (13209/13209), done.
> > Note: checking out 'releases/lucene-solr/4.2.1'.
> >
> > You are in 'detached HEAD' state. You can look around, make experimental
> > changes and commit them, and you can discard any commits you make in
> > this state without impacting any branches by performing another checkout.
> >
> > If you want to create a new branch to retain commits you create, you may
> > do so (now or later) by using -b with the checkout command again.
> Example:
> >
> >    git checkout -b <new-branch-name>
> >
> > HEAD is now at 50c41a3e5c Lucene Java 4.2.1 release.
> > --------------------
> >
> > Thanks,
> > Shawn
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Reverse-engineering existing installation

Doug Reeder
Thanks, xmlstarlet makes it straightforward to get the canonical XML.

It looks like our schema.xml files are rather different from files
like solr/example/solr/collection1/conf/schema.xml

Any suggestions of sections I should focus on?

On Sat, May 4, 2019 at 8:11 AM Alexandre Rafalovitch <[hidden email]>
wrote:

> XMLStarlet still works just fine. So if you want the fast way, that is the
> one.
>
> Otherwise, some xml editors can do it (not sure which ones) or you can look
> for XSLT or XQuery examples on the web.
>
> XMLStarlet actually just spits out XSLT internally, or even externally if
> you ask.
>
> Regards,
>      Alex
>
>
> On Fri, May 3, 2019, 10:30 PM Doug Reeder, <[hidden email]>
> wrote:
>
> > Thanks! Diffs for solr.xml and zoo.cfg were easy, but it looks like we'll
> > need to strip the comments before we can get a useful diff of
> > solrconfig.xml or schema.xml.  Can you recommend tools to normalize XML
> > files?  XMLStarlet is hosted on SourceForge, which I no longer trust, and
> > hasn't been updated in years.
> >
> >
> > On Fri, May 3, 2019 at 4:24 PM Shawn Heisey <[hidden email]> wrote:
> >
> > > On 5/3/2019 1:44 PM, Erick Erickson wrote:
> > > > Then git will let you check out any previous branch. 4.2 is from
> before
> > > we switched to Git, co I’m not sure you can go that far back, but 4x is
> > > probably close enough for comparing configs.
> > >
> > > Git has all of Lucene's history, and most of Solr's history, back to
> > > when Lucene and Solr were merged before the 3.1.0 release.  So the 4.x
> > > releases are there:
> > >
> > > --------------------
> > > elyograg@smeagol:~/asf/lucene-solr$ git checkout
> > > releases/lucene-solr/4.2.1
> > > Checking out files: 100% (13209/13209), done.
> > > Note: checking out 'releases/lucene-solr/4.2.1'.
> > >
> > > You are in 'detached HEAD' state. You can look around, make
> experimental
> > > changes and commit them, and you can discard any commits you make in
> > > this state without impacting any branches by performing another
> checkout.
> > >
> > > If you want to create a new branch to retain commits you create, you
> may
> > > do so (now or later) by using -b with the checkout command again.
> > Example:
> > >
> > >    git checkout -b <new-branch-name>
> > >
> > > HEAD is now at 50c41a3e5c Lucene Java 4.2.1 release.
> > > --------------------
> > >
> > > Thanks,
> > > Shawn
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Reverse-engineering existing installation

Erick Erickson
Unfortunately…everything. You may have to compare tag-by-tag, especially in solrconfig.xml. In the schema, all your fieldTypes and the associated fields are critical….



> On May 6, 2019, at 11:25 AM, Doug Reeder <[hidden email]> wrote:
>
> Thanks, xmlstarlet makes it straightforward to get the canonical XML.
>
> It looks like our schema.xml files are rather different from files
> like solr/example/solr/collection1/conf/schema.xml
>
> Any suggestions of sections I should focus on?
>
> On Sat, May 4, 2019 at 8:11 AM Alexandre Rafalovitch <[hidden email]>
> wrote:
>
>> XMLStarlet still works just fine. So if you want the fast way, that is the
>> one.
>>
>> Otherwise, some xml editors can do it (not sure which ones) or you can look
>> for XSLT or XQuery examples on the web.
>>
>> XMLStarlet actually just spits out XSLT internally, or even externally if
>> you ask.
>>
>> Regards,
>>     Alex
>>
>>
>> On Fri, May 3, 2019, 10:30 PM Doug Reeder, <[hidden email]>
>> wrote:
>>
>>> Thanks! Diffs for solr.xml and zoo.cfg were easy, but it looks like we'll
>>> need to strip the comments before we can get a useful diff of
>>> solrconfig.xml or schema.xml.  Can you recommend tools to normalize XML
>>> files?  XMLStarlet is hosted on SourceForge, which I no longer trust, and
>>> hasn't been updated in years.
>>>
>>>
>>> On Fri, May 3, 2019 at 4:24 PM Shawn Heisey <[hidden email]> wrote:
>>>
>>>> On 5/3/2019 1:44 PM, Erick Erickson wrote:
>>>>> Then git will let you check out any previous branch. 4.2 is from
>> before
>>>> we switched to Git, co I’m not sure you can go that far back, but 4x is
>>>> probably close enough for comparing configs.
>>>>
>>>> Git has all of Lucene's history, and most of Solr's history, back to
>>>> when Lucene and Solr were merged before the 3.1.0 release.  So the 4.x
>>>> releases are there:
>>>>
>>>> --------------------
>>>> elyograg@smeagol:~/asf/lucene-solr$ git checkout
>>>> releases/lucene-solr/4.2.1
>>>> Checking out files: 100% (13209/13209), done.
>>>> Note: checking out 'releases/lucene-solr/4.2.1'.
>>>>
>>>> You are in 'detached HEAD' state. You can look around, make
>> experimental
>>>> changes and commit them, and you can discard any commits you make in
>>>> this state without impacting any branches by performing another
>> checkout.
>>>>
>>>> If you want to create a new branch to retain commits you create, you
>> may
>>>> do so (now or later) by using -b with the checkout command again.
>>> Example:
>>>>
>>>>   git checkout -b <new-branch-name>
>>>>
>>>> HEAD is now at 50c41a3e5c Lucene Java 4.2.1 release.
>>>> --------------------
>>>>
>>>> Thanks,
>>>> Shawn
>>>>
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Reverse-engineering existing installation

Alexandre Rafalovitch
In reply to this post by Doug Reeder
I would focus on fields not being used. Then, on types not used. Then, you
will see what was actually custom to your setup.

In solrconfig.xml, I would focus on request handlers and maybe defaults
used.

Regards,
     Alex

On Mon, May 6, 2019, 2:25 PM Doug Reeder, <[hidden email]> wrote:

> Thanks, xmlstarlet makes it straightforward to get the canonical XML.
>
> It looks like our schema.xml files are rather different from files
> like solr/example/solr/collection1/conf/schema.xml
>
> Any suggestions of sections I should focus on?
>
> On Sat, May 4, 2019 at 8:11 AM Alexandre Rafalovitch <[hidden email]>
> wrote:
>
> > XMLStarlet still works just fine. So if you want the fast way, that is
> the
> > one.
> >
> > Otherwise, some xml editors can do it (not sure which ones) or you can
> look
> > for XSLT or XQuery examples on the web.
> >
> > XMLStarlet actually just spits out XSLT internally, or even externally if
> > you ask.
> >
> > Regards,
> >      Alex
> >
> >
> > On Fri, May 3, 2019, 10:30 PM Doug Reeder, <[hidden email]>
> > wrote:
> >
> > > Thanks! Diffs for solr.xml and zoo.cfg were easy, but it looks like
> we'll
> > > need to strip the comments before we can get a useful diff of
> > > solrconfig.xml or schema.xml.  Can you recommend tools to normalize XML
> > > files?  XMLStarlet is hosted on SourceForge, which I no longer trust,
> and
> > > hasn't been updated in years.
> > >
> > >
> > > On Fri, May 3, 2019 at 4:24 PM Shawn Heisey <[hidden email]>
> wrote:
> > >
> > > > On 5/3/2019 1:44 PM, Erick Erickson wrote:
> > > > > Then git will let you check out any previous branch. 4.2 is from
> > before
> > > > we switched to Git, co I’m not sure you can go that far back, but 4x
> is
> > > > probably close enough for comparing configs.
> > > >
> > > > Git has all of Lucene's history, and most of Solr's history, back to
> > > > when Lucene and Solr were merged before the 3.1.0 release.  So the
> 4.x
> > > > releases are there:
> > > >
> > > > --------------------
> > > > elyograg@smeagol:~/asf/lucene-solr$ git checkout
> > > > releases/lucene-solr/4.2.1
> > > > Checking out files: 100% (13209/13209), done.
> > > > Note: checking out 'releases/lucene-solr/4.2.1'.
> > > >
> > > > You are in 'detached HEAD' state. You can look around, make
> > experimental
> > > > changes and commit them, and you can discard any commits you make in
> > > > this state without impacting any branches by performing another
> > checkout.
> > > >
> > > > If you want to create a new branch to retain commits you create, you
> > may
> > > > do so (now or later) by using -b with the checkout command again.
> > > Example:
> > > >
> > > >    git checkout -b <new-branch-name>
> > > >
> > > > HEAD is now at 50c41a3e5c Lucene Java 4.2.1 release.
> > > > --------------------
> > > >
> > > > Thanks,
> > > > Shawn
> > > >
> > >
> >
>