Miserable Experience Using Solr. Again.

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Miserable Experience Using Solr. Again.

Aaron Greenspan-2
Hi,

I have been on this list for some time because I know that any time I try to do anything related to Solr I’m going to have to spend hours on it, wondering why everything has to be so awful, and I just want somewhere to provide feedback with the dim hope that the product might improve one day. (So far, for my purposes, it hasn’t.) Sure enough, I still absolutely hate using Solr, and I have more feedback.

I started with a confusing error on the web console, which I still can’t figure out how to password protect without going through an insanely process involving "ZooKeeper," which I don’t know anything about, or have, to the best of my knowledge:

Problem accessing /solr/. Reason:

    Forbidden

According to logs, this apparently meant that a MySQL query had failed due to a field name change. Since I would have to change my XML configuration files, I decided to use the opportunity to upgrade from Solr 5.1.4 to 6.2.0. It broke everything.

First I was getting errors about "Unsupported major.minor version 52.0", so I needed to install the Linux x64 JRE 1.8.0, which I managed on CentOS 6 with...

yum install openjdk-1.8.0

...going to Oracle’s web site, downloading the latest JRE 1.8 build, and then running...

yum localinstall jre-8u101-linux-x64.rpm

So far so good. But I didn’t have JAVA_HOME set properly apparently, so I needed to do the not-exactly-intuitive…

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre/

As usual, I manually moved over my mysql-connector-java-5.1.38-bin.jar file from the dist/ folder in the old version to the new one. Then after stopping the old process (with kill -9, since there seems to be no graceful way to shut down Solr—README.txt doesn’t mention bin/solr stop) I moved over my two core folders from the old server/solr/ folder. I tried to start it up with bin/solr start, and watched the errors roll in.

There was some kind of problem with StopFilterFactory and the text_general field type. Thanks to Stack Overflow I was able to determine that the apparent problem was that there was a parameter, previously fine, which was no longer fine. So I removed all instances of enablePositionIncrements="true". That helped, but then I ran into a broader error: "Plugin Initializing failure for [schema.xml] fieldType". It didn’t say which field type. Buried in the logs I found a reference in the Java stack trace—which *disappears* (and distorts the viewing window horribly) after a few seconds when you try to view it in the web log UI—to the string "units="degrees"". Sure enough, this string appeared in my schema.xml for a class called "solr.SpatialRecursivePrefixTreeFieldType" that I’m pretty sure I never use. I removed that parameter, and moved on to the next set of errors.

Apparently there is some aspect of the Thai text field type that Solr 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.

Now Solr was complaining about "Error loading class 'solr.admin.AdminHandlers'". So I found the reference to solr.admin.AdminHandlers in solrconfig.xml for each of my cores and commented it out. Only then did Solr work again.

This was not a smooth process. It took about two hours. The user interface is still as buggy as an early alpha of most products, the errors are difficult to understand when they don’t actually specify what’s wrong (and they almost never do), and there should have been an automatic process to highlight and fix problems in old (pre-6) configuration files. Never mind the fact that the XML-based configuration process is an antiquated nightmare when the rest of the world has long since moved onto databases.

Maybe this will help someone else out there.

Aaron

PlainSite | http://www.plainsite.org
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

JohnB
For what it's worth - I found enough frustration upgrading that I decided
to "upgrade by replacement"

Now, I suppose if you've got a huge dataset to re-index that could be a
problem, but just in case an option like that helps you, I'll suggest this.

1. Install 6.x on a new machine using the "install for production"
instructions
2. Use the configs from one of the sample projects to create an
appropriately-named collection
3. Use the ability to "include" your configs into the other configs (they
live in separate files)
          I can provide more help here if you're interested
4. Re-index all your data into the new version of SOLR...

I have rough, but useable docs on this if you are interested in attempting
this approach.

On Mon, Sep 12, 2016 at 3:48 PM, Aaron Greenspan <
[hidden email]> wrote:

> Hi,
>
> I have been on this list for some time because I know that any time I try
> to do anything related to Solr I’m going to have to spend hours on it,
> wondering why everything has to be so awful, and I just want somewhere to
> provide feedback with the dim hope that the product might improve one day.
> (So far, for my purposes, it hasn’t.) Sure enough, I still absolutely hate
> using Solr, and I have more feedback.
>
> I started with a confusing error on the web console, which I still can’t
> figure out how to password protect without going through an insanely
> process involving "ZooKeeper," which I don’t know anything about, or have,
> to the best of my knowledge:
>
> Problem accessing /solr/. Reason:
>
>     Forbidden
>
> According to logs, this apparently meant that a MySQL query had failed due
> to a field name change. Since I would have to change my XML configuration
> files, I decided to use the opportunity to upgrade from Solr 5.1.4 to
> 6.2.0. It broke everything.
>
> First I was getting errors about "Unsupported major.minor version 52.0",
> so I needed to install the Linux x64 JRE 1.8.0, which I managed on CentOS 6
> with...
>
> yum install openjdk-1.8.0
>
> ...going to Oracle’s web site, downloading the latest JRE 1.8 build, and
> then running...
>
> yum localinstall jre-8u101-linux-x64.rpm
>
> So far so good. But I didn’t have JAVA_HOME set properly apparently, so I
> needed to do the not-exactly-intuitive…
>
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.
> el6_8.x86_64/jre/
>
> As usual, I manually moved over my mysql-connector-java-5.1.38-bin.jar
> file from the dist/ folder in the old version to the new one. Then after
> stopping the old process (with kill -9, since there seems to be no graceful
> way to shut down Solr—README.txt doesn’t mention bin/solr stop) I moved
> over my two core folders from the old server/solr/ folder. I tried to start
> it up with bin/solr start, and watched the errors roll in.
>
> There was some kind of problem with StopFilterFactory and the text_general
> field type. Thanks to Stack Overflow I was able to determine that the
> apparent problem was that there was a parameter, previously fine, which was
> no longer fine. So I removed all instances of enablePositionIncrements="true".
> That helped, but then I ran into a broader error: "Plugin Initializing
> failure for [schema.xml] fieldType". It didn’t say which field type. Buried
> in the logs I found a reference in the Java stack trace—which *disappears*
> (and distorts the viewing window horribly) after a few seconds when you try
> to view it in the web log UI—to the string "units="degrees"". Sure enough,
> this string appeared in my schema.xml for a class called "solr.
> SpatialRecursivePrefixTreeFieldType" that I’m pretty sure I never use. I
> removed that parameter, and moved on to the next set of errors.
>
> Apparently there is some aspect of the Thai text field type that Solr
> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.
>
> Now Solr was complaining about "Error loading class
> 'solr.admin.AdminHandlers'". So I found the reference to
> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and
> commented it out. Only then did Solr work again.
>
> This was not a smooth process. It took about two hours. The user interface
> is still as buggy as an early alpha of most products, the errors are
> difficult to understand when they don’t actually specify what’s wrong (and
> they almost never do), and there should have been an automatic process to
> highlight and fix problems in old (pre-6) configuration files. Never mind
> the fact that the XML-based configuration process is an antiquated
> nightmare when the rest of the world has long since moved onto databases.
>
> Maybe this will help someone else out there.
>
> Aaron
>
> PlainSite | http://www.plainsite.org
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

JohnB
I would also add that dealing with Java versions has always been a pain
until you get used to the whole "JAVA HOME" thing, but that this isn't
anything to do with SOLR per se - it's just part and parcel of dealing with
open source software that uses Java...

Big changes between major versions of any software are common - there is
some documentation here that may help...

https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+5+to+Solr+6

It was reading this that made me decide to try "upgrade by replace" to
avoid the whole "update" issue entirely - although I also had to upgrade
Java on my VMs...

On Mon, Sep 12, 2016 at 4:05 PM, John Bickerstaff <[hidden email]>
wrote:

> For what it's worth - I found enough frustration upgrading that I decided
> to "upgrade by replacement"
>
> Now, I suppose if you've got a huge dataset to re-index that could be a
> problem, but just in case an option like that helps you, I'll suggest this.
>
> 1. Install 6.x on a new machine using the "install for production"
> instructions
> 2. Use the configs from one of the sample projects to create an
> appropriately-named collection
> 3. Use the ability to "include" your configs into the other configs (they
> live in separate files)
>           I can provide more help here if you're interested
> 4. Re-index all your data into the new version of SOLR...
>
> I have rough, but useable docs on this if you are interested in attempting
> this approach.
>
> On Mon, Sep 12, 2016 at 3:48 PM, Aaron Greenspan <
> [hidden email]> wrote:
>
>> Hi,
>>
>> I have been on this list for some time because I know that any time I try
>> to do anything related to Solr I’m going to have to spend hours on it,
>> wondering why everything has to be so awful, and I just want somewhere to
>> provide feedback with the dim hope that the product might improve one day.
>> (So far, for my purposes, it hasn’t.) Sure enough, I still absolutely hate
>> using Solr, and I have more feedback.
>>
>> I started with a confusing error on the web console, which I still can’t
>> figure out how to password protect without going through an insanely
>> process involving "ZooKeeper," which I don’t know anything about, or have,
>> to the best of my knowledge:
>>
>> Problem accessing /solr/. Reason:
>>
>>     Forbidden
>>
>> According to logs, this apparently meant that a MySQL query had failed
>> due to a field name change. Since I would have to change my XML
>> configuration files, I decided to use the opportunity to upgrade from Solr
>> 5.1.4 to 6.2.0. It broke everything.
>>
>> First I was getting errors about "Unsupported major.minor version 52.0",
>> so I needed to install the Linux x64 JRE 1.8.0, which I managed on CentOS 6
>> with...
>>
>> yum install openjdk-1.8.0
>>
>> ...going to Oracle’s web site, downloading the latest JRE 1.8 build, and
>> then running...
>>
>> yum localinstall jre-8u101-linux-x64.rpm
>>
>> So far so good. But I didn’t have JAVA_HOME set properly apparently, so I
>> needed to do the not-exactly-intuitive…
>>
>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el
>> 6_8.x86_64/jre/
>>
>> As usual, I manually moved over my mysql-connector-java-5.1.38-bin.jar
>> file from the dist/ folder in the old version to the new one. Then after
>> stopping the old process (with kill -9, since there seems to be no graceful
>> way to shut down Solr—README.txt doesn’t mention bin/solr stop) I moved
>> over my two core folders from the old server/solr/ folder. I tried to start
>> it up with bin/solr start, and watched the errors roll in.
>>
>> There was some kind of problem with StopFilterFactory and the
>> text_general field type. Thanks to Stack Overflow I was able to determine
>> that the apparent problem was that there was a parameter, previously fine,
>> which was no longer fine. So I removed all instances of
>> enablePositionIncrements="true". That helped, but then I ran into a
>> broader error: "Plugin Initializing failure for [schema.xml] fieldType". It
>> didn’t say which field type. Buried in the logs I found a reference in the
>> Java stack trace—which *disappears* (and distorts the viewing window
>> horribly) after a few seconds when you try to view it in the web log UI—to
>> the string "units="degrees"". Sure enough, this string appeared in my
>> schema.xml for a class called "solr.SpatialRecursivePrefixTreeFieldType"
>> that I’m pretty sure I never use. I removed that parameter, and moved on to
>> the next set of errors.
>>
>> Apparently there is some aspect of the Thai text field type that Solr
>> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.
>>
>> Now Solr was complaining about "Error loading class
>> 'solr.admin.AdminHandlers'". So I found the reference to
>> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and
>> commented it out. Only then did Solr work again.
>>
>> This was not a smooth process. It took about two hours. The user
>> interface is still as buggy as an early alpha of most products, the errors
>> are difficult to understand when they don’t actually specify what’s wrong
>> (and they almost never do), and there should have been an automatic process
>> to highlight and fix problems in old (pre-6) configuration files. Never
>> mind the fact that the XML-based configuration process is an antiquated
>> nightmare when the rest of the world has long since moved onto databases.
>>
>> Maybe this will help someone else out there.
>>
>> Aaron
>>
>> PlainSite | http://www.plainsite.org
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience ..... Again.

Erik Hatcher-4
In reply to this post by Aaron Greenspan-2
Aaron - I for one sympathize.  When I pause to think of the stacks upon stacks of technologies that something like Solr are built upon… my head spins and I feel for the folks coming to computer science these days and having the whole Java and Big Data stacks and all that goes along with that (JVM/mem/GC up to network topology and architecture with 3xZK, plus NxM Solr’s, and beyond to data modeling, schema design, and query parameter adjusting).

---

It’s good for us to hear the ugly/painful side of folks experiences.  It’s driven us to to where I find myself iterating with Solr in my day job like this….

   $ bin/solr create -c my_collection
   $ bin/post -c my_collection /data/docs.json

and http://… /select?q=…&wt=csv…

So “it works for me”, but that’s not a nice way to approach the struggles of users.   Though we’ve come a long way, we’ve got a ways to go as well.

        Erik

p.s. -

> Never mind the fact that the XML-based configuration process is an antiquated nightmare when the rest of the world has long since moved onto databases.

Well, to that point - the world that I work in really boils down to at least plain text (alas, mostly JSON these days, but even that’s an implementation detail) stuffed into git repositories, and played into new Solr environments by uploading configuration files, or more modernly, hitting the Solr configuration API’s to add/configure fields, set up request handlers, and the basics of what needs to be done.  No XML needed these days.   No (relational, JDBC) databases either, for that matter :)

> Maybe this will help someone else out there.

Thanks for taking the time to detail your struggles to the community.  It is helpful to see where the rough edges are in this whole business, and smoothing them out.   But it’s no easy business, having these stacks of dependencies and complexities piled on top of one another and trying to get it all fired up properly and usably.

        Erik

Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience ..... Again.

Joel Bernstein
I'm currently working on upgrading Alfresco from Solr 6.0 to Solr 6.2.
Should be easy. Think again. Lucene analyzer changes between Solr 6.0 and
Solr 6.2 and a new assert in ConjunctionDISI have caused days of work to
perform this simple upgrade.

Joel Bernstein
http://joelsolr.blogspot.com/

On Mon, Sep 12, 2016 at 7:05 PM, Erik Hatcher <[hidden email]>
wrote:

> Aaron - I for one sympathize.  When I pause to think of the stacks upon
> stacks of technologies that something like Solr are built upon… my head
> spins and I feel for the folks coming to computer science these days and
> having the whole Java and Big Data stacks and all that goes along with that
> (JVM/mem/GC up to network topology and architecture with 3xZK, plus NxM
> Solr’s, and beyond to data modeling, schema design, and query parameter
> adjusting).
>
> ---
>
> It’s good for us to hear the ugly/painful side of folks experiences.  It’s
> driven us to to where I find myself iterating with Solr in my day job like
> this….
>
>    $ bin/solr create -c my_collection
>    $ bin/post -c my_collection /data/docs.json
>
> and http://… /select?q=…&wt=csv…
>
> So “it works for me”, but that’s not a nice way to approach the struggles
> of users.   Though we’ve come a long way, we’ve got a ways to go as well.
>
>         Erik
>
> p.s. -
>
> > Never mind the fact that the XML-based configuration process is an
> antiquated nightmare when the rest of the world has long since moved onto
> databases.
>
> Well, to that point - the world that I work in really boils down to at
> least plain text (alas, mostly JSON these days, but even that’s an
> implementation detail) stuffed into git repositories, and played into new
> Solr environments by uploading configuration files, or more modernly,
> hitting the Solr configuration API’s to add/configure fields, set up
> request handlers, and the basics of what needs to be done.  No XML needed
> these days.   No (relational, JDBC) databases either, for that matter :)
>
> > Maybe this will help someone else out there.
>
> Thanks for taking the time to detail your struggles to the community.  It
> is helpful to see where the rough edges are in this whole business, and
> smoothing them out.   But it’s no easy business, having these stacks of
> dependencies and complexities piled on top of one another and trying to get
> it all fired up properly and usably.
>
>         Erik
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience ..... Again.

Bradley Belyeu
In reply to this post by Erik Hatcher-4
I agree with what Erik wrote and some of Aaron’s original post. I’m relatively new to the Solr system (yes, pun intended) having just started diving into it a little over a year ago. I was in the “envious” position of being the only person who wanted to learn it and support it after our previous “expert” left the team. :D

It recently took me a full two week sprint cycle to upgrade our old style master-slaves cluster from 4.3 to 5.1 (we simultaneously went from java 1.7 to 1.8 and updated our NewRelic wrapper). There were many pain points along the way especially since I’m not much of a Java developer (I’m a Python/PHP/JS guy). BUT that being said I learned such an incredible amount over that sprint about the JVM, Solr configs & classes, caching & index tuning, and Lucene itself that I wouldn’t have changed it a bit. Ok, that’s a lie, I would have fixed a problem with NGrams before it caused a couple hour production outage. But basically, the amount of knowledge gained was very well worth the time and effort put into working through the upgrade pain points.

@Aaron another thing I’ve learned over the years is to only change 1 thing at a time, and definitely don’t mix troubleshooting with upgrading.
https://en.wikibooks.org/wiki/Computer_Programming_Principles/Maintaining/Debugging#Change_one_thing_at_a_time

One thing I don’t understand from your email is when you said you were “wondering why everything has to be so awful” about Solr. The initial problems you described were related to a MySQL database change and issues with Zookeeper. I think choosing to do the Java updates and Solr updates are what led to the other issues. The Solr docs packaged with the release do a fairly good job of explaining the breaking changes with each version. I chose to do all my updates one minor version at a time so I could keep up with the changes in the docs (4.3->4.4->…4.10->5.0->5.1) which took longer but made troubleshooting much easier.

As far as the user interface goes, at least it has one unlike some other popular search tools (cough ElasticSearch cough). And as far as XML based configuration is concerned, I personally prefer it to scripts with JSON blobs and Rest calls to set up your collections/entities/docs/etc. But that’s just b/c I’m weird.

I do agree that the error messages are often not helpful, BUT I found it easiest to just look at the source code to find the root cause of the issue. Which is what I think this boils down to, is that rather than complain, we should work to make Solr what we want it to be. If you can identify an issue, then you can either solve it yourself with a pull request or create a Jira issue asking for help to get it fixed.
Here’s a quote I love from Maya Angelou, “What you're supposed to do when you don't like a thing is change it. If you can't change it, change the way you think about it. Don't complain.”


On 9/12/16, 6:05 PM, "Erik Hatcher" <[hidden email]> wrote:

    Aaron - I for one sympathize.  When I pause to think of the stacks upon stacks of technologies that something like Solr are built upon… my head spins and I feel for the folks coming to computer science these days and having the whole Java and Big Data stacks and all that goes along with that (JVM/mem/GC up to network topology and architecture with 3xZK, plus NxM Solr’s, and beyond to data modeling, schema design, and query parameter adjusting).
   
    ---
   
    It’s good for us to hear the ugly/painful side of folks experiences.  It’s driven us to to where I find myself iterating with Solr in my day job like this….
   
       $ bin/solr create -c my_collection
       $ bin/post -c my_collection /data/docs.json
   
    and http://… /select?q=…&wt=csv…
   
    So “it works for me”, but that’s not a nice way to approach the struggles of users.   Though we’ve come a long way, we’ve got a ways to go as well.
   
    Erik
   
    p.s. -
   
    > Never mind the fact that the XML-based configuration process is an antiquated nightmare when the rest of the world has long since moved onto databases.
   
    Well, to that point - the world that I work in really boils down to at least plain text (alas, mostly JSON these days, but even that’s an implementation detail) stuffed into git repositories, and played into new Solr environments by uploading configuration files, or more modernly, hitting the Solr configuration API’s to add/configure fields, set up request handlers, and the basics of what needs to be done.  No XML needed these days.   No (relational, JDBC) databases either, for that matter :)
   
    > Maybe this will help someone else out there.
   
    Thanks for taking the time to detail your struggles to the community.  It is helpful to see where the rough edges are in this whole business, and smoothing them out.   But it’s no easy business, having these stacks of dependencies and complexities piled on top of one another and trying to get it all fired up properly and usably.
   
    Erik
   
   


Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Billnbell
In reply to this post by JohnB
Interested for sure

Bill Bell
Sent from mobile


> On Sep 12, 2016, at 4:05 PM, John Bickerstaff <[hidden email]> wrote:
>
> For what it's worth - I found enough frustration upgrading that I decided
> to "upgrade by replacement"
>
> Now, I suppose if you've got a huge dataset to re-index that could be a
> problem, but just in case an option like that helps you, I'll suggest this.
>
> 1. Install 6.x on a new machine using the "install for production"
> instructions
> 2. Use the configs from one of the sample projects to create an
> appropriately-named collection
> 3. Use the ability to "include" your configs into the other configs (they
> live in separate files)
>          I can provide more help here if you're interested
> 4. Re-index all your data into the new version of SOLR...
>
> I have rough, but useable docs on this if you are interested in attempting
> this approach.
>
> On Mon, Sep 12, 2016 at 3:48 PM, Aaron Greenspan <
> [hidden email]> wrote:
>
>> Hi,
>>
>> I have been on this list for some time because I know that any time I try
>> to do anything related to Solr I’m going to have to spend hours on it,
>> wondering why everything has to be so awful, and I just want somewhere to
>> provide feedback with the dim hope that the product might improve one day.
>> (So far, for my purposes, it hasn’t.) Sure enough, I still absolutely hate
>> using Solr, and I have more feedback.
>>
>> I started with a confusing error on the web console, which I still can’t
>> figure out how to password protect without going through an insanely
>> process involving "ZooKeeper," which I don’t know anything about, or have,
>> to the best of my knowledge:
>>
>> Problem accessing /solr/. Reason:
>>
>>    Forbidden
>>
>> According to logs, this apparently meant that a MySQL query had failed due
>> to a field name change. Since I would have to change my XML configuration
>> files, I decided to use the opportunity to upgrade from Solr 5.1.4 to
>> 6.2.0. It broke everything.
>>
>> First I was getting errors about "Unsupported major.minor version 52.0",
>> so I needed to install the Linux x64 JRE 1.8.0, which I managed on CentOS 6
>> with...
>>
>> yum install openjdk-1.8.0
>>
>> ...going to Oracle’s web site, downloading the latest JRE 1.8 build, and
>> then running...
>>
>> yum localinstall jre-8u101-linux-x64.rpm
>>
>> So far so good. But I didn’t have JAVA_HOME set properly apparently, so I
>> needed to do the not-exactly-intuitive…
>>
>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.
>> el6_8.x86_64/jre/
>>
>> As usual, I manually moved over my mysql-connector-java-5.1.38-bin.jar
>> file from the dist/ folder in the old version to the new one. Then after
>> stopping the old process (with kill -9, since there seems to be no graceful
>> way to shut down Solr—README.txt doesn’t mention bin/solr stop) I moved
>> over my two core folders from the old server/solr/ folder. I tried to start
>> it up with bin/solr start, and watched the errors roll in.
>>
>> There was some kind of problem with StopFilterFactory and the text_general
>> field type. Thanks to Stack Overflow I was able to determine that the
>> apparent problem was that there was a parameter, previously fine, which was
>> no longer fine. So I removed all instances of enablePositionIncrements="true".
>> That helped, but then I ran into a broader error: "Plugin Initializing
>> failure for [schema.xml] fieldType". It didn’t say which field type. Buried
>> in the logs I found a reference in the Java stack trace—which *disappears*
>> (and distorts the viewing window horribly) after a few seconds when you try
>> to view it in the web log UI—to the string "units="degrees"". Sure enough,
>> this string appeared in my schema.xml for a class called "solr.
>> SpatialRecursivePrefixTreeFieldType" that I’m pretty sure I never use. I
>> removed that parameter, and moved on to the next set of errors.
>>
>> Apparently there is some aspect of the Thai text field type that Solr
>> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.
>>
>> Now Solr was complaining about "Error loading class
>> 'solr.admin.AdminHandlers'". So I found the reference to
>> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and
>> commented it out. Only then did Solr work again.
>>
>> This was not a smooth process. It took about two hours. The user interface
>> is still as buggy as an early alpha of most products, the errors are
>> difficult to understand when they don’t actually specify what’s wrong (and
>> they almost never do), and there should have been an automatic process to
>> highlight and fix problems in old (pre-6) configuration files. Never mind
>> the fact that the XML-based configuration process is an antiquated
>> nightmare when the rest of the world has long since moved onto databases.
>>
>> Maybe this will help someone else out there.
>>
>> Aaron
>>
>> PlainSite | http://www.plainsite.org
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

JohnB
Sure - ping me off the list and I'll send my text file docs.

They're rough and (of course) focused on what I'm doing, but they just
might relieve some of the pain.

Caveat - all on Linux and command line - no Admin UI api's -- I like the
feel of the command line so I use it.

On Mon, Sep 12, 2016 at 8:41 PM, <[hidden email]> wrote:

> Interested for sure
>
> Bill Bell
> Sent from mobile
>
>
> > On Sep 12, 2016, at 4:05 PM, John Bickerstaff <[hidden email]>
> wrote:
> >
> > For what it's worth - I found enough frustration upgrading that I decided
> > to "upgrade by replacement"
> >
> > Now, I suppose if you've got a huge dataset to re-index that could be a
> > problem, but just in case an option like that helps you, I'll suggest
> this.
> >
> > 1. Install 6.x on a new machine using the "install for production"
> > instructions
> > 2. Use the configs from one of the sample projects to create an
> > appropriately-named collection
> > 3. Use the ability to "include" your configs into the other configs (they
> > live in separate files)
> >          I can provide more help here if you're interested
> > 4. Re-index all your data into the new version of SOLR...
> >
> > I have rough, but useable docs on this if you are interested in
> attempting
> > this approach.
> >
> > On Mon, Sep 12, 2016 at 3:48 PM, Aaron Greenspan <
> > [hidden email]> wrote:
> >
> >> Hi,
> >>
> >> I have been on this list for some time because I know that any time I
> try
> >> to do anything related to Solr I’m going to have to spend hours on it,
> >> wondering why everything has to be so awful, and I just want somewhere
> to
> >> provide feedback with the dim hope that the product might improve one
> day.
> >> (So far, for my purposes, it hasn’t.) Sure enough, I still absolutely
> hate
> >> using Solr, and I have more feedback.
> >>
> >> I started with a confusing error on the web console, which I still can’t
> >> figure out how to password protect without going through an insanely
> >> process involving "ZooKeeper," which I don’t know anything about, or
> have,
> >> to the best of my knowledge:
> >>
> >> Problem accessing /solr/. Reason:
> >>
> >>    Forbidden
> >>
> >> According to logs, this apparently meant that a MySQL query had failed
> due
> >> to a field name change. Since I would have to change my XML
> configuration
> >> files, I decided to use the opportunity to upgrade from Solr 5.1.4 to
> >> 6.2.0. It broke everything.
> >>
> >> First I was getting errors about "Unsupported major.minor version 52.0",
> >> so I needed to install the Linux x64 JRE 1.8.0, which I managed on
> CentOS 6
> >> with...
> >>
> >> yum install openjdk-1.8.0
> >>
> >> ...going to Oracle’s web site, downloading the latest JRE 1.8 build, and
> >> then running...
> >>
> >> yum localinstall jre-8u101-linux-x64.rpm
> >>
> >> So far so good. But I didn’t have JAVA_HOME set properly apparently, so
> I
> >> needed to do the not-exactly-intuitive…
> >>
> >> export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.
> >> el6_8.x86_64/jre/
> >>
> >> As usual, I manually moved over my mysql-connector-java-5.1.38-bin.jar
> >> file from the dist/ folder in the old version to the new one. Then after
> >> stopping the old process (with kill -9, since there seems to be no
> graceful
> >> way to shut down Solr—README.txt doesn’t mention bin/solr stop) I moved
> >> over my two core folders from the old server/solr/ folder. I tried to
> start
> >> it up with bin/solr start, and watched the errors roll in.
> >>
> >> There was some kind of problem with StopFilterFactory and the
> text_general
> >> field type. Thanks to Stack Overflow I was able to determine that the
> >> apparent problem was that there was a parameter, previously fine, which
> was
> >> no longer fine. So I removed all instances of enablePositionIncrements="
> true".
> >> That helped, but then I ran into a broader error: "Plugin Initializing
> >> failure for [schema.xml] fieldType". It didn’t say which field type.
> Buried
> >> in the logs I found a reference in the Java stack trace—which
> *disappears*
> >> (and distorts the viewing window horribly) after a few seconds when you
> try
> >> to view it in the web log UI—to the string "units="degrees"". Sure
> enough,
> >> this string appeared in my schema.xml for a class called "solr.
> >> SpatialRecursivePrefixTreeFieldType" that I’m pretty sure I never use.
> I
> >> removed that parameter, and moved on to the next set of errors.
> >>
> >> Apparently there is some aspect of the Thai text field type that Solr
> >> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.
> >>
> >> Now Solr was complaining about "Error loading class
> >> 'solr.admin.AdminHandlers'". So I found the reference to
> >> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and
> >> commented it out. Only then did Solr work again.
> >>
> >> This was not a smooth process. It took about two hours. The user
> interface
> >> is still as buggy as an early alpha of most products, the errors are
> >> difficult to understand when they don’t actually specify what’s wrong
> (and
> >> they almost never do), and there should have been an automatic process
> to
> >> highlight and fix problems in old (pre-6) configuration files. Never
> mind
> >> the fact that the XML-based configuration process is an antiquated
> >> nightmare when the rest of the world has long since moved onto
> databases.
> >>
> >> Maybe this will help someone else out there.
> >>
> >> Aaron
> >>
> >> PlainSite | http://www.plainsite.org
>
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Bram Van Dam
In reply to this post by Aaron Greenspan-2
I'm sorry you're having a "miserable" experience "again". That's
certainly not my experience with Solr. That being said:

> First I was getting errors about "Unsupported major.minor version 52.0", so I needed to install the Linux x64 JRE 1.8.0, which I managed on CentOS 6 with...
> yum install openjdk-1.8.0

This is not a Solr problem. Solr requires Java 8. Java 7 has been
officially end-of-lifed since april 2015. This means no more patches, no
more performance improvements and no more security updates (unless
you're paying Oracle). This is clearly stated in the (very decent) Solr
documentation. To use your own words: Java 7 is an antiquated nightmare
and the rest of the world has moved on to Java 8.

> So far so good. But I didn’t have JAVA_HOME set properly apparently, so I needed to do the not-exactly-intuitive…
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre/

You don't need to set JAVA_HOME to run Solr. But if you do have a
JAVA_HOME environment variable, and it points to a wrong Java version,
you're going to have a bad time.

> Then after stopping the old process (with kill -9, since there seems to be no graceful way to shut down Solr)

There's a stop command, which is documented. It's a non-surprising
location and has a non-surprising name. And even if there wasn't, "kill"
would have sufficed.

> There was some kind of problem with StopFilterFactory and the text_general field type. Thanks to Stack Overflow I was able to determine that the apparent problem was that there was a parameter, previously fine, which was no longer fine. So I removed all instances of enablePositionIncrements="true". That helped, but then I ran into a broader error: "Plugin Initializing failure for [schema.xml] fieldType". It didn’t say which field type. Buried in the logs I found a reference in the Java stack trace—which *disappears* (and distorts the viewing window horribly) after a few seconds when you try to view it in the web log UI—to the string "units="degrees"". Sure enough, this string appeared in my schema.xml for a class called "solr.SpatialRecursivePrefixTreeFieldType" that I’m pretty sure I never use. I removed that parameter, and moved on to the next set of errors.

Releases come with release notes and -- when required -- upgrade
instructions and admonitions. It's certainly possible that there's been
an oversight here or there and you're more than welcome to point those out.

> The user interface is still as buggy as an early alpha of most
products, the errors are difficult to understand when they don’t
actually specify what’s wrong (and they almost never do), and there
should have been an automatic process to highlight and fix problems in
old (pre-6) configuration files.

What user interface? Are you talking about the Admin UI? That's a
convenience feature which helps you manage Solr. It makes life a lot
easier, even if it's not perfect. The logs are generally quite good at
explaining what's wrong.

> Never mind the fact that the XML-based configuration process is an
antiquated nightmare when the rest of the world has long since moved
onto databases.

An antiquated nightmare? The rest of the world? How would this work?
What benefit would it possibly have?

You're more than welcome to report any bugs you find
(https://issues.apache.org/jira/browse/SOLR). But I feel like general
ranting on the mailing list isn't very productive. Well, I suppose
venting feels good, so there's that.

Things that would be more productive:

1. Reading the documentation.
2. Taking a basic system administration class or two.
3. Pointing out -- or contributing to -- parts of the documentation that
aren't up to par. Either on the mailing list, or on Jira. Preferably in
a constructive way instead of a "miserable experience"-way.

I feel like you're missing the part where most open source development,
documentation, release management etc is done by volunteers. Volunteers
who tend to scratch their own itch first, and are then kind enough to
donate the fruit of their labour to the rest of the world. You can
certainly make requests, and you can certainly hope for things to improve.

If you're having a "miserable" time "again", then you can always hire a
Solr consultant to do the work for you. You can't demand free stuff to
scratch your every itch. You can either invest your time and figure out
how to do things yourself, or your money and have things done for you.
But there's no such thing as a free lunch.

 - Bram

Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Alessandro Benedetti
In reply to this post by Aaron Greenspan-2
First of all I second Bram, I am sorry you had a bad experience with Solr,
but I think that:
-  without a minimum study and documentation
- without trying to follow the best practices
I think you are going to have a "miserable" experience with any software,
don't you ?

In addition to Bram :

On Mon, Sep 12, 2016 at 10:48 PM, Aaron Greenspan <
[hidden email]> wrote:
>
>  It didn’t say which field type. Buried in the logs I found a reference in
> the Java stack trace—which *disappears* (and distorts the viewing window
> horribly) after a few seconds when you try to view it in the web log UI—to
> the string "units="degrees"".
>

This si a bug, and it is really annoying, not sure anyone already raised
it, if not I suggest you to do that :)
But you can use the logs themselves without any problem.

>
> Apparently there is some aspect of the Thai text field type that Solr
> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.
>

If you were not using the Thai text, why had you the Thai Text field type
defined ?
Keep It Simple Stupid is the way :)
I find tons of Solr instances in production mith monster solrconfig.xml and
schema.xml. basically the old default ones, without any particular reason.
Don't do that !

>
> Now Solr was complaining about "Error loading class
> 'solr.admin.AdminHandlers'". So I found the reference to
> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and
> commented it out. Only then did Solr work again.
>

Seems to be you didn't take care of reading the update release notes, did
you ?


Cheers
--
--------------------------

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Yago Riveiro
I stuck in 5.3.1 because if upgrade to 5.5 or 6.x my cluster dies.  
 
Doing a rolling upgrade, when I upgrade the second node to 5.5 both die in the
per-sync phase, I don't know what changes in 5.5 but it's demanding a huge
quantity of memory to check if the replica it's in sync.  
 
This kind of stuff and the full re-index (12T)  between major releases are
indeed a pain.  
 
Cryptical errors and a deficient system to get metrics from what it's going on
inside the cluster is another issue, I'm unable to get the throughput in a
collection as a whole, the number of http connection in each node, the
utilization of the jetty thread pool and stuff like that.  
 
Solr is a great tool, but it's hard, too hard to get in.  
\--

 

/Yago Riveiro

![](https://link.nylas.com/open/m7fkqw0yim04itb62itnp7r9/local-
89046b47-a272?r=c29sci11c2VyQGx1Y2VuZS5hcGFjaGUub3Jn)

 
On Sep 13 2016, at 10:46 am, Alessandro Benedetti <[hidden email]>
wrote:  

> First of all I second Bram, I am sorry you had a bad experience with Solr,  
but I think that:  
\- without a minimum study and documentation  
\- without trying to follow the best practices  
I think you are going to have a "miserable" experience with any software,  
don't you ?

>

> In addition to Bram :

>

> On Mon, Sep 12, 2016 at 10:48 PM, Aaron Greenspan <  
[hidden email]> wrote:  
>  
> It didn’t say which field type. Buried in the logs I found a reference in  
> the Java stack trace—which *disappears* (and distorts the viewing window  
> horribly) after a few seconds when you try to view it in the web log UI—to  
> the string "units="degrees"".  
>

>

> This si a bug, and it is really annoying, not sure anyone already raised  
it, if not I suggest you to do that :)  
But you can use the logs themselves without any problem.

>

> >  
> Apparently there is some aspect of the Thai text field type that Solr  
> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text.  
>

>

> If you were not using the Thai text, why had you the Thai Text field type  
defined ?  
Keep It Simple Stupid is the way :)  
I find tons of Solr instances in production mith monster solrconfig.xml and  
schema.xml. basically the old default ones, without any particular reason.  
Don't do that !

>

> >  
> Now Solr was complaining about "Error loading class  
> 'solr.admin.AdminHandlers'". So I found the reference to  
> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and  
> commented it out. Only then did Solr work again.  
>

>

> Seems to be you didn't take care of reading the update release notes, did  
you ?

>

>  
Cheers  
\--  
\--------------------------

>

> Benedetti Alessandro  
Visiting card : [http://about.me/alessandro_benedetti](http://about.me/alessan
dro_benedetti&r=c29sci11c2VyQGx1Y2VuZS5hcGFjaGUub3Jn)

>

> "Tyger, tyger burning bright  
In the forests of the night,  
What immortal hand or eye  
Could frame thy fearful symmetry?"

>

> William Blake - Songs of Experience -1794 England

Best regards /Yago
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Alexandre Rafalovitch
In reply to this post by Alessandro Benedetti
On 13 September 2016 at 16:46, Alessandro Benedetti
<[hidden email]> wrote:
>>  It didn’t say which field type. Buried in the logs I found a reference in
>> the Java stack trace—which *disappears* (and distorts the viewing window
>> horribly) after a few seconds when you try to view it in the web log UI—to
>> the string "units="degrees"".
>>
>
> This si a bug, and it is really annoying, not sure anyone already raised
> it, if not I suggest you to do that :)

I don't remember seeing it opened. So this could be a great practice
for somebody who is good at finding bugs and issues. :-)

I love a good rant. I used to produce those myself. I love even more a
good rant that includes  specific granular improvements others can get
behind. The bug report as suggested above would be a great one example
of such a granular thing.

I would also point out that most of the contributors to Lucene/Solr
open source are able to contribute because somebody _pays_ them to
develop something on top of/with those projects and they hit
limitations they cannot solve in other easier ways. Those _usually_
are the cutting edge features such as CDCR, new performance
improvements, etc. We could always do with _more_ people who will
focus on the more user-oriented features or on making those new
cutting edge features more easily accessible.

Regards,
   Alex.
P.s. And if anybody wants to rant and will be at Lucene/Solr
revolution, I will be more than happy to sit and listen to you during
any of the food breaks. And I'll help figuring out what those granular
improvement suggestions could be. Feel free to reach out directly if
you want to have a rant scheduled too, instead of catching me
organically :-)

----
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Shawn Heisey-2
In reply to this post by Aaron Greenspan-2
On 9/12/2016 3:48 PM, Aaron Greenspan wrote:
> I have been on this list for some time because I know that any time I
> try to do anything related to Solr I’m going to have to spend hours on
> it, wondering why everything has to be so awful, and I just want
> somewhere to provide feedback with the dim hope that the product might
> improve one day. (So far, for my purposes, it hasn’t.) Sure enough, I
> still absolutely hate using Solr, and I have more feedback.

First, let me thank you for mentioning your experiences.  It's harsh
feedback, but I still welcome it.  I'm going to say some things that may
sound to you like excuses ... and you aren't really wrong to think that,
but we *do* take your comments seriously, and in many cases, we already
know that there are problems we need to solve.

As others have said, and as I'm sure you probably know, open source is
created by people trying to solve a problem for themselves, and is done
on  a volunteer basis.  If a project is lucky, it attracts interested
volunteers and truly magical things happen for the project users.  I
think Solr is a good project, with an awesome community.

Beginner documentation is one of the hardest things to write.  It's
difficult for people who live and breathe the software to view the
system from the perspective of someone who has never touched it at all,
and to write something that explains to that novice exactly how to make
it work.

> I started with a confusing error on the web console, which I still
> can’t figure out how to password protect without going through an
> insanely process involving "ZooKeeper," which I don’t know anything
> about, or have, to the best of my knowledge: Problem accessing /solr/.
> Reason: Forbidden

This particular part of your message involves an old fight in the Solr
project:  Security.  Those of us who have been in the industry forever
have learned that many of the security features that people expect for
Internet-facing services are not at all helpful for the security of
internal systems like Solr.  The best thing you can do to prevent
problems is to place Solr in a location where it cannot be reached by
people who cannot be trusted.  If you can trust those who have access,
then there's no need for intrinsic security features.

Any security that you layer on top of Solr (encryption, authentication,
etc) is useless if somebody compromises the system that talks to Solr,
which already has all the keys/passwords/etc required to get right in.

As evidenced by the fact that authentication has come to Solr, we *are*
listening to our users that demand security features.  The
authentication feature that you are trying to use, which involves basic
username/password authentication of the API calls that the admin UI
makes (*not* the admin UI itself), was originally developed for
SolrCloud -- which utilizes Zookeeper as a central configuration
database.  Work is underway right now to bring basic authentication to
standalone Solr, but it is not going to be available until at least 6.3,
and may take even longer to finish.  It will also require separate
configuration on every host, which is not required for SolrCloud.

> According to logs, this apparently meant that a MySQL query had failed
> due to a field name change. Since I would have to change my XML
> configuration files, I decided to use the opportunity to upgrade from
> Solr 5.1.4 to 6.2.0. It broke everything.

For almost ANY software, but especially for an open source package,
upgrading to a new major version is a good way to cause problems, not
usually a good way to solve them.  Solr does maintain compatibility with
configs that are completely current for the later versions in the
previous major release ... but there are a LOT of configs out in the
world (even in the latest versions of their software!) that were
originally designed for Solr 3.x, 4.x, or *early* 5.x.  Solr makes zero
guarantees about configs designed for software that old.

For an example of a similar situation in a different software package:
If you try to copy configs for the Apache webserver (httpd) from a 2.2
install to a 2.4 install, chances are excellent that you're going to
have to change those configs before Apache will even start, much less
operate as expected.  Upgrading Apache from 2.2 to 2.4 is technically a
"minor" version upgrade, but in terms of capability and configuration,
is similar to a major Solr version upgrade.

> First I was getting errors about "Unsupported major.minor version
> 52.0", so I needed to install the Linux x64 JRE 1.8.0, which I managed
> on CentOS 6 with... yum install openjdk-1.8.0 ...going to Oracle’s web
> site, downloading the latest JRE 1.8 build, and then running... yum
> localinstall jre-8u101-linux-x64.rpm So far so good. But I didn’t have
> JAVA_HOME set properly apparently, so I needed to do the
> not-exactly-intuitive… export
> JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre/

Others have already covered this topic.  Managing Java installations is
required before Solr will work, and there are so many ways to handle it
that Solr cannot cover the topic in its documentation.

> As usual, I manually moved over my mysql-connector-java-5.1.38-bin.jar
> file from the dist/ folder in the old version to the new one. Then
> after stopping the old process (with kill -9, since there seems to be
> no graceful way to shut down Solr—README.txt doesn’t mention bin/solr
> stop) I moved over my two core folders from the old server/solr/
> folder. I tried to start it up with bin/solr start, and watched the
> errors roll in.

A standard "kill" would have stopped Solr gracefully, with no need for
the -9 signal.  It can take a while to happen, though.  Using "kill -9"
can cause large problems on restart that are difficult to track down.

If you change the the extracted download or the installed program root
(normally /opt/solr on Linux), and type "bin/solr" without options, it
gives you enough information to track down the stop command.  If you do
use the installer script for Linux with default options, you can
typically type "service solr stop" because the installer handles an init
script.

> There was some kind of problem with StopFilterFactory and the
> text_general field type. Thanks to Stack Overflow I was able to
> determine that the apparent problem was that there was a parameter,
> previously fine, which was no longer fine. So I removed all instances
> of enablePositionIncrements="true". That helped, but then I ran into a
> broader error: "Plugin Initializing failure for [schema.xml]
> fieldType". It didn’t say which field type. Buried in the logs I found
> a reference in the Java stack trace—which *disappears* (and distorts
> the viewing window horribly) after a few seconds when you try to view
> it in the web log UI—to the string "units="degrees"". Sure enough,
> this string appeared in my schema.xml for a class called
> "solr.SpatialRecursivePrefixTreeFieldType" that I’m pretty sure I
> never use. I removed that parameter, and moved on to the next set of
> errors.

The actual logfile (solr.log) is generally MUCH more useful for tracking
down problems than the Logging tab in the admin UI.  This is another
example of something that is not immediately obvious to a novice.

> Apparently there is some aspect of the Thai text field type that Solr
> 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text. Now Solr
> was complaining about "Error loading class
> 'solr.admin.AdminHandlers'". So I found the reference to
> solr.admin.AdminHandlers in solrconfig.xml for each of my cores and
> commented it out. Only then did Solr work again. This was not a smooth
> process. It took about two hours. The user interface is still as buggy
> as an early alpha of most products, the errors are difficult to
> understand when they don’t actually specify what’s wrong (and they
> almost never do), and there should have been an automatic process to
> highlight and fix problems in old (pre-6) configuration files. Never
> mind the fact that the XML-based configuration process is an
> antiquated nightmare when the rest of the world has long since moved
> onto databases.

I've already mentioned how configs designed for very old versions won't
work.  It sounds like your configs were designed for 4.x, which means
they will usually work in 5.x, but won't work if you go higher.

Zookeeper is a configuration database.  If it is properly set up,
SolrCloud uses zookeeper to build a true cluster of nodes with no single
point of failure and very little config stored on individual servers.

It's true that the config files that you add to the zookeeper database
are still in XML format.  XML is a well-defined standard that is
particularly good at holding structured configuration information.
Switching to another format like JSON is possible, but that has a strong
potential to cause *additional* upgrade headaches, and I think you've
already encountered enough of those!

Solr's documentation is not where we want it to be.  Detailed
step-by-step instructions for novices (and tools to match) is the
biggest omission.  It would also be useful if we had more automated
tools for upgrading.  Error messages are another area for improvement.

Writing all these things is not easy.  Good documentation is HARD.
Writing tools that work right takes time.  It's a struggle to provide
error messages that are helpful for *developers* ... error messages that
are helpful for *users* are even harder.

The change in Solr 5.0 where we took control of the servlet container
away from the user was one of the first steps in making the end-user
experience more consistent and easier to document.  It was only an
initial step, though.  There is lots of work to do.  If you're able to
help, we would appreciate it.

https://wiki.apache.org/solr/WhyNoWar

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Aaron Greenspan-2
In reply to this post by Aaron Greenspan-2
Hello again…

I get this on digest mode (and wasn’t even sure my initial message went through to the list), so please forgive the delay in responding.

I think the various reactions to my post suggest that a sizable number of users (and by "users" I mean those who are not affiliated with Apache and who are not core contributors) find Solr difficult to use. For me, this was confirmed many months ago when a family friend—a non-technical CEO twice my age of a company recently acquired for a very sizable sum—came over for dinner and without any prompting from anyone began complaining about this impossible program at work called Solr that none of his engineers could get to work. By his telling, he had several experienced engineers working on it.

I’m aware that issues with Java are not Solr’s fault. But most programs still manage to gracefully fail when they are missing a dependency, and then clearly report what’s missing. If you’re not actually a Java programmer, which I am not, "major.minor 52.0" (for example) is meaningless gibberish. "Please download and install JRE 1.8 to run this software" would be considerably clearer. How is it that Solr can search through millions of files, but it can’t do that?

As for the suggestions that I should (1) read the documentation; (2) file reports on JIRA; and (3) hire a consultant if my own skills aren’t up to snuff:

1. I did. The documentation is severely lacking, apparently having been written by project contributors who have vastly different goals than their users. Example #1: the security issue (I still can’t figure out how to password-protect the Solr web UI, a convenience perhaps, but a convenience I depend upon because I cannot spend all day handling the combination of the command line and Java, neither can most people, and I still can’t figure out how to install or use "Zookeeper") is documented here:

https://cwiki.apache.org/confluence/display/solr/Securing+Solr

Note the red section at the bottom (which originally wasn’t even there): "No Solr API, including the Admin UI, is designed to be exposed to non-trusted parties. Tune your firewall so that only trusted computers and people are allowed access." If one of my employees tried to pull this I would fire them. Admin UIs in every other product I’ve ever seen are password-protected. Always. Netscape Enterprise Server in 1996 had a password for its admin UI. (See https://docs.oracle.com/cd/E19957-01/816-5654-10/816-5654-10.pdf, page 62.) Cobalt RaQs in 1999 had passwords on their Admin UIs. Home routers have had passwords since time immemorial. It’s 2016. Solr is on major release version 6. Don’t tell me how to configure my firewall, and certainly don’t tell me that firewalls can programmatically block access to "people". (My understanding of firewalls is that they block access to IP addresses and/or ports—if there was a product that could magically always block certain people, who would bother with firewalls?) Even if I configure my firewall to restrict [IP]:8983 to one IP, many people may use that IP, especially at a large organization with enough data to merit a product like Solr! Fix your dangerously insecure product, give it an install script that handles the SSL cert generation, and if I for some reason need to do something to turn that fix on, please tell me how.

Note also, going back to Zookeeper, that on https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin, the documentation states, "Run the following command to upload it to Zookeeper. (ensure that the Zookeeper port is correct)". First, what is it talking about? I’ve never heard of Zookeeper outside of an actual zoo. Second, it runs on a port? Third, which port? Is it 9983, as is only cryptically alluded to below? What if it’s not running yet? How do I start it? Is it secure? Is it part of Java? Is it part of Solr? Is it even installed? Why is this my problem? Can you imagine if any other piece of software involving a password worked this way? (Yes, I have read the Apache Zookeeper Wikipedia article. My point about flaws in the documentation stands. It is confusing to new users—those who need it most.)

Example #2: the potential bug involving the fieldType error message. I searched for documentation on the fieldType. Something about the Solr API (https://lucene.apache.org/solr/6_2_0/solr-core/org/apache/solr/schema/FieldType.html) came up which was not relevant or helpful. It’s easy to say RTFM, but what if the product is full of bugs? Those tend not to be in manuals.

After doing this dance enough times I’ve learned that the Solr documentation is most often out-of-date or unhelpful. So here I am.

2. I have filed several reports on JIRA. Here’s the kind of response I have received in the past:

https://issues.apache.org/jira/browse/SOLR-7896?focusedCommentId=14661324&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14661324

"Please bring this kind of thing up on the user's list rather than raise JIRAs to be sure you're not simply misunderstanding things. If it's a real problem in Solr, then raise a JIRA."

So here I am! I’m on the user’s list! And I’m being given the runaround. The developers often don’t think user feedback about critical features amounts to a "real problem", but the mailing list is the wrong place to complain because it’s not structured enough. This is potentially why (in my view) Solr never really improves.

3. This is a potential solution, but not one I choose to pursue. For one thing, I am not an idiot. I’ve managed Linux systems for about 18 years now and I’ve been programming for 20. I have learned that I am rarely the best at anything, so sure, I fully admit that there will always be others with much better skills than my own. But I’m an intelligent person with experience in software trying to leave constructive feedback, and being told that my feedback essentially reflects on my own stupidity kind of misses the point. I’m providing feedback because things need fixing.

As for Bram Van Dam’s question about how a settings database would work, I don’t think it’s worth getting too specific here, but my general response would be, if you need a good model for how to widely deploy software—not a perfect model, but a good one—look at WordPress. A lot of people use WordPress. Like any software, it has its flaws. But average people are able to sign in, with a password (!), change their admin settings, and save those settings I’m pretty certain to a MySQL schema. I’d love to be able to do that with Solr.

Lastly, I run a non-profit foundation devoted to transparency, and I think Solr could do a lot to help further my foundation's goals. That’s why I’m using it at all. It’s the kind of project I’d be willing to fund (since I don’t think I can write the code myself in this instance)—except that very few people working on Solr ever take my concerns seriously (even though users seem to). If funding is the reason (or even a reason) that Solr isn’t in a better state, the way users are treated might be part of the problem.

Aaron

PlainSite | http://www.plainsite.org
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Alexandre Rafalovitch
On 14 September 2016 at 06:42, Aaron Greenspan
<[hidden email]> wrote:
> This is a potential solution, but not one I choose to pursue. For one thing, I am not an idiot. I’ve managed Linux systems for about 18 years now and I’ve been programming for 20. I have learned that I am rarely the best at anything, so sure, I fully admit that there will always be others with much better skills than my own. But I’m an intelligent person with experience in software trying to leave constructive feedback, and being told that my feedback essentially reflects on my own stupidity kind of misses the point. I’m providing feedback because things need fixing.

<counter-rant>
And that's the part I am confused about. Managing LInux systems is a
real pain in the ... Config files locations differ between
distributions. Upgrades are confusing, error messages are truly
critic. Debugging by dmesg and truss/strace is dark arts. Reading the
logs is nearly an art.

But with that experience, Zookeeper port is an lsof away. Or a ps away
if you want to read it from the parameters. Or a netstat away. Binding
anything to a local subnet is something of a standard firewall
operation. Couple of other things are output as part of "bin/solr
start --help" as well as part of log messages of running examples. I
understand the frustration. I truly do. And my own - committer - focus
is on improving beginner's experience (no, not looking for funding).
But the problems you list are definitely should be minor, not major
pain points to a Linux system administrator.

Solr is not like a WordPress. WordPress is designed for external users
and so is optimized for ease of used at expense of everything else.
Not correctness, not internal security (passwords are plain text,
plugins have access to everything), not ease of "good" development.
And WordPress is a _small_ product. It is a wrong comparison to the
point that apples and oranges are in the same category of fruit.

Solr is like a MySQL at least. And, frankly, changing a root password
in MySQL is also quite a pain. Or BEA weblogic (which I could...)

As to JIRA, the question was of a very high granularity on a use case
that is complicated and is not a bug/feature distinction. It also has
been discussed multiple times on the User list.

</counter-rant>

Anyway, I see one JIRA in this so far (Admin UI reclosing the log
message). If nobody else opens it, I will and have a go at it in the
next couple of days.

Regards,
   Alex
P.s. Aaron, perhaps you missed it with the digest mode, but I JUST
asked for feedback on an example reading group idea. I would have
expected you to jump on an opportunity as that would mean a direct
access to somebody contributing their contributor (mine!) time to
improve your understanding of Solr at whatever level of knowledge you
currently have. And - if that's not obvious - to see what kind of
things people find difficult to feed it into the next version of Solr.
Several people already showed interest, but you are not among them.
Hopefully, you will see that email and join us on your next read
through the digest. For easy reference, the survey link again is:
https://www.surveymonkey.com/r/JH8S666

----
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Jan Høydahl / Cominvent
In reply to this post by Aaron Greenspan-2
> 14. sep. 2016 kl. 01.42 skrev Aaron Greenspan <[hidden email]>:

First of all, thanks for spending some time to give feedback and opening JIRAs (even if some get closed because it is a question, not a bug report).
This list is exactly the right forum to bring up frustrations newbie users might have with Solr, and I think we should LISTEN carefully and identify low hanging fruits, as Alexandre is also focusing on!

> I’m aware that issues with Java are not Solr’s fault. But most programs still manage to gracefully fail when they are missing a dependency, and then clearly report what’s missing. If you’re not actually a Java programmer, which I am not, "major.minor 52.0" (for example) is meaningless gibberish. "Please download and install JRE 1.8 to run this software" would be considerably clearer. How is it that Solr can search through millions of files, but it can’t do that?

I agree that it would help big time if bin/solr would validate correct Java version. Found https://issues.apache.org/jira/browse/SOLR-8080 <https://issues.apache.org/jira/browse/SOLR-8080> for this, will try to cook up something :)
Also, over in https://issues.apache.org/jira/browse/SOLR-9508 <https://issues.apache.org/jira/browse/SOLR-9508> I added a check for Java, so it will prompt you to install Java before you can install Solr. Should perhaps check for min-version here as well.

> 1. I did. The documentation is severely lacking, apparently having been written by project contributors who have vastly different goals than their users.

Yea, the ref-guide is a huge beast and aims to list every single setting.
Then we have the tutorials that aim to walk new users through installing, indexing and searching. But they don’t cover upgrading etc of course.
Then of course you have all the books - which is perhaps the best option right now to get quickly up to speed..

If you could decide, what kind of documentation would you want from the project? A very short “Solr Quick start guide”? with step-by-step instructions for the most common tasks from a User perspective?

> Note the red section at the bottom (which originally wasn’t even there): "No Solr API, including the Admin UI, is designed to be exposed to non-trusted parties. Tune your firewall so that only trusted computers and people are allowed access." If one of my employees tried to pull this I would fire them. Admin UIs in every other product I’ve ever seen are password-protected. Always. Netscape Enterprise Server in 1996 had a password for its admin UI.

The warning is an honest way to tell admins that Solr is not designed to be an internet-facing program, like httpd or nginx. That is not to say that you cannot secure Solr pretty well with what we already got, but there will probably be a bunch of security holes since an internet-facing service is not the goal of Solr. It is not either an excuse for not having a password protection that is easier to understand.

Still, the truth is that you CAN add authentication to all of Solr, including the UI. What is confusing though, is that the static (non-sensitive) parts of the UI will load and display, but as soon as the UI attempts to request any kind of information from the Solr APIs, it will fail.

In my opinion this is perceived by our users as the Admin UI being insecure, and even if technically not true, we should continue work on https://issues.apache.org/jira/browse/SOLR-7896 <https://issues.apache.org/jira/browse/SOLR-7896> (Add a login page for Solr Administrative Interface).

> 2. I have filed several reports on JIRA. Here’s the kind of response I have received in the past:

Again, thanks for contributing all of this, without users who care and suggest stuff there would be no progress…
And I apologise if we as a community have not been understanding or had a welcoming attitude to the suggestions.

> https://issues.apache.org/jira/browse/SOLR-7896?focusedCommentId=14661324&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14661324

Security is a special case I would say, since the project had an “official” attitude that we’d not add any kind of security whatsoever, to not mislead people into exposing Solr publicly. But then things changed in 5.2 and all the new security stuff got no vetoes anymore, and we’re now in a pretty good position on the API level of security. Wrt SOLR-7896, I re-opened that one after it got prematurely closed. But apparently not enough users have felt the need for it for someone to invest the time or money it takes to implement it. And that’s how open source works, as I’m sure you know.

> Lastly, I run a non-profit foundation devoted to transparency, and I think Solr could do a lot to help further my foundation's goals. That’s why I’m using it at all. It’s the kind of project I’d be willing to fund (since I don’t think I can write the code myself in this instance)—except that very few people working on Solr ever take my concerns seriously (even though users seem to). If funding is the reason (or even a reason) that Solr isn’t in a better state, the way users are treated might be part of the problem.

In Apache, you cannot really pay (or bribe) for getting a feature committed per se. But you are free to contact myself or other another committer that also work as a consultant, with a request to prioritise a certain feature. Note that there would be no guarantee that the hours you pay for would ever get committed; All development in Solr happens in the open, in a consensus-driven fashion, see http://theapacheway.com/community <http://theapacheway.com/community>

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Shawn Heisey-2
In reply to this post by Aaron Greenspan-2
On 9/13/2016 5:42 PM, Aaron Greenspan wrote:
> I get this on digest mode (and wasn’t even sure my initial message
> went through to the list), so please forgive the delay in responding.

I've added you as BCC so you'll get this as soon as I send it.  I wrote
most of it last night, and left it to complete in the morning -- and now
I see that Jan has replied with similar information.

> I think the various reactions to my post suggest that a sizable number
> of users (and by "users" I mean those who are not affiliated with
> Apache and who are not core contributors) find Solr difficult to use.
> For me, this was confirmed many months ago when a family friend—a
> non-technical CEO twice my age of a company recently acquired for a
> very sizable sum—came over for dinner and without any prompting from
> anyone began complaining about this impossible program at work called
> Solr that none of his engineers could get to work. By his telling, he
> had several experienced engineers working on it.

I've been using Solr for about six years now.  When I first got started,
I spent a HUGE amount of time figuring out the most basic things, and I
asked plenty of dumb questions right here on this list.  I think it took
me about three days to get from that initial download of the 1.4.0
archive to a working server that had something besides "collection1" on
it.  It took another month or so beyond that before I could demonstrate
anything usable to my team, and after that had to start writing tools
that would actually create the index without manual intervention.  One
of those tools was an init script.  Now Solr will install an init script
on Unix-like operating systems.

My active production indexes are running on a couple of different 4.x
versions.  I have production 5.x indexes on servers serving a hot
standby role, but they have not been fully vetted, so the primaries
remain on older versions.  It'll be a while before I get around to 6.x.

> I’m aware that issues with Java are not Solr’s fault. But most
> programs still manage to gracefully fail when they are missing a
> dependency, and then clearly report what’s missing. If you’re not
> actually a Java programmer, which I am not, "major.minor 52.0" (for
> example) is meaningless gibberish. "Please download and install JRE
> 1.8 to run this software" would be considerably clearer. How is it
> that Solr can search through millions of files, but it can’t do that?

I know that in the 5.x days, we had Java version detection in the start
script, so that the start would complain if certain buggy versions of
Java 7 were detected.  I think it would even refuse to start if the
version wasn't new enough.  If we have lost that with 6.x, that needs to
go back in, and we will look at that problem immediately.

On password security:  I hear you.  Part of the issue is that Solr can't
*directly* do security.  It's sitting behind another piece of software
that handles the network and HTTP -- Jetty.  Until recently, Solr really
didn't touch the servlet container, allowing it to do its thing
according to its config files.  Part of this was due to the fact that
before 5.0, we did not know what container was being used -- the user
had the option of deploying in several different containers, and none of
them handled security in quite the same way.  Since 5.0, the only
officially supported container is the Jetty that Solr includes, so we
CAN put container-specific code into Solr.  This is why 5.3 and later
have good support for authentication.

TL;DR info:  When you password protect Solr, the admin UI actually
doesn't get protected.  It is nothing more than static HTML, CSS,
Javascript, and images.  The admin UI actually runs in your browser, not
on the server.  What gets password protection is the HTTP API used for
information, queries, and updates.

You're absolutely right that our documentation and error messages are
completely inadequate for a novice user.  The error messages sometimes
aren't even adequate for an experienced Java developer to know what went
wrong, at least not without examining the source code.

> As for Bram Van Dam’s question about how a settings database would
> work, I don’t think it’s worth getting too specific here, but my
> general response would be, if you need a good model for how to widely
> deploy software—not a perfect model, but a good one—look at WordPress.
> A lot of people use WordPress. Like any software, it has its flaws.
> But average people are able to sign in, with a password (!), change
> their admin settings, and save those settings I’m pretty certain to a
> MySQL schema. I’d love to be able to do that with Solr.

I concur with what Alexandre said about Wordpress compared to Solr.  The
target audience and deployment method are quite different ... but I take
your point too -- we can learn a lot from projects like WordPress, which
has had to address "first contact" issues in their documentation.

The addition of Zookeeper capability to Solr in version 4.0 created
SolrCloud, which automates the job of using multiple Solr machines as a
scalable and redundant cluster.  Unfortunately, Zookeeper is not trivial
to set up, so we traded a super-hard problem for a different and
slightly less challenging problem.  Managing zookeeper is easier than
setting up all of the infrastructure required for a cluster when
SolrCloud is NOT used.

Solr has a running mode where it embeds a Zookeeper server inside of
Solr itself.  The port number is usually 9983.  The startup script will
set that port number to the Solr port plus 1000, unless a specific port
number is configured.

The cloud example (bin/solr start -e cloud) starts an embedded zookeeper
on the first Solr instance.  This arrangement is suitable for a demo or
proof of concept, but a different setup is highly recommended for
production: an external ensemble of three or more zookeeper instances
running on separate physical boxes.  Three servers are required for
zookeeper redundancy, and running them outside Solr is recommended so
that the entire ZK ensemble stays up even when a Solr process is
stopped.  Zookeeper does NOT need to run on separate hardware from Solr,
though a large and busy cluster will perform better if zookeeper uses a
different storage volume than Solr does.

I hate that you've had a bad experience with Solr.  Your feedback has
given us some pointers about specific things we can improve.  I hope
you'll be willing to continue providing guidance.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Jan Høydahl / Cominvent
In reply to this post by Jan Høydahl / Cominvent
> If you could decide, what kind of documentation would you want from the project? A very short “Solr Quick start guide”? with step-by-step instructions for the most common tasks from a User perspective?

I just became aware of StackOverflow’s Documentation project, which also has a solr topic:
http://stackoverflow.com/documentation/solr <http://stackoverflow.com/documentation/solr>
Perhaps that could also be a good place to contribute HOWTOs and more end-user focused docs?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Gus Heck
While stack overflow is a great place, and the more good info that exists
there, the merrier, I think Solr should have it's own complete docs, in
addition to anything found on 3rd party sites. Each hop to a new location
is a chance for the user to get lost, and the content on 3rd party sites
could be wrong, out of date without folks here being aware of it as
quickly.

Also, I just noticed that there seem to be some links at the bottom of the
admin UI, but they often run off the bottom and can be easily missed. The
"documentation" link doesn't actually lead to the documentation... Maybe
those should be along the top where they would be easily seen and the
link(s?) fixed up?

-Gus

On Wed, Sep 14, 2016 at 3:27 PM, Jan Høydahl <[hidden email]> wrote:

> > If you could decide, what kind of documentation would you want from the
> project? A very short “Solr Quick start guide”? with step-by-step
> instructions for the most common tasks from a User perspective?
>
> I just became aware of StackOverflow’s Documentation project, which also
> has a solr topic:
> http://stackoverflow.com/documentation/solr <http://stackoverflow.com/
> documentation/solr>
> Perhaps that could also be a good place to contribute HOWTOs and more
> end-user focused docs?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>


--
http://www.the111shift.com
Reply | Threaded
Open this post in threaded view
|

Re: Miserable Experience Using Solr. Again.

Jan Høydahl / Cominvent
I was not proposing StackOverflow as some official docs.
Solr’s only official doc is the RefGuide, and the official user-contributed
docs is at http://wiki.apache.org/solr/ <http://wiki.apache.org/solr/>

But it may be that some of you Solr users want to contribute to StackOverflow’s
Solr topic, both in requesting topics, writing short snippets and fixing what is
wrong.

But I wonder if we should consider creating an official, slim Solr User Guide
as well, for end users, structured as a getting-started guide and with focus
on how you achieve a task, not documenting all 99 parameters a plugin can
take.

Regarding the links in the admin, the “Documentation” link should perhaps
link to http://lucene.apache.org/solr/resources.html#documentation <http://lucene.apache.org/solr/resources.html#documentation>
The other links seem ok though.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 14. sep. 2016 kl. 23.11 skrev Gus Heck <[hidden email]>:
>
> While stack overflow is a great place, and the more good info that exists
> there, the merrier, I think Solr should have it's own complete docs, in
> addition to anything found on 3rd party sites. Each hop to a new location
> is a chance for the user to get lost, and the content on 3rd party sites
> could be wrong, out of date without folks here being aware of it as
> quickly.
>
> Also, I just noticed that there seem to be some links at the bottom of the
> admin UI, but they often run off the bottom and can be easily missed. The
> "documentation" link doesn't actually lead to the documentation... Maybe
> those should be along the top where they would be easily seen and the
> link(s?) fixed up?
>
> -Gus
>
> On Wed, Sep 14, 2016 at 3:27 PM, Jan Høydahl <[hidden email]> wrote:
>
>>> If you could decide, what kind of documentation would you want from the
>> project? A very short “Solr Quick start guide”? with step-by-step
>> instructions for the most common tasks from a User perspective?
>>
>> I just became aware of StackOverflow’s Documentation project, which also
>> has a solr topic:
>> http://stackoverflow.com/documentation/solr <http://stackoverflow.com/
>> documentation/solr>
>> Perhaps that could also be a good place to contribute HOWTOs and more
>> end-user focused docs?
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>>
>
>
> --
> http://www.the111shift.com

12