Solr7.3.1 Installation

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr7.3.1 Installation

tapan1707
We are trying to install solr-7.3.1 into our existing system (We have also
made some changes by adding one custom query parser).

I am having some build issues and it would be really helpful if someone can
help.

While running ant test(in the process of building the solr package), it
terminates because of failed tests.
At first time (build with ant-1.9)
Tests with failures [seed: C2C0D761AEAAE8A4] (first 10 out of 23):
21:25:20    [junit4]   -
org.apache.solr.client.solrj.response.TestSuggesterResponse (suite)
21:25:20    [junit4]   -
org.apache.solr.client.solrj.response.TermsResponseTest (suite)
21:25:20    [junit4]   - org.apache.solr.client.solrj.TestSolrJErrorHandling
(suite)
21:25:20    [junit4]   - org.apache.solr.client.solrj.GetByIdTest (suite)
21:25:20    [junit4]   -
org.apache.solr.client.solrj.response.TestSpellCheckResponse (suite)
21:25:20    [junit4]   -
org.apache.solr.client.solrj.embedded.LargeVolumeEmbeddedTest (suite)
21:25:20    [junit4]   -
org.apache.solr.client.solrj.embedded.JettyWebappTest.testAdminUI
21:25:20    [junit4]   -
org.apache.solr.client.solrj.embedded.SolrExampleStreamingBinaryTest (suite)
21:25:20    [junit4]   - org.apache.solr.client.solrj.SolrExampleBinaryTest
(suite)
21:25:20    [junit4]   -
org.apache.solr.client.solrj.embedded.LargeVolumeBinaryJettyTest (suite)

Running the same ant test command without doing any changes (build with
ant-1.10)
Tests with failures [seed: 7E004642A6008D89]:
11:30:57    [junit4]   -
org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove  

Thirds time (build with ant 1.10)
[junit4] Tests with failures [seed: EFD939D82A6EC707]:
[junit4]   - org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test

Even though I'm not making any changes, build is failing with different
failed tests. Can anyone help me with this, I mean if there is any problem
with the code then shouldn't it fail with same test cases?
Also, all above-mentioned test cases work fine if I check them individually.
(using ant test -Dtests.class=<test class>)

Also, does ant version has any effects in build??

At last, at present, we are using solr-6.4.2 which has zookeeper-3.4.6
dependency but for solr-7, the zookeeper dependency has been upgraded to
3.4.10, so my question is, At what extent does this might affect our system
performance? Can we use zookeeper-3.4.6 with solr-7?
(same with the jetty version)

Thanks in advance



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr7.3.1 Installation

Emir Arnautović
Hi,
Why are you building Solr? Because you added your custom query parser? If that’s the case, then it is not the way to do it. You should set up separate project for your query parser, build it and include jar in your Solr setup.
It is not query parser, but here is blog/code for simple update processor: https://www.od-bits.com/2018/02/solr-docvalues-on-analysed-field.html <https://www.od-bits.com/2018/02/solr-docvalues-on-analysed-field.html>

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 11 Jul 2018, at 07:20, tapan1707 <[hidden email]> wrote:
>
> We are trying to install solr-7.3.1 into our existing system (We have also
> made some changes by adding one custom query parser).
>
> I am having some build issues and it would be really helpful if someone can
> help.
>
> While running ant test(in the process of building the solr package), it
> terminates because of failed tests.
> At first time (build with ant-1.9)
> Tests with failures [seed: C2C0D761AEAAE8A4] (first 10 out of 23):
> 21:25:20    [junit4]   -
> org.apache.solr.client.solrj.response.TestSuggesterResponse (suite)
> 21:25:20    [junit4]   -
> org.apache.solr.client.solrj.response.TermsResponseTest (suite)
> 21:25:20    [junit4]   - org.apache.solr.client.solrj.TestSolrJErrorHandling
> (suite)
> 21:25:20    [junit4]   - org.apache.solr.client.solrj.GetByIdTest (suite)
> 21:25:20    [junit4]   -
> org.apache.solr.client.solrj.response.TestSpellCheckResponse (suite)
> 21:25:20    [junit4]   -
> org.apache.solr.client.solrj.embedded.LargeVolumeEmbeddedTest (suite)
> 21:25:20    [junit4]   -
> org.apache.solr.client.solrj.embedded.JettyWebappTest.testAdminUI
> 21:25:20    [junit4]   -
> org.apache.solr.client.solrj.embedded.SolrExampleStreamingBinaryTest (suite)
> 21:25:20    [junit4]   - org.apache.solr.client.solrj.SolrExampleBinaryTest
> (suite)
> 21:25:20    [junit4]   -
> org.apache.solr.client.solrj.embedded.LargeVolumeBinaryJettyTest (suite)
>
> Running the same ant test command without doing any changes (build with
> ant-1.10)
> Tests with failures [seed: 7E004642A6008D89]:
> 11:30:57    [junit4]   -
> org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove  
>
> Thirds time (build with ant 1.10)
> [junit4] Tests with failures [seed: EFD939D82A6EC707]:
> [junit4]   - org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test
>
> Even though I'm not making any changes, build is failing with different
> failed tests. Can anyone help me with this, I mean if there is any problem
> with the code then shouldn't it fail with same test cases?
> Also, all above-mentioned test cases work fine if I check them individually.
> (using ant test -Dtests.class=<test class>)
>
> Also, does ant version has any effects in build??
>
> At last, at present, we are using solr-6.4.2 which has zookeeper-3.4.6
> dependency but for solr-7, the zookeeper dependency has been upgraded to
> 3.4.10, so my question is, At what extent does this might affect our system
> performance? Can we use zookeeper-3.4.6 with solr-7?
> (same with the jetty version)
>
> Thanks in advance
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Reply | Threaded
Open this post in threaded view
|

Re: Solr7.3.1 Installation

tapan1707
Hi Emir,

Thanks for your reply.

Here building Solr has nothing to do with custom query parser.
Our system has been designed in such a way that package is created by
running following commands (there are other commands too but not related to
solr, hence omitting those)

${ANT} -buildfile ${SOLR_DIR}/build.xml ivy-bootstrap
${ANT} -buildfile ${SOLR_DIR}/build.xml compile
${ANT} -buildfile ${SOLR_DIR}/build.xml test
${ANT} -buildfile ${SOLR_DIR}/solr/build.xml dist
${ANT} -buildfile ${SOLR_DIR}/solr/build.xml server
${ANT} -buildfile ${SOLR_DIR}/lucene/build.xml package

And while building solr package, all are failing at *ant test* command.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr7.3.1 Installation

Shawn Heisey-2
In reply to this post by tapan1707
On 7/10/2018 11:20 PM, tapan1707 wrote:
> We are trying to install solr-7.3.1 into our existing system (We have also
> made some changes by adding one custom query parser).
>
> I am having some build issues and it would be really helpful if someone can
> help.
>
> While running ant test(in the process of building the solr package), it
> terminates because of failed tests.

This is a known problem.  Solr's tests are in not in a good state. 
Sometimes they pass, sometimes they fail.  Since there are so many tests
and a fair number of them do fail intermittently, this creates a
situation where on most test runs, there is at least one test failure. 
Run the tests enough times, and eventually they will all pass ... but
this usually takes many runs.

Looking at the commands you're using in your script:  After a user has
run the "ant ivy-bootstrap" command once, ivy is downloaded into the
user's home directory and does not need to be downloaded again.  Only
the "ant package" command (run in the "solr" subdirectory) is actually
needed to build Solr.  The rest of the commands are not needed.

As Emir said, you don't need to build Solr at all, even when using
custom plugins.  You can download and use the binary package.

There is effort underway to solve the problem with Solr tests. The
initial phase of that effort is to disable the tests that fail most
frequently.  The second overlapping phase of the effort is to actually
fix those tests so that they don't fail - either by fixing bugs in the
tests themselves, or by fixing real bugs in Solr.

> Also, does ant version has any effects in build??

Ant 1.8 and 1.9 should work.  Versions 1.10.0, 1.10.1, as well as 1.10.3
and later should be fine, but 1.10.2 has a bug that results in the
lucene-solr build failing:

https://issues.apache.org/jira/browse/LUCENE-8189

> At last, at present, we are using solr-6.4.2 which has zookeeper-3.4.6
> dependency but for solr-7, the zookeeper dependency has been upgraded to
> 3.4.10, so my question is, At what extent does this might affect our system
> performance? Can we use zookeeper-3.4.6 with solr-7?
> (same with the jetty version)

You should be able to use any ZK 3.4.x server version with any version
of Solr.  Most versions of Solr should also work with 3.5.x (still in
beta) servers.  Early 4.x version s shipped with ZK 3.3.x, and the ZK
project does not guarantee compatibility between 3.3.x and 3.5.x.

I can't guarantee that you won't run into bugs, but ZK is generally a
very stable piece of software.  Each new release of ZK includes a very
large list of bugfixes.  I have no idea what implications there are for
performance.  You would need to ask a ZK support resource that
question.  The latest stable release that is compatible with your
software is the recommended version.  Currently that is 3.4.12.  The
3.5.x releases are in beta.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Solr7.3.1 Installation

Erick Erickson
Note that the native test runs have the know-flaky tests _enabled_ by
default, run tests with

-Dtests.badapples=true

to disable them.

Second possibility is to look at the tests that failed and if there is
an annotation
@BadApple
or
@AwaitsFix
ignore the failure if you can get the tests to pass when running individually.

As Shawn says, this is a known issue that we're working on, but the
technical debt is such that it'll
be a long-term issue to fix.

Best,
Erick



On Wed, Jul 11, 2018 at 7:13 AM, Shawn Heisey <[hidden email]> wrote:

> On 7/10/2018 11:20 PM, tapan1707 wrote:
>>
>> We are trying to install solr-7.3.1 into our existing system (We have also
>> made some changes by adding one custom query parser).
>>
>> I am having some build issues and it would be really helpful if someone
>> can
>> help.
>>
>> While running ant test(in the process of building the solr package), it
>> terminates because of failed tests.
>
>
> This is a known problem.  Solr's tests are in not in a good state.
> Sometimes they pass, sometimes they fail.  Since there are so many tests and
> a fair number of them do fail intermittently, this creates a situation where
> on most test runs, there is at least one test failure.  Run the tests enough
> times, and eventually they will all pass ... but this usually takes many
> runs.
>
> Looking at the commands you're using in your script:  After a user has run
> the "ant ivy-bootstrap" command once, ivy is downloaded into the user's home
> directory and does not need to be downloaded again.  Only the "ant package"
> command (run in the "solr" subdirectory) is actually needed to build Solr.
> The rest of the commands are not needed.
>
> As Emir said, you don't need to build Solr at all, even when using custom
> plugins.  You can download and use the binary package.
>
> There is effort underway to solve the problem with Solr tests. The initial
> phase of that effort is to disable the tests that fail most frequently.  The
> second overlapping phase of the effort is to actually fix those tests so
> that they don't fail - either by fixing bugs in the tests themselves, or by
> fixing real bugs in Solr.
>
>> Also, does ant version has any effects in build??
>
>
> Ant 1.8 and 1.9 should work.  Versions 1.10.0, 1.10.1, as well as 1.10.3 and
> later should be fine, but 1.10.2 has a bug that results in the lucene-solr
> build failing:
>
> https://issues.apache.org/jira/browse/LUCENE-8189
>
>> At last, at present, we are using solr-6.4.2 which has zookeeper-3.4.6
>> dependency but for solr-7, the zookeeper dependency has been upgraded to
>> 3.4.10, so my question is, At what extent does this might affect our
>> system
>> performance? Can we use zookeeper-3.4.6 with solr-7?
>> (same with the jetty version)
>
>
> You should be able to use any ZK 3.4.x server version with any version of
> Solr.  Most versions of Solr should also work with 3.5.x (still in beta)
> servers.  Early 4.x version s shipped with ZK 3.3.x, and the ZK project does
> not guarantee compatibility between 3.3.x and 3.5.x.
>
> I can't guarantee that you won't run into bugs, but ZK is generally a very
> stable piece of software.  Each new release of ZK includes a very large list
> of bugfixes.  I have no idea what implications there are for performance.
> You would need to ask a ZK support resource that question.  The latest
> stable release that is compatible with your software is the recommended
> version.  Currently that is 3.4.12.  The 3.5.x releases are in beta.
>
> Thanks,
> Shawn
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr7.3.1 Installation

Jason Gerlowski
(I think Erick made a slight typo above: to disable "bad apple" tests,
use the flag "-Dtests.badapples=false")
On Wed, Jul 11, 2018 at 11:14 AM Erick Erickson <[hidden email]> wrote:

>
> Note that the native test runs have the know-flaky tests _enabled_ by
> default, run tests with
>
> -Dtests.badapples=true
>
> to disable them.
>
> Second possibility is to look at the tests that failed and if there is
> an annotation
> @BadApple
> or
> @AwaitsFix
> ignore the failure if you can get the tests to pass when running individually.
>
> As Shawn says, this is a known issue that we're working on, but the
> technical debt is such that it'll
> be a long-term issue to fix.
>
> Best,
> Erick
>
>
>
> On Wed, Jul 11, 2018 at 7:13 AM, Shawn Heisey <[hidden email]> wrote:
> > On 7/10/2018 11:20 PM, tapan1707 wrote:
> >>
> >> We are trying to install solr-7.3.1 into our existing system (We have also
> >> made some changes by adding one custom query parser).
> >>
> >> I am having some build issues and it would be really helpful if someone
> >> can
> >> help.
> >>
> >> While running ant test(in the process of building the solr package), it
> >> terminates because of failed tests.
> >
> >
> > This is a known problem.  Solr's tests are in not in a good state.
> > Sometimes they pass, sometimes they fail.  Since there are so many tests and
> > a fair number of them do fail intermittently, this creates a situation where
> > on most test runs, there is at least one test failure.  Run the tests enough
> > times, and eventually they will all pass ... but this usually takes many
> > runs.
> >
> > Looking at the commands you're using in your script:  After a user has run
> > the "ant ivy-bootstrap" command once, ivy is downloaded into the user's home
> > directory and does not need to be downloaded again.  Only the "ant package"
> > command (run in the "solr" subdirectory) is actually needed to build Solr.
> > The rest of the commands are not needed.
> >
> > As Emir said, you don't need to build Solr at all, even when using custom
> > plugins.  You can download and use the binary package.
> >
> > There is effort underway to solve the problem with Solr tests. The initial
> > phase of that effort is to disable the tests that fail most frequently.  The
> > second overlapping phase of the effort is to actually fix those tests so
> > that they don't fail - either by fixing bugs in the tests themselves, or by
> > fixing real bugs in Solr.
> >
> >> Also, does ant version has any effects in build??
> >
> >
> > Ant 1.8 and 1.9 should work.  Versions 1.10.0, 1.10.1, as well as 1.10.3 and
> > later should be fine, but 1.10.2 has a bug that results in the lucene-solr
> > build failing:
> >
> > https://issues.apache.org/jira/browse/LUCENE-8189
> >
> >> At last, at present, we are using solr-6.4.2 which has zookeeper-3.4.6
> >> dependency but for solr-7, the zookeeper dependency has been upgraded to
> >> 3.4.10, so my question is, At what extent does this might affect our
> >> system
> >> performance? Can we use zookeeper-3.4.6 with solr-7?
> >> (same with the jetty version)
> >
> >
> > You should be able to use any ZK 3.4.x server version with any version of
> > Solr.  Most versions of Solr should also work with 3.5.x (still in beta)
> > servers.  Early 4.x version s shipped with ZK 3.3.x, and the ZK project does
> > not guarantee compatibility between 3.3.x and 3.5.x.
> >
> > I can't guarantee that you won't run into bugs, but ZK is generally a very
> > stable piece of software.  Each new release of ZK includes a very large list
> > of bugfixes.  I have no idea what implications there are for performance.
> > You would need to ask a ZK support resource that question.  The latest
> > stable release that is compatible with your software is the recommended
> > version.  Currently that is 3.4.12.  The 3.5.x releases are in beta.
> >
> > Thanks,
> > Shawn
> >
Reply | Threaded
Open this post in threaded view
|

Re: Solr7.3.1 Installation

Erick Erickson
Gah! Jason is right.... Siiiggggh. That'll teach me to try to do two
things at once.

On Wed, Jul 11, 2018 at 11:02 AM, Jason Gerlowski <[hidden email]> wrote:

> (I think Erick made a slight typo above: to disable "bad apple" tests,
> use the flag "-Dtests.badapples=false")
> On Wed, Jul 11, 2018 at 11:14 AM Erick Erickson <[hidden email]> wrote:
>>
>> Note that the native test runs have the know-flaky tests _enabled_ by
>> default, run tests with
>>
>> -Dtests.badapples=true
>>
>> to disable them.
>>
>> Second possibility is to look at the tests that failed and if there is
>> an annotation
>> @BadApple
>> or
>> @AwaitsFix
>> ignore the failure if you can get the tests to pass when running individually.
>>
>> As Shawn says, this is a known issue that we're working on, but the
>> technical debt is such that it'll
>> be a long-term issue to fix.
>>
>> Best,
>> Erick
>>
>>
>>
>> On Wed, Jul 11, 2018 at 7:13 AM, Shawn Heisey <[hidden email]> wrote:
>> > On 7/10/2018 11:20 PM, tapan1707 wrote:
>> >>
>> >> We are trying to install solr-7.3.1 into our existing system (We have also
>> >> made some changes by adding one custom query parser).
>> >>
>> >> I am having some build issues and it would be really helpful if someone
>> >> can
>> >> help.
>> >>
>> >> While running ant test(in the process of building the solr package), it
>> >> terminates because of failed tests.
>> >
>> >
>> > This is a known problem.  Solr's tests are in not in a good state.
>> > Sometimes they pass, sometimes they fail.  Since there are so many tests and
>> > a fair number of them do fail intermittently, this creates a situation where
>> > on most test runs, there is at least one test failure.  Run the tests enough
>> > times, and eventually they will all pass ... but this usually takes many
>> > runs.
>> >
>> > Looking at the commands you're using in your script:  After a user has run
>> > the "ant ivy-bootstrap" command once, ivy is downloaded into the user's home
>> > directory and does not need to be downloaded again.  Only the "ant package"
>> > command (run in the "solr" subdirectory) is actually needed to build Solr.
>> > The rest of the commands are not needed.
>> >
>> > As Emir said, you don't need to build Solr at all, even when using custom
>> > plugins.  You can download and use the binary package.
>> >
>> > There is effort underway to solve the problem with Solr tests. The initial
>> > phase of that effort is to disable the tests that fail most frequently.  The
>> > second overlapping phase of the effort is to actually fix those tests so
>> > that they don't fail - either by fixing bugs in the tests themselves, or by
>> > fixing real bugs in Solr.
>> >
>> >> Also, does ant version has any effects in build??
>> >
>> >
>> > Ant 1.8 and 1.9 should work.  Versions 1.10.0, 1.10.1, as well as 1.10.3 and
>> > later should be fine, but 1.10.2 has a bug that results in the lucene-solr
>> > build failing:
>> >
>> > https://issues.apache.org/jira/browse/LUCENE-8189
>> >
>> >> At last, at present, we are using solr-6.4.2 which has zookeeper-3.4.6
>> >> dependency but for solr-7, the zookeeper dependency has been upgraded to
>> >> 3.4.10, so my question is, At what extent does this might affect our
>> >> system
>> >> performance? Can we use zookeeper-3.4.6 with solr-7?
>> >> (same with the jetty version)
>> >
>> >
>> > You should be able to use any ZK 3.4.x server version with any version of
>> > Solr.  Most versions of Solr should also work with 3.5.x (still in beta)
>> > servers.  Early 4.x version s shipped with ZK 3.3.x, and the ZK project does
>> > not guarantee compatibility between 3.3.x and 3.5.x.
>> >
>> > I can't guarantee that you won't run into bugs, but ZK is generally a very
>> > stable piece of software.  Each new release of ZK includes a very large list
>> > of bugfixes.  I have no idea what implications there are for performance.
>> > You would need to ask a ZK support resource that question.  The latest
>> > stable release that is compatible with your software is the recommended
>> > version.  Currently that is 3.4.12.  The 3.5.x releases are in beta.
>> >
>> > Thanks,
>> > Shawn
>> >
Reply | Threaded
Open this post in threaded view
|

Re: Solr7.3.1 Installation

tapan1707
Thanks, everyone for your replies.

I tried using "-Dtests.badapples=false" flag but it didn't help.

I will try again without *ant test* command.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html