Things are not quite stable...

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Things are not quite stable...

Jack L

I'm running the default solr package with my data, about 10 million
small documents.  I'm not sure if it's jetty or solr, but often
times solr stops functioning properly after running for a few days.
The symptom is search returning nothing, or when I go to /solr/admin/,
I get file browsing page showing a list of files (.css, etc),
not the admin interface page. Typically restarting solr solves this
kind of problems. Is solr unstable? Or is it jetty?

Or it's something that I'm doing wrong? What is everyone else's
experience?

The jetty in the download seems to be Jetty/5.1.11RC0. The latest
Jetty version is 6.1 from their site. I don't know much about jetty
but I'm always hoping that the new versions are better than old
versions. Is there any plan to upgrade the embedded Jetty?
Will there be any benefit to upgrade, in terms of stability?

--
Best regards,
Jack

Reply | Threaded
Open this post in threaded view
|

Re: Things are not quite stable...

Bertrand Delacretaz
On 4/25/07, Jack L <[hidden email]> wrote:

> ...solr stops functioning properly after running for a few days.
> The symptom is search returning nothing, or when I go to /solr/admin/,
> I get file browsing page showing a list of files (.css, etc),...

Are your symptoms similar to those of
https://issues.apache.org/jira/browse/SOLR-118 ?

I'm seeing this as well on a Solr instance that runs a snapshot from
early 2007, but I haven't found exactly what's going on yet. In our
case, searches continue to work, but it seems like the part of Jetty
that serves static files stops working.

It might be related to some "too many open files" problem, maybe
https://issues.apache.org/jira/browse/SOLR-173

Are you running a Solr release, or a snapshot? It'd be interesting to
know If you're running code that predates the fix done in SOLR-173. In
my case, on the production system the code is older than that.

-Bertrand
Reply | Threaded
Open this post in threaded view
|

Re[2]: Things are not quite stable...

Jack L
Hello Bertrand,

>> ...solr stops functioning properly after running for a few days.
>> The symptom is search returning nothing, or when I go to /solr/admin/,
>> I get file browsing page showing a list of files (.css, etc),...

> Are your symptoms similar to those of
> https://issues.apache.org/jira/browse/SOLR-118 ?

Yes. Something similar. Sometimes, when the admin page is dead,
solr still functions. Sometimes it doesn't (less often than the
former case.)

> Are you running a Solr release, or a snapshot? It'd be interesting to
> know If you're running code that predates the fix done in SOLR-173. In
> my case, on the production system the code is older than that.

I'm running the 1.1.0 release.

Maybe it's time to think about upgrading Jetty.
Using jetty RC version in solr Release version is less than ideal
anyway.

Jack

Reply | Threaded
Open this post in threaded view
|

Re: Re[2]: Things are not quite stable...

Bertrand Delacretaz
On 4/25/07, Jack L <[hidden email]> wrote:

> ...Maybe it's time to think about upgrading Jetty...

It's in the pipeline, see https://issues.apache.org/jira/browse/SOLR-128

-Bertrand
Reply | Threaded
Open this post in threaded view
|

Re: Re[2]: Things are not quite stable...

Mike Klaas
In reply to this post by Jack L
On 4/25/07, Jack L <[hidden email]> wrote:

> > Are you running a Solr release, or a snapshot? It'd be interesting to
> > know If you're running code that predates the fix done in SOLR-173. In
> > my case, on the production system the code is older than that.
>
> I'm running the 1.1.0 release.
>
> Maybe it's time to think about upgrading Jetty.
> Using jetty RC version in solr Release version is less than ideal
> anyway.

I believe it is being considered, but keep in mind that the container
shipped with Solr is just an example, and is not meant to be an
endorsement of or limitation to a particular package.

-Mike
Reply | Threaded
Open this post in threaded view
|

Re[4]: Things are not quite stable...

Jack L
That is understood.

For people who are not familiar with Java servlet containers, like
myself, the embedded web server takes the least effort to set up.
And jetty is supposed be a production quality quality package, it'll be
great that the default package with jetty can be used for production.

--
Jack

> I believe it is being considered, but keep in mind that the container
> shipped with Solr is just an example, and is not meant to be an
> endorsement of or limitation to a particular package.

> -Mike

Reply | Threaded
Open this post in threaded view
|

Re: Re[4]: Things are not quite stable...

Mike Klaas
On 4/25/07, Jack L <[hidden email]> wrote:
> That is understood.
>
> For people who are not familiar with Java servlet containers, like
> myself, the embedded web server takes the least effort to set up.
> And jetty is supposed be a production quality quality package, it'll be
> great that the default package with jetty can be used for production.

Do you know that Jetty is the culprit?  We've been successfully using
it for production purposes.

-Mike
Reply | Threaded
Open this post in threaded view
|

Re[6]: Things are not quite stable...

Jack L
Mike,

I don't know for sure. But from the symptom, when error 404 happens,
solr seems to still work fine (so does that part of Jetty.) Then those
404'ed pages may have some solr specific logic in it?

Regardless, I think it's a good idea to use a newer, released (not RC)
version in general, considering 5.1 is one major version behind.
And I'm glad to know that jetty works well for you - because I'm counting
on it, and am not planning on learning Tomcat soon :) Do you mind mentioning
what version of Jetty you are using in production?

--
Best regards,
Jack

Wednesday, April 25, 2007, 12:27:41 PM, you wrote:

> On 4/25/07, Jack L <[hidden email]> wrote:
>> That is understood.
>>
>> For people who are not familiar with Java servlet containers, like
>> myself, the embedded web server takes the least effort to set up.
>> And jetty is supposed be a production quality quality package, it'll be
>> great that the default package with jetty can be used for production.

> Do you know that Jetty is the culprit?  We've been successfully using
> it for production purposes.

> -Mike

Reply | Threaded
Open this post in threaded view
|

Re: Re[4]: Things are not quite stable...

Erik Hatcher
In reply to this post by Mike Klaas

On Apr 25, 2007, at 3:27 PM, Mike Klaas wrote:

> On 4/25/07, Jack L <[hidden email]> wrote:
>> That is understood.
>>
>> For people who are not familiar with Java servlet containers, like
>> myself, the embedded web server takes the least effort to set up.
>> And jetty is supposed be a production quality quality package,  
>> it'll be
>> great that the default package with jetty can be used for production.
>
> Do you know that Jetty is the culprit?  We've been successfully using
> it for production purposes.

Same here.  It's been solid for us on several applications already,  
just using the example Jetty binary.

        Erik

Reply | Threaded
Open this post in threaded view
|

Re: Re[6]: Things are not quite stable...

Mike Klaas
In reply to this post by Jack L
On 4/25/07, Jack L <[hidden email]> wrote:

> I don't know for sure. But from the symptom, when error 404 happens,
> solr seems to still work fine (so does that part of Jetty.) Then those
> 404'ed pages may have some solr specific logic in it?

I'm really not sure--I've built several long-running, huge indices and
never experienced this issue.

> Regardless, I think it's a good idea to use a newer, released (not RC)
> version in general, considering 5.1 is one major version behind.
> And I'm glad to know that jetty works well for you - because I'm counting
> on it, and am not planning on learning Tomcat soon :) Do you mind mentioning
> what version of Jetty you are using in production?

Sure: I'm using the provided, 5.1RC0 version.  I didn't know anything
about servlet containers when I first started using Solr (and still
don't, really), and since the provided container has never given me
any (significant[1]) issues, I've kept with it.

[1] Aside from XML-escaping irregularities that were discussed on the
list last year.

-Mike
Reply | Threaded
Open this post in threaded view
|

Re: Re[6]: Things are not quite stable...

Bertrand Delacretaz
In reply to this post by Jack L
On 4/25/07, Jack L <[hidden email]> wrote:

> ...Regardless, I think it's a good idea to use a newer, released (not RC)
> version in general, considering 5.1 is one major version behind....

Agreed, but note that we don't have any factual evidence that the
Jetty RC that we use is indeed the cause of SOLR-118, so upgrading
might not solve the problem.

We're just at the wild guess stage at this point, and many of us have
never seen the problem. In my case, we have more urgent stuff to do
before looking at the problem in more detail.

-Bertrand
Reply | Threaded
Open this post in threaded view
|

Re: Re[4]: Things are not quite stable...

Otis Gospodnetic-2
In reply to this post by Jack L
Jetty is quite good and its developers are very responsive and agile.  Simpy.com is a fairly high-traffic site and runs Jetty (5.*, soon 6.*) behind Apache.

Otis
 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share

----- Original Message ----
From: Erik Hatcher <[hidden email]>
To: [hidden email]
Sent: Wednesday, April 25, 2007 4:26:41 PM
Subject: Re: Re[4]: Things are not quite stable...


On Apr 25, 2007, at 3:27 PM, Mike Klaas wrote:

> On 4/25/07, Jack L <[hidden email]> wrote:
>> That is understood.
>>
>> For people who are not familiar with Java servlet containers, like
>> myself, the embedded web server takes the least effort to set up.
>> And jetty is supposed be a production quality quality package,  
>> it'll be
>> great that the default package with jetty can be used for production.
>
> Do you know that Jetty is the culprit?  We've been successfully using
> it for production purposes.

Same here.  It's been solid for us on several applications already,  
just using the example Jetty binary.

    Erik




Reply | Threaded
Open this post in threaded view
|

Re[8]: Things are not quite stable...

Jack L
In reply to this post by Bertrand Delacretaz
Hello Bertrand,

Is there a build script that automagically grab files from jetty's
source tree (local) and build a solr release? In other words, I can
try building with a newer version of jetty if it doesn't take too
much work - I don't know much about jetty or solr at the code level.

--
Best regards,
Jack

> Agreed, but note that we don't have any factual evidence that the
> Jetty RC that we use is indeed the cause of SOLR-118, so upgrading
> might not solve the problem.

> We're just at the wild guess stage at this point, and many of us have
> never seen the problem. In my case, we have more urgent stuff to do
> before looking at the problem in more detail.

> -Bertrand

Reply | Threaded
Open this post in threaded view
|

Re: Re[8]: Things are not quite stable...

Bertrand Delacretaz
Hi Jack,

On 4/27/07, Jack L <[hidden email]> wrote:
> ...Is there a build script that automagically grab files from jetty's
> source tree (local) and build a solr release? In other words, I can
> try building with a newer version of jetty if it doesn't take too
> much work - I don't know much about jetty or solr at the code level....

You don't need the source code of Jetty to do that, jars from a
released version will do, probably with some changes to our Jetty
configuration files, according to Jetty's docs.

For the Solr example configuration, Jetty configs are in example/etc/,
and the Jetty jars are in example/lib/ (except start.jar which might
also change with newer Jetty releases).

If you adapt them to a more recent Jetty, patches would be appreciated!

http://wiki.apache.org/solr/SolrJetty has some additional info.

HTH,
-Bertrand