JVM best tune? help ... solr1.4

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

JVM best tune? help ... solr1.4

sunnyShiny06
Hi,

Just applied replication by requestHandler.
And since this the Qtime went mad and can reach long time <int name="QTime">9068</int>
Without this replication Qtime can be around 1sec.

I've 14Mdocs stores for 11G. so not a lot of data stores.
I've servers with 8G and tomcat use 7G.
I'm updating every 30mn which is about 50 000docs.
Have a look as well at my cpu which are aswell quite full ?

Have you an idea? Do I miss a patch ?
Thanks a lot,


Solr Specification Version: 1.3.0.2009.01.22.13.51.22
Solr Implementation Version: 1.4-dev exported - root - 2009-01-22 13:51:22

CPU.jpg
Reply | Threaded
Open this post in threaded view
|

Re: JVM best tune? help ... solr1.4

Noble Paul നോബിള്‍  नोब्ळ्
If you are looking at the QTime on the master it is likely to be
skewed by ReplicationHandler becaus ethe files are downloaded using a
request. On a slave it should not be a problem.

I guess we must not add the qtimes of ReplicationHandler
--Noble

On Thu, Apr 2, 2009 at 5:34 PM, sunnyfr <[hidden email]> wrote:

>
> Hi,
>
> Just applied replication by requestHandler.
> And since this the Qtime went mad and can reach long time <int
> name="QTime">9068</int>
> Without this replication Qtime can be around 1sec.
>
> I've 14Mdocs stores for 11G. so not a lot of data stores.
> I've servers with 8G and tomcat use 7G.
> I'm updating every 30mn which is about 50 000docs.
> Have a look as well at my cpu which are aswell quite full ?
>
> Have you an idea? Do I miss a patch ?
> Thanks a lot,
>
> Solr Specification Version: 1.3.0.2009.01.22.13.51.22
> Solr Implementation Version: 1.4-dev exported - root - 2009-01-22 13:51:22
>
> http://www.nabble.com/file/p22846336/CPU.jpg CPU.jpg
> --
> View this message in context: http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846336.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: JVM best tune? help ... solr1.4

sunnyShiny06
I think about the slave.
When I start in multi thread 20 request second my cpu is very bad.
I'm sure I don't manage properly my gc. I've 8G per slave it should be fine.

I wonder, I shouldn't put 7G to xmx jvm, I don't know,
but slave is as well a little problem during replication from the master.


Noble Paul നോബിള്‍ नोब्ळ् wrote
If you are looking at the QTime on the master it is likely to be
skewed by ReplicationHandler becaus ethe files are downloaded using a
request. On a slave it should not be a problem.

I guess we must not add the qtimes of ReplicationHandler
--Noble

On Thu, Apr 2, 2009 at 5:34 PM, sunnyfr <johanna.34@gmail.com> wrote:
>
> Hi,
>
> Just applied replication by requestHandler.
> And since this the Qtime went mad and can reach long time <int
> name="QTime">9068</int>
> Without this replication Qtime can be around 1sec.
>
> I've 14Mdocs stores for 11G. so not a lot of data stores.
> I've servers with 8G and tomcat use 7G.
> I'm updating every 30mn which is about 50 000docs.
> Have a look as well at my cpu which are aswell quite full ?
>
> Have you an idea? Do I miss a patch ?
> Thanks a lot,
>
> Solr Specification Version: 1.3.0.2009.01.22.13.51.22
> Solr Implementation Version: 1.4-dev exported - root - 2009-01-22 13:51:22
>
> http://www.nabble.com/file/p22846336/CPU.jpg CPU.jpg
> --
> View this message in context: http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846336.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: JVM best tune? help ... solr1.4

Noble Paul നോബിള്‍  नोब्ळ्
http://people.apache.org/~hossman/#threadhijack

On Thu, Apr 2, 2009 at 5:47 PM, sunnyfr <[hidden email]> wrote:

>
> I think about the slave.
> When I start in multi thread 20 request second my cpu is very bad.
> I'm sure I don't manage properly my gc. I've 8G per slave it should be fine.
>
> I wonder, I shouldn't put 7G to xmx jvm, I don't know,
> but slave is as well a little problem during replication from the master.
>
>
>
> Noble Paul നോബിള്‍  नोब्ळ् wrote:
>>
>> If you are looking at the QTime on the master it is likely to be
>> skewed by ReplicationHandler becaus ethe files are downloaded using a
>> request. On a slave it should not be a problem.
>>
>> I guess we must not add the qtimes of ReplicationHandler
>> --Noble
>>
>> On Thu, Apr 2, 2009 at 5:34 PM, sunnyfr <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> Just applied replication by requestHandler.
>>> And since this the Qtime went mad and can reach long time <int
>>> name="QTime">9068</int>
>>> Without this replication Qtime can be around 1sec.
>>>
>>> I've 14Mdocs stores for 11G. so not a lot of data stores.
>>> I've servers with 8G and tomcat use 7G.
>>> I'm updating every 30mn which is about 50 000docs.
>>> Have a look as well at my cpu which are aswell quite full ?
>>>
>>> Have you an idea? Do I miss a patch ?
>>> Thanks a lot,
>>>
>>> Solr Specification Version: 1.3.0.2009.01.22.13.51.22
>>> Solr Implementation Version: 1.4-dev exported - root - 2009-01-22
>>> 13:51:22
>>>
>>> http://www.nabble.com/file/p22846336/CPU.jpg CPU.jpg
>>> --
>>> View this message in context:
>>> http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846336.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>>
>
> --
> View this message in context: http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846546.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: JVM best tune? help ... solr1.4

Noble Paul നോബിള്‍  नोब्ळ्
In reply to this post by sunnyShiny06
it is not a good idea to optimize the index everytime you commit. That
is why your downloads are taking so long

2009/4/2 Johanna Mortemousque <[hidden email]>:

> I've so many update, almost 2 000 every 20mn, that lucene merge my index
> folder,
> so everytime my slave replicate its a new index folder merged so every time
> it brings back 10G datas.
>
> And during this time my repond time of my request are very slow.
> What can I check?
>
> Thanks Paul
>
> 2009/4/2 Noble Paul നോബിള്‍ नोब्ळ् <[hidden email]>
>>
>> slave would not show increased request times because of replication.
>> If it does should be some bug
>>
>> On Thu, Apr 2, 2009 at 6:00 PM,  <[hidden email]> wrote:
>> > I think its the same problem, tune jvm for multi thread ... 20request
>> > seconde.
>> > no??
>> >
>> >
>> >
>> > Noble Paul നോബിള്‍  नोब्ळ् wrote:
>> >>
>> >> http://people.apache.org/~hossman/#threadhijack
>> >>
>> >> On Thu, Apr 2, 2009 at 5:47 PM, sunnyfr <[hidden email]> wrote:
>> >>>
>> >>> I think about the slave.
>> >>> When I start in multi thread 20 request second my cpu is very bad.
>> >>> I'm sure I don't manage properly my gc. I've 8G per slave it should be
>> >>> fine.
>> >>>
>> >>> I wonder, I shouldn't put 7G to xmx jvm, I don't know,
>> >>> but slave is as well a little problem during replication from the
>> >>> master.
>> >>>
>> >>>
>> >>>
>> >>> Noble Paul നോബിള്‍  नोब्ळ् wrote:
>> >>>>
>> >>>> If you are looking at the QTime on the master it is likely to be
>> >>>> skewed by ReplicationHandler becaus ethe files are downloaded using a
>> >>>> request. On a slave it should not be a problem.
>> >>>>
>> >>>> I guess we must not add the qtimes of ReplicationHandler
>> >>>> --Noble
>> >>>>
>> >>>> On Thu, Apr 2, 2009 at 5:34 PM, sunnyfr <[hidden email]> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> Just applied replication by requestHandler.
>> >>>>> And since this the Qtime went mad and can reach long time <int
>> >>>>> name="QTime">9068</int>
>> >>>>> Without this replication Qtime can be around 1sec.
>> >>>>>
>> >>>>> I've 14Mdocs stores for 11G. so not a lot of data stores.
>> >>>>> I've servers with 8G and tomcat use 7G.
>> >>>>> I'm updating every 30mn which is about 50 000docs.
>> >>>>> Have a look as well at my cpu which are aswell quite full ?
>> >>>>>
>> >>>>> Have you an idea? Do I miss a patch ?
>> >>>>> Thanks a lot,
>> >>>>>
>> >>>>> Solr Specification Version: 1.3.0.2009.01.22.13.51.22
>> >>>>> Solr Implementation Version: 1.4-dev exported - root - 2009-01-22
>> >>>>> 13:51:22
>> >>>>>
>> >>>>> http://www.nabble.com/file/p22846336/CPU.jpg CPU.jpg
>> >>>>> --
>> >>>>> View this message in context:
>> >>>>>
>> >>>>> http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846336.html
>> >>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> --Noble Paul
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>> >>> http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846546.html
>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> --Noble Paul
>> >>
>> >>
>> > Quoted from:
>> >
>> > http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846716.html
>> >
>> >
>>
>>
>>
>> --
>> --Noble Paul
>
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: JVM best tune? help ... solr1.4

sunnyShiny06
I don't optimize at all.
my delta-import&optimize=false

I didn't turnd on optimize, I think it merges segment alone, because size increase too quickly?

Noble Paul നോബിള്‍ नोब्ळ् wrote
it is not a good idea to optimize the index everytime you commit. That
is why your downloads are taking so long

2009/4/2 Johanna Mortemousque <johanna.34@gmail.com>:
> I've so many update, almost 2 000 every 20mn, that lucene merge my index
> folder,
> so everytime my slave replicate its a new index folder merged so every time
> it brings back 10G datas.
>
> And during this time my repond time of my request are very slow.
> What can I check?
>
> Thanks Paul
>
> 2009/4/2 Noble Paul നോബിള്‍ नोब्ळ् <noble.paul@gmail.com>
>>
>> slave would not show increased request times because of replication.
>> If it does should be some bug
>>
>> On Thu, Apr 2, 2009 at 6:00 PM,  <johanna.34@gmail.com> wrote:
>> > I think its the same problem, tune jvm for multi thread ... 20request
>> > seconde.
>> > no??
>> >
>> >
>> >
>> > Noble Paul നോബിള്‍  नोब्ळ् wrote:
>> >>
>> >> http://people.apache.org/~hossman/#threadhijack
>> >>
>> >> On Thu, Apr 2, 2009 at 5:47 PM, sunnyfr <johanna.34@gmail.com> wrote:
>> >>>
>> >>> I think about the slave.
>> >>> When I start in multi thread 20 request second my cpu is very bad.
>> >>> I'm sure I don't manage properly my gc. I've 8G per slave it should be
>> >>> fine.
>> >>>
>> >>> I wonder, I shouldn't put 7G to xmx jvm, I don't know,
>> >>> but slave is as well a little problem during replication from the
>> >>> master.
>> >>>
>> >>>
>> >>>
>> >>> Noble Paul നോബിള്‍  नोब्ळ् wrote:
>> >>>>
>> >>>> If you are looking at the QTime on the master it is likely to be
>> >>>> skewed by ReplicationHandler becaus ethe files are downloaded using a
>> >>>> request. On a slave it should not be a problem.
>> >>>>
>> >>>> I guess we must not add the qtimes of ReplicationHandler
>> >>>> --Noble
>> >>>>
>> >>>> On Thu, Apr 2, 2009 at 5:34 PM, sunnyfr <johanna.34@gmail.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> Just applied replication by requestHandler.
>> >>>>> And since this the Qtime went mad and can reach long time <int
>> >>>>> name="QTime">9068</int>
>> >>>>> Without this replication Qtime can be around 1sec.
>> >>>>>
>> >>>>> I've 14Mdocs stores for 11G. so not a lot of data stores.
>> >>>>> I've servers with 8G and tomcat use 7G.
>> >>>>> I'm updating every 30mn which is about 50 000docs.
>> >>>>> Have a look as well at my cpu which are aswell quite full ?
>> >>>>>
>> >>>>> Have you an idea? Do I miss a patch ?
>> >>>>> Thanks a lot,
>> >>>>>
>> >>>>> Solr Specification Version: 1.3.0.2009.01.22.13.51.22
>> >>>>> Solr Implementation Version: 1.4-dev exported - root - 2009-01-22
>> >>>>> 13:51:22
>> >>>>>
>> >>>>> http://www.nabble.com/file/p22846336/CPU.jpg CPU.jpg
>> >>>>> --
>> >>>>> View this message in context:
>> >>>>>
>> >>>>> http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846336.html
>> >>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> --Noble Paul
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>> >>> http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846546.html
>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> --Noble Paul
>> >>
>> >>
>> > Quoted from:
>> >
>> > http://www.nabble.com/JVM-best-tune--help-...-solr1.4-tp22846336p22846716.html
>> >
>> >
>>
>>
>>
>> --
>> --Noble Paul
>
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: JVM best tune? help ... solr1.4

sunnyShiny06
In reply to this post by Noble Paul നോബിള്‍ नोब्ळ्
This is my conf :

solrconfig.xml

And this is my delta import:

*/20 * * * * /usr/bin/wget -q --output-document=/home/video_import.txt http:/master.com:8180/solr/video/dataimport?command=delta-import&optimize=false