Some performance questions....

classic Classic list List threaded Threaded
64 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

BlackIce
Shawn:
well the idea was to utilize system resources more efficiently.. this is
not due so much to Solr, as I sayd I don't know that much about Solr,
except Shema.xml and Solarconfig.xml - However the main app that will be
running is more or less a single threated app which takes advantage when
run under several instances, ie: parallelism, so I thought, since I'm at it
I may give solr a few instances as well... but the more I read, the more
confused I get.. I've read about some guy running 8 Solr instances on his
dual Xeon 26xx series, each VM with 12 GB ram......

Deepak:

Well its kinda a given that when running ANYTHING under a VM you have an
overhead.. so since I control the hardware, ie: not sharing space on some
hosted VM by some ISP... why not skip the whole VM thing entirely?

Thnx for the Heap pointer.. I've read, from some Professor.. that Solr
actually is more efficient with a very small Heap and to have everything
mapped to virtual memory... Which brings me to the next question.. is the
Virtual memory mapping done by the OS or Solar? Does the Virtual memory
reside on the OS HDD? Or on the Solr HDD?.. and if the Virtual memory
mapping is done on the OS HDD, wouldn't it be beneficial to run the OS off
a SSD?

For now.. my FEELING is to run one Solr instance on this particular
machine.. by the time the RAM is outgrown add another machine and so
forth... I've had a small set-back: due to the chasis configuration I could
only fit in Half of the HDD's I intented.. the rest collide with the CPU
heatsinks (Don't ask)
 so my entire initial set-up has changed and with it my initial "growth
strategy"

On Wed, Mar 14, 2018 at 4:15 PM, Shawn Heisey <[hidden email]> wrote:

> On 3/14/2018 5:49 AM, BlackIce wrote:
>
>> I was just thinking.... Do I really need separate VM's in order to run
>> multiple Solr instances? Doesn't it suffice to have each instance in its
>> own user account?
>>
>
> You can run multiple instances all under the same account on one machine.
> But for a single machine, why do you need multiple Solr instances at all?
> One instance can handle many indexes, and will probably do it more
> efficiently than multiple instances.
>
> The only time I would *ever* recommend multiple Solr instances is when a
> single instance would need an ENORMOUS Java heap -- something much larger
> than 32GB.  If something like that can be split into multiple instances
> where each one has a heap that's 31GB heap or less, then memory usage will
> be more efficient and Java's garbage collection will work better.
>
> FYI -- Running Java with a 32GB heap actually has LESS memory available
> than running it with a 31GB heap.  This is because when the heap reaches
> 32GB, Java must switch to 64-bit pointers, so every little allocation
> requires a little bit more memory.
>
> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Deepak Goel
Please see inline...



Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
[hidden email]

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Thu, Mar 15, 2018 at 6:04 PM, BlackIce <[hidden email]> wrote:

> Shawn:
> well the idea was to utilize system resources more efficiently.. this is
> not due so much to Solr, as I sayd I don't know that much about Solr,
> except Shema.xml and Solarconfig.xml - However the main app that will be
> running is more or less a single threated app which takes advantage when
> run under several instances, ie: parallelism, so I thought, since I'm at it
> I may give solr a few instances as well... but the more I read, the more
> confused I get.. I've read about some guy running 8 Solr instances on his
> dual Xeon 26xx series, each VM with 12 GB ram......
>
> Deepak:
>
> Well its kinda a given that when running ANYTHING under a VM you have an
> overhead..

***Deepak***
You mean you are assuming without any facts (performance benchmark with n
without VM)
 ***Deepak***

> so since I control the hardware, ie: not sharing space on some
> hosted VM by some ISP... why not skip the whole VM thing entirely?
>
> Thnx for the Heap pointer.. I've read, from some Professor.. that Solr
> actually is more efficient with a very small Heap and to have everything
> mapped to virtual memory... Which brings me to the next question.. is the
> Virtual memory mapping done by the OS or Solar? Does the Virtual memory
> reside on the OS HDD? Or on the Solr HDD?.. and if the Virtual memory
> mapping is done on the OS HDD, wouldn't it be beneficial to run the OS off
> a SSD?
>
> ***Deepak***
The OS does mapping itself to virtual memory (Atleast Unix does). However
am not sure of the internal mechanism of Solr
***Deepak***


> For now.. my FEELING is to run one Solr instance on this particular
> machine.. by the time the RAM is outgrown add another machine and so
> forth...

***Deepak***
I wonder if there are any performance benchmarks showing how Solr scales at
higher loads on a single machine (is it linear or non linear). Most
software don't scale linearly at higher loads
 ***Deepak***

> I've had a small set-back: due to the chasis configuration I could
> only fit in Half of the HDD's I intented.. the rest collide with the CPU
> heatsinks (Don't ask)
>  so my entire initial set-up has changed and with it my initial "growth
> strategy"
>
> On Wed, Mar 14, 2018 at 4:15 PM, Shawn Heisey <[hidden email]> wrote:
>
> > On 3/14/2018 5:49 AM, BlackIce wrote:
> >
> >> I was just thinking.... Do I really need separate VM's in order to run
> >> multiple Solr instances? Doesn't it suffice to have each instance in its
> >> own user account?
> >>
> >
> > You can run multiple instances all under the same account on one machine.
> > But for a single machine, why do you need multiple Solr instances at all?
> > One instance can handle many indexes, and will probably do it more
> > efficiently than multiple instances.
> >
> > The only time I would *ever* recommend multiple Solr instances is when a
> > single instance would need an ENORMOUS Java heap -- something much larger
> > than 32GB.  If something like that can be split into multiple instances
> > where each one has a heap that's 31GB heap or less, then memory usage
> will
> > be more efficient and Java's garbage collection will work better.
> >
> > FYI -- Running Java with a 32GB heap actually has LESS memory available
> > than running it with a 31GB heap.  This is because when the heap reaches
> > 32GB, Java must switch to 64-bit pointers, so every little allocation
> > requires a little bit more memory.
> >
> > Thanks,
> > Shawn
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Alessandro Benedetti
*Single Solr Instance VS Multiple Solr instances on Single Server
*

I think there is no benefit in having multiple Solr instances on a single
server, unless the heap memory required by the JVM is too big.
And remember that this has relatively to do with the index size ( inverted
index is memory mapped OFF heap and docValues as well).
On the other hand of course Apache Solr uses plenty of JVM heap memory as
well ( caches, temporary data structures during indexing, ect ect)

> Deepak:
>
> Well its kinda a given that when running ANYTHING under a VM you have an
> overhead..

***Deepak***
You mean you are assuming without any facts (performance benchmark with n
without VM)
 ***Deepak***
I think Shawn detailed this quite extensively, I am no sys admin or OS
expert, but there is no need of benchmarks and I don't even understand your
doubts.
In Information technology anytime you add additional layers of software you
need adapters which means additional instructions executed.
It is obvious  that having :
metal -> OS -> APP is cheaper instruction wise then
metal -> OS -> VM -> APP
The APP will execute instruction in the VM that will be responsible to
translate those instructions for the underlining OS.
Going direct you skip one passage.
you can think about this when you emulate different OS, is it cheaper to run
windows on a machine directly to execute windows applications or run a
Windows VM on top of another OS to execute windows applications ?



-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Shawn Heisey
In reply to this post by BlackIce
On 3/15/2018 6:34 AM, BlackIce wrote:
> However the main app that will be
> running is more or less a single threated app which takes advantage when
> run under several instances, ie: parallelism, so I thought, since I'm at it
> I may give solr a few instances as well

Solr is a fully threaded app, capable of doing LOTS of things at the
same time, without multiple instances.

> Thnx for the Heap pointer.. I've read, from some Professor.. that Solr
> actually is more efficient with a very small Heap and to have everything
> mapped to virtual memory... Which brings me to the next question.. is the
> Virtual memory mapping done by the OS or Solar? Does the Virtual memory
> reside on the OS HDD? Or on the Solr HDD?.. and if the Virtual memory
> mapping is done on the OS HDD, wouldn't it be beneficial to run the OS off
> a SSD?

There appears to be some confusion here.

The virtual memory doesn't reside on ANY hard drive, unless you've
REALLY configured the system badly and the system starts using swap
space.  If the system starts using swap, performance is going to be
terrible, no matter how fast the disk where swap resides is.

The "mapping to virtual memory" feature is something the operating
system does.  Lucene/Solr utilizes MMAP code in Java, which then turns
around and uses MMAP functionality provided by the OS.

At that point, that file can be accessed by the application as if it
were a very large block of memory.  Mapping the file doesn't immediately
use any memory at all.  The OS manages the access to the file.  If the
part of the file that is being accessed has not been accessed before,
then the OS will read the data off the disk, place it into the OS disk
cache, and provide it to whatever requested it.  If it has been accessed
before and is still in the disk cache, then it won't read the disk, it
will just provide the data from the cache.  Getting most data from cache
is *required* for good Solr performance.

http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

Running with your indexes on SSD might indeed help performance, and
regardless of anything that's going on, WILL help performance in the
short term, when you first turn the machine on.  But if it also helps
with long-term query performance, then chances are that the machine
doesn't have enough memory.When Solr servers are sized correctly,
running on SSD is typically not going to make a big difference, unless
the machine does a lot more indexing than querying.

> For now.. my FEELING is to run one Solr instance on this particular
> machine.. by the time the RAM is outgrown add another machine and so
> forth...

Any plans you have for a growth strategy with multiple Solr instances
are extremely likely to still be possible with only one instance, with
very little change.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Deepak Goel
In reply to this post by Alessandro Benedetti
>I think there is no benefit in having multiple Solr instances on a single
>server, unless the heap memory required by the JVM is too big.
****Deepak***
I would try multiple Solr instances rather a single Solr instance (it
definitely will give a performance boost)
****Deepak***
>And remember that this has relatively to do with the index size ( inverted
>index is memory mapped OFF heap and docValues as well).
>On the other hand of course Apache Solr uses plenty of JVM heap memory as
>well ( caches, temporary data structures during indexing, ect ect)

> Deepak:
>
> Well its kinda a given that when running ANYTHING under a VM you have an
> overhead..

>***Deepak***
>You mean you are assuming without any facts (performance benchmark with n
>without VM)
 >***Deepak***

>I think Shawn detailed this quite extensively, I am no sys admin or OS
>expert, but there is no need of benchmarks and I don't even understand your
>doubts.
>In Information technology anytime you add additional layers of software you
>need adapters which means additional instructions executed.
>It is obvious  that having :
>metal -> OS -> APP is cheaper instruction wise then
>metal -> OS -> VM -> APP
>The APP will execute instruction in the VM that will be responsible to
>translate those instructions for the underlining OS.
****Deepak***
I had past experience with VM's. They absolutely do not take any overheads.
Since we have conflicting opinions, it is best to benchmark it yourself
****Deepak***
>Going direct you skip one passage.
>you can think about this when you emulate different OS, is it cheaper to
run
>windows on a machine directly to execute windows applications or run a
>Windows VM on top of another OS to execute windows applications ?









Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
[hidden email]

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Thu, Mar 15, 2018 at 9:43 PM, Alessandro Benedetti <[hidden email]>
wrote:

> *Single Solr Instance VS Multiple Solr instances on Single Server
> *
>
> I think there is no benefit in having multiple Solr instances on a single
> server, unless the heap memory required by the JVM is too big.
> And remember that this has relatively to do with the index size ( inverted
> index is memory mapped OFF heap and docValues as well).
> On the other hand of course Apache Solr uses plenty of JVM heap memory as
> well ( caches, temporary data structures during indexing, ect ect)
>
> > Deepak:
> >
> > Well its kinda a given that when running ANYTHING under a VM you have an
> > overhead..
>
> ***Deepak***
> You mean you are assuming without any facts (performance benchmark with n
> without VM)
>  ***Deepak***
> I think Shawn detailed this quite extensively, I am no sys admin or OS
> expert, but there is no need of benchmarks and I don't even understand your
> doubts.
> In Information technology anytime you add additional layers of software you
> need adapters which means additional instructions executed.
> It is obvious  that having :
> metal -> OS -> APP is cheaper instruction wise then
> metal -> OS -> VM -> APP
> The APP will execute instruction in the VM that will be responsible to
> translate those instructions for the underlining OS.
> Going direct you skip one passage.
> you can think about this when you emulate different OS, is it cheaper to
> run
> windows on a machine directly to execute windows applications or run a
> Windows VM on top of another OS to execute windows applications ?
>
>
>
> -----
> ---------------
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Deepak Goel
In reply to this post by Shawn Heisey
On Fri, Mar 16, 2018 at 6:03 PM, Shawn Heisey <[hidden email]> wrote:

> On 3/15/2018 6:34 AM, BlackIce wrote:
>
>> However the main app that will be
>> running is more or less a single threated app which takes advantage when
>> run under several instances, ie: parallelism, so I thought, since I'm at
>> it
>> I may give solr a few instances as well
>>
>
> ***Deepak***

I did a performance study of Solr a while back. And I found that it does
not scale beyond a particular point on a single machine (could be due to
the way its coded). Hence multiple instances might make sense.

https://docs.google.com/document/d/1kUqEcZl3NhOo6SLklo5Icg3fMnn9OtLY_lwnc6wbXus/edit?usp=sharing

***Deepak***



> Solr is a fully threaded app, capable of doing LOTS of things at the same
> time, without multiple instances.
>
> Thnx for the Heap pointer.. I've read, from some Professor.. that Solr
>> actually is more efficient with a very small Heap and to have everything
>> mapped to virtual memory... Which brings me to the next question.. is the
>> Virtual memory mapping done by the OS or Solar? Does the Virtual memory
>> reside on the OS HDD? Or on the Solr HDD?.. and if the Virtual memory
>> mapping is done on the OS HDD, wouldn't it be beneficial to run the OS off
>> a SSD?
>>
>
> ***Deepak***
If you have a small RAM (I am assuming that is what you mean by a small
heap), then OS will do swapping or demand paging to manage your memory
requirements. SSD will help. However it might be better to have a larger
RAM than rely on SSD.
***Deepak***

> There appears to be some confusion here.
>
> The virtual memory doesn't reside on ANY hard drive, unless you've REALLY
> configured the system badly and the system starts using swap space.  If the
> system starts using swap, performance is going to be terrible, no matter
> how fast the disk where swap resides is.
>
> The "mapping to virtual memory" feature is something the operating system
> does.  Lucene/Solr utilizes MMAP code in Java, which then turns around and
> uses MMAP functionality provided by the OS.
>
> At that point, that file can be accessed by the application as if it were
> a very large block of memory.  Mapping the file doesn't immediately use any
> memory at all.  The OS manages the access to the file.  If the part of the
> file that is being accessed has not been accessed before, then the OS will
> read the data off the disk, place it into the OS disk cache, and provide it
> to whatever requested it.  If it has been accessed before and is still in
> the disk cache, then it won't read the disk, it will just provide the data
> from the cache.  Getting most data from cache is *required* for good Solr
> performance.
>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> Running with your indexes on SSD might indeed help performance, and
> regardless of anything that's going on, WILL help performance in the short
> term, when you first turn the machine on.  But if it also helps with
> long-term query performance, then chances are that the machine doesn't have
> enough memory.When Solr servers are sized correctly, running on SSD is
> typically not going to make a big difference, unless the machine does a lot
> more indexing than querying.
>
> For now.. my FEELING is to run one Solr instance on this particular
>> machine.. by the time the RAM is outgrown add another machine and so
>> forth...
>>
>
> Any plans you have for a growth strategy with multiple Solr instances are
> extremely likely to still be possible with only one instance, with very
> little change.
>
> Thanks,
> Shawn







Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
[hidden email]

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Fri, Mar 16, 2018 at 6:03 PM, Shawn Heisey <[hidden email]> wrote:

> On 3/15/2018 6:34 AM, BlackIce wrote:
>
>> However the main app that will be
>> running is more or less a single threated app which takes advantage when
>> run under several instances, ie: parallelism, so I thought, since I'm at
>> it
>> I may give solr a few instances as well
>>
>
> Solr is a fully threaded app, capable of doing LOTS of things at the same
> time, without multiple instances.
>
> Thnx for the Heap pointer.. I've read, from some Professor.. that Solr
>> actually is more efficient with a very small Heap and to have everything
>> mapped to virtual memory... Which brings me to the next question.. is the
>> Virtual memory mapping done by the OS or Solar? Does the Virtual memory
>> reside on the OS HDD? Or on the Solr HDD?.. and if the Virtual memory
>> mapping is done on the OS HDD, wouldn't it be beneficial to run the OS off
>> a SSD?
>>
>
> There appears to be some confusion here.
>
> The virtual memory doesn't reside on ANY hard drive, unless you've REALLY
> configured the system badly and the system starts using swap space.  If the
> system starts using swap, performance is going to be terrible, no matter
> how fast the disk where swap resides is.
>
> The "mapping to virtual memory" feature is something the operating system
> does.  Lucene/Solr utilizes MMAP code in Java, which then turns around and
> uses MMAP functionality provided by the OS.
>
> At that point, that file can be accessed by the application as if it were
> a very large block of memory.  Mapping the file doesn't immediately use any
> memory at all.  The OS manages the access to the file.  If the part of the
> file that is being accessed has not been accessed before, then the OS will
> read the data off the disk, place it into the OS disk cache, and provide it
> to whatever requested it.  If it has been accessed before and is still in
> the disk cache, then it won't read the disk, it will just provide the data
> from the cache.  Getting most data from cache is *required* for good Solr
> performance.
>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> Running with your indexes on SSD might indeed help performance, and
> regardless of anything that's going on, WILL help performance in the short
> term, when you first turn the machine on.  But if it also helps with
> long-term query performance, then chances are that the machine doesn't have
> enough memory.When Solr servers are sized correctly, running on SSD is
> typically not going to make a big difference, unless the machine does a lot
> more indexing than querying.
>
> For now.. my FEELING is to run one Solr instance on this particular
>> machine.. by the time the RAM is outgrown add another machine and so
>> forth...
>>
>
> Any plans you have for a growth strategy with multiple Solr instances are
> extremely likely to still be possible with only one instance, with very
> little change.
>
> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Walter Underwood
In reply to this post by Deepak Goel
> On Mar 16, 2018, at 6:26 AM, Deepak Goel <[hidden email]> wrote:
>
> I would try multiple Solr instances rather a single Solr instance (it
> definitely will give a performance boost)


I would avoid multiple Solr instances on single machine. I can use all 36 cores on our servers with one Solr process.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Walter Underwood
In reply to this post by Deepak Goel
On Mar 16, 2018, at 6:38 AM, Deepak Goel <[hidden email]> wrote:
>
> I did a performance study of Solr a while back. And I found that it does
> not scale beyond a particular point on a single machine (could be due to
> the way its coded). Hence multiple instances might make sense.
>
> https://docs.google.com/document/d/1kUqEcZl3NhOo6SLklo5Icg3fMnn9OtLY_lwnc6wbXus/edit?usp=sharing <https://docs.google.com/document/d/1kUqEcZl3NhOo6SLklo5Icg3fMnn9OtLY_lwnc6wbXus/edit?usp=sharing>
>
> ***Deepak***

That benchmark is on Windows, so not interesting for most of us.

Windows has very different handling for threads, memory, and files compared to Unix. I had to do a lot of Windows-specific tuning for Ultraseek Server to get decent performance. For example, merge speed was terrible unless I opened files with a Windows-specific caching hint.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Deepak Goel
In reply to this post by Walter Underwood
> On Mar 16, 2018, at 6:26 AM, Deepak Goel <[hidden email]> wrote:
>
> I would try multiple Solr instances rather a single Solr instance (it
> definitely will give a performance boost)
> I would avoid multiple Solr instances on single machine. I can use all 36
cores on our servers with one Solr process.

Is your load scaling linearly? Can you please post the results?




Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
[hidden email]

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Fri, Mar 16, 2018 at 9:39 PM, Walter Underwood <[hidden email]>
wrote:

> > On Mar 16, 2018, at 6:26 AM, Deepak Goel <[hidden email]> wrote:
> >
> > I would try multiple Solr instances rather a single Solr instance (it
> > definitely will give a performance boost)
>
>
> I would avoid multiple Solr instances on single machine. I can use all 36
> cores on our servers with one Solr process.
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Deepak Goel
In reply to this post by Walter Underwood
> That benchmark is on Windows, so not interesting for most of us.

I guess I must have missed this in the author's question. Did he describe
his OS?

Also other applications scale well on Windows. Why would Solr be different?
The Solr page does not say about any performance limits on windows
(shouldn't they say that upfront in that case!)

https://lucene.apache.org/solr/guide/6_6/installing-solr.html#got-java
(You can install Solr in any system where a suitable Java Runtime
Environment (JRE) is available, as detailed below. Currently this includes
Linux, OS X, and Microsoft Windows.)

> Windows has very different handling for threads, memory, and files
compared to Unix. I had to do a lot of Windows-specific tuning for > >
Ultraseek Server to get decent performance. For example, merge speed was
terrible unless I opened files with a Windows-specific > > > >caching hint.





Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
[hidden email]

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"
On Fri, Mar 16, 2018 at 9:43 PM, Walter Underwood <[hidden email]>
wrote:

> On Mar 16, 2018, at 6:38 AM, Deepak Goel <[hidden email]> wrote:
> >
> > I did a performance study of Solr a while back. And I found that it does
> > not scale beyond a particular point on a single machine (could be due to
> > the way its coded). Hence multiple instances might make sense.
> >
> > https://docs.google.com/document/d/1kUqEcZl3NhOo6SLklo5Icg3fMnn9O
> tLY_lwnc6wbXus/edit?usp=sharing <https://docs.google.com/document/d/
> 1kUqEcZl3NhOo6SLklo5Icg3fMnn9OtLY_lwnc6wbXus/edit?usp=sharing>
> >
> > ***Deepak***
>
> That benchmark is on Windows, so not interesting for most of us.
>
> Windows has very different handling for threads, memory, and files
> compared to Unix. I had to do a lot of Windows-specific tuning for
> Ultraseek Server to get decent performance. For example, merge speed was
> terrible unless I opened files with a Windows-specific caching hint.
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Shawn Heisey-2
In reply to this post by Deepak Goel
On 3/16/2018 7:38 AM, Deepak Goel wrote:
> I did a performance study of Solr a while back. And I found that it does
> not scale beyond a particular point on a single machine (could be due to
> the way its coded). Hence multiple instances might make sense.
>
> https://docs.google.com/document/d/1kUqEcZl3NhOo6SLklo5Icg3fMnn9OtLY_lwnc6wbXus/edit?usp=sharing

How did you *use* that code that you've shown?  That is not apparent (at
least to me) from the document.

If every usage of the SolrJ code went through ALL of the code you've
shown, then it's not done well.  It appears that you're creating and
closing a client object with every query.  This will be VERY inefficient.

The client object should be created during an initialization step, and
then passed to the benchmark step to be used there.  One client object
can be used by many threads.  Very likely the ES client works the same,
but you'd need to ask them to be sure.

That code seems to be doing an identical query on every run.  If that's
what's happening, it's not a good indicator of performance.  Running the
same query over and over will show better performance than you can
expect from a real-world query load.

What evidence do you see that Solr isn't scaling like you expect?

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Deepak Goel
On Sat, Mar 17, 2018 at 1:06 AM, Shawn Heisey <[hidden email]> wrote:

> On 3/16/2018 7:38 AM, Deepak Goel wrote:
> > I did a performance study of Solr a while back. And I found that it does
> > not scale beyond a particular point on a single machine (could be due to
> > the way its coded). Hence multiple instances might make sense.
> >
> > https://docs.google.com/document/d/1kUqEcZl3NhOo6SLklo5Icg3fMnn9O
> tLY_lwnc6wbXus/edit?usp=sharing
>
> How did you *use* that code that you've shown?  That is not apparent (at
> least to me) from the document.
>
> If every usage of the SolrJ code went through ALL of the code you've
> shown, then it's not done well.  It appears that you're creating and
> closing a client object with every query.  This will be VERY inefficient.
>
> The client object should be created during an initialization step, and
> then passed to the benchmark step to be used there.  One client object
> can be used by many threads.


I wanted to test how many max connections can Solr handle concurrently.
Also I would have to implement an 'connection pooling' of the client-object
connections rather than a single connection thread

However a single client object with thousands of queries coming in would
surely become a bottleneck. I can test this scenario too.

Very likely the ES client works the same,
> but you'd need to ask them to be sure.
>
> That code seems to be doing an identical query on every run.  If that's
> what's happening, it's not a good indicator of performance.  Running the
> same query over and over will show better performance than you can
> expect from a real-world query load

What evidence do you see that Solr isn't scaling like you expect?
>
> The problem is the max throughput which I can get on the machine is around
28 tps, even though I increase the load further & only 65% CPU is utilised
(there is still 35% which is not being used). This clearly indicates the
software is a problem as there is enough hardware resources.

Also very soon I would have a Linux environment with me, so I can conduct
the test in the document on Linux too (for the users interested in Linux
and not Windows)


> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

RE: Some performance questions....

Davis, Daniel (NIH/NLM) [C]
Deepak,

A better test of multi-user support might be to vary the queries and try to simulate a realistic 'working set' of search data.

I've made this same performance analysis mistake with the search index of www.indexengines.com, which I developed (in part).   Somewhat different from Lucene, inside, although.

What we cared a lot about were these things:

- if a query was done warm, e.g. with results cached in memory, response time should be very fast.
- If a query was done cold, e.g. with results from disk, response time should still be acceptable.
- If a lot of different queries were done, that we think simulate the real behavior of N users, that the memory usage of cache should be acceptable, e.g. the cache should get warm and there should be few cache misses.

This last test was key - if we have designed our caching properly, then the queries of X users will fit in Y memory, and we will be able to develop a simple understanding of that, with our target users.

Generating that realistic amount of query behavior for X users is hard.   Using real search logs from your previous search product is a good idea.   For instance, if you look at the top 1000 queries performed by your users over a particular period of time, you can then say that some percentage of user queries were covered by the top 1000 queries, e.g. 90%.   Then, maybe you measure of that same period your queries per second (QPS).

Now, you can say that if you randomly sample those top 1000 queries while generating the same QPS with an exponential distribution generator, that you have covered 90% of your real traffic.   Your queries are much more randomly distributed, but that's OK, because what you want to know is whether it all fits in cache memory, the effect of # of CPUs, amount of Memory, number of cluster nodes, sharding, and replication on the response time and such.

Depending on your user community, top 1000 queries may not be enough to hit 90%, it may only hit 70%.   Maybe you also need to look at the rate of "advanced search" and "search", or account for queries that drive business intelligence reports.   It really depends on your use case.   I wish I'd had the cloud available to test performance with - we were really naïve and did all this testing with our metal because, well, we thought our stuff relied on that.

I recommend you read the first couple chapters of Ran Jain's Art of Computer Systems Performance Analysis.   It’s a great book even if you totally skip the later chapters on Queuing System analysis, and just think about what and how to test.

Hope this helps,

-Dan





-----Original Message-----
From: Deepak Goel [mailto:[hidden email]]
Sent: Friday, March 16, 2018 4:22 PM
To: [hidden email]
Subject: Re: Some performance questions....

On Sat, Mar 17, 2018 at 1:06 AM, Shawn Heisey <[hidden email]> wrote:

> On 3/16/2018 7:38 AM, Deepak Goel wrote:
> > I did a performance study of Solr a while back. And I found that it
> > does not scale beyond a particular point on a single machine (could
> > be due to the way its coded). Hence multiple instances might make sense.
> >
> > https://docs.google.com/document/d/1kUqEcZl3NhOo6SLklo5Icg3fMnn9O
> tLY_lwnc6wbXus/edit?usp=sharing
>
> How did you *use* that code that you've shown?  That is not apparent
> (at least to me) from the document.
>
> If every usage of the SolrJ code went through ALL of the code you've
> shown, then it's not done well.  It appears that you're creating and
> closing a client object with every query.  This will be VERY inefficient.
>
> The client object should be created during an initialization step, and
> then passed to the benchmark step to be used there.  One client object
> can be used by many threads.


I wanted to test how many max connections can Solr handle concurrently.
Also I would have to implement an 'connection pooling' of the client-object connections rather than a single connection thread

However a single client object with thousands of queries coming in would surely become a bottleneck. I can test this scenario too.

Very likely the ES client works the same,
> but you'd need to ask them to be sure.
>
> That code seems to be doing an identical query on every run.  If
> that's what's happening, it's not a good indicator of performance.  
> Running the same query over and over will show better performance than
> you can expect from a real-world query load

What evidence do you see that Solr isn't scaling like you expect?
>
> The problem is the max throughput which I can get on the machine is
> around
28 tps, even though I increase the load further & only 65% CPU is utilised (there is still 35% which is not being used). This clearly indicates the software is a problem as there is enough hardware resources.

Also very soon I would have a Linux environment with me, so I can conduct the test in the document on Linux too (for the users interested in Linux and not Windows)


> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Shawn Heisey-2
In reply to this post by Deepak Goel
On 3/16/2018 2:21 PM, Deepak Goel wrote:
> I wanted to test how many max connections can Solr handle concurrently.
> Also I would have to implement an 'connection pooling' of the client-object
> connections rather than a single connection thread
>
> However a single client object with thousands of queries coming in would
> surely become a bottleneck. I can test this scenario too.

Handling thousands of simultaneous queries is NOT something you can
expect a single Solr server to do.  It's not going to happen.  It
wouldn't happen with ES, either.  Handling that much load requires load
balancing to a LOT of servers.  The server would much more of a
bottleneck than the client.

> The problem is the max throughput which I can get on the machine is around
> 28 tps, even though I increase the load further & only 65% CPU is utilised
> (there is still 35% which is not being used). This clearly indicates the
> software is a problem as there is enough hardware resources.

If your code is creating a client object before every single query, that
could be part of the issue.  The benchmark code should be using the same
client for all requests.  I really don't know how long it takes to
create HttpSolrClient objects, but I don't imagine that it's super-fast.

What version of SolrJ were you using?

Depending on the SolrJ version you may need to create the client with a
custom HttpClient object in order to allow it to handle plenty of
threads.  This is how I create client objects in my SolrJ code:

  RequestConfig rc = RequestConfig.custom().setConnectTimeout(2000)
    .setSocketTimeout(60000).build();
  CloseableHttpClient httpClient =
HttpClients.custom().setDefaultRequestConfig(rc).setMaxConnPerRoute(1024)
    .setMaxConnTotal(4096).disableAutomaticRetries().build();

  SolrClient sc = new HttpSolrClient.Builder().withBaseSolrUrl(solrUrl)
    .withHttpClient(httpClient).build();

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Walter Underwood
In reply to this post by Deepak Goel
> On Mar 16, 2018, at 1:21 PM, Deepak Goel <[hidden email]> wrote:
>
> However a single client object with thousands of queries coming in would
> surely become a bottleneck. I can test this scenario too.

No it isn’t. The single client object is thread-safe and manages a pool of connections.

Your benchmark is probably the bottleneck. I have no problem driving 36 CPUs to beyond
65% utilization with a benchmark.

Using one client object is not a scenario. It is how SolrJ was designed to be used.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Deepak Goel
In reply to this post by Shawn Heisey-2
On Sat, Mar 17, 2018 at 2:56 AM, Shawn Heisey <[hidden email]> wrote:

> On 3/16/2018 2:21 PM, Deepak Goel wrote:
> > I wanted to test how many max connections can Solr handle concurrently.
> > Also I would have to implement an 'connection pooling' of the
> client-object
> > connections rather than a single connection thread
> >
> > However a single client object with thousands of queries coming in would
> > surely become a bottleneck. I can test this scenario too.
>
> Handling thousands of simultaneous queries is NOT something you can
> expect a single Solr server to do.  It's not going to happen.  It
> wouldn't happen with ES, either.  Handling that much load requires load
> balancing to a LOT of servers.  The server would much more of a
> bottleneck than the client.
>

The problem is not server in my case. The server has hardware resources.
It's the software which is a problem.


>
> > The problem is the max throughput which I can get on the machine is
> around
> > 28 tps, even though I increase the load further & only 65% CPU is
> utilised
> > (there is still 35% which is not being used). This clearly indicates the
> > software is a problem as there is enough hardware resources.
>
> If your code is creating a client object before every single query, that
> could be part of the issue.  The benchmark code should be using the same
> client for all requests.  I really don't know how long it takes to
> create HttpSolrClient objects, but I don't imagine that it's super-fast.
>
>
It is taking less than 100ms to create a HttpSolrClient Object


> What version of SolrJ were you using?
>

Solr 7.2.0


> Depending on the SolrJ version you may need to create the client with a
> custom HttpClient object in order to allow it to handle plenty of
> threads.  This is how I create client objects in my SolrJ code:
>
>   RequestConfig rc = RequestConfig.custom().setConnectTimeout(2000)
>     .setSocketTimeout(60000).build();
>   CloseableHttpClient httpClient =
> HttpClients.custom().setDefaultRequestConfig(rc).setMaxConnPerRoute(1024)
>     .setMaxConnTotal(4096).disableAutomaticRetries().build();
>
>   SolrClient sc = new HttpSolrClient.Builder().withBaseSolrUrl(solrUrl)
>     .withHttpClient(httpClient).build();
>
>
I can give the above configuration a spin and test if the results improve


> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Deepak Goel
In reply to this post by Walter Underwood
On Sat, Mar 17, 2018 at 3:11 AM, Walter Underwood <[hidden email]>
wrote:

> > On Mar 16, 2018, at 1:21 PM, Deepak Goel <[hidden email]> wrote:
> >
> > However a single client object with thousands of queries coming in would
> > surely become a bottleneck. I can test this scenario too.
>
> No it isn’t. The single client object is thread-safe and manages a pool of
> connections.
>
> Your benchmark is probably the bottleneck. I have no problem driving 36
> CPUs to beyond
> 65% utilization with a benchmark.
>
>
Can you please post results of your test?

Please tell us the tps at 25%, 50%, 75%, 100% of your CPU resource


> Using one client object is not a scenario. It is how SolrJ was designed to
> be used.
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Walter Underwood
> On Mar 16, 2018, at 3:26 PM, Deepak Goel <[hidden email]> wrote:
>
> Can you please post results of your test?
>
> Please tell us the tps at 25%, 50%, 75%, 100% of your CPU resource


I could, but it probably would not be useful for your documents or your queries.

We have 22 million homework problems. Our queries are often hundreds of words long,
because students copy and paste entire problems. After pre-processing, the average query
is still 25 words.

For load benchmarking, I use access logs from production. I typically gather over a half-million
lines of log. Using production logs means that queries have the same statistical distribution
as prod, so the cache hit rates are reasonable.

Before each benchmark, I restart all the Solr instances to clear the caches. Then the first part
of the query log is used to warm the caches, typically about 4000 queries.

After that, the measured benchmark run starts. This uses JMeter with 100-500 threads. Each
thread is configured with a constant throughput timer so a constant load is offered. Test run
one or two hours. Recently, I ran a test with a rate of 1000 requests/minute for one hour.

During the benchmark, I monitor the CPU usage. Our systems are configured with enough RAM
so that disk is not accessed for search indexes. If the CPU goes over 75-80%, there is congestion
and queries will slow down. Also, if the run queue (load average) increases over the number of
CPUs, there will be congestion.

After the benchmark run, the JMeter log is analyzed to report response time percentiles for
each Solr request handler.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

Deepak Goel
On 17 Mar 2018 05:19, "Walter Underwood" <[hidden email]> wrote:

> On Mar 16, 2018, at 3:26 PM, Deepak Goel <[hidden email]> wrote:
>
> Can you please post results of your test?
>
> Please tell us the tps at 25%, 50%, 75%, 100% of your CPU resource


I could, but it probably would not be useful for your documents or your
queries.

We have 22 million homework problems. Our queries are often hundreds of
words long,
because students copy and paste entire problems. After pre-processing, the
average query
is still 25 words.

For load benchmarking, I use access logs from production. I typically
gather over a half-million
lines of log. Using production logs means that queries have the same
statistical distribution
as prod, so the cache hit rates are reasonable.

Before each benchmark, I restart all the Solr instances to clear the
caches. Then the first part
of the query log is used to warm the caches, typically about 4000 queries.

After that, the measured benchmark run starts. This uses JMeter with
100-500 threads. Each
thread is configured with a constant throughput timer so a constant load is
offered. Test run
one or two hours. Recently, I ran a test with a rate of 1000
requests/minute for one hour.

During the benchmark, I monitor the CPU usage. Our systems are configured
with enough RAM
so that disk is not accessed for search indexes. If the CPU goes over
75-80%, there is congestion
and queries will slow down. Also, if the run queue (load average) increases
over the number of
CPUs, there will be congestion.

After the benchmark run, the JMeter log is analyzed to report response time
percentiles for
each Solr request handler.


Sorry for being rude. But the ' results ' please, not the ' road to the
results '


wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)
Reply | Threaded
Open this post in threaded view
|

Re: Some performance questions....

BlackIce
Looks like I've opened up a very interesting can of worms....

Thank you to all that are posting to this thread, I'm learning a lot...
 The way I see it now... a Single Solr instance on this machine, seems like
the most intelligent choice.
And then as upgrade path, adding in-expensive machines. This adds me
storage space, cpu power and starts to build up on the parallezation
cluster and load balancing.

Greetz

On Sat, Mar 17, 2018 at 11:23 AM, Deepak Goel <[hidden email]> wrote:

> On 17 Mar 2018 05:19, "Walter Underwood" <[hidden email]> wrote:
>
> > On Mar 16, 2018, at 3:26 PM, Deepak Goel <[hidden email]> wrote:
> >
> > Can you please post results of your test?
> >
> > Please tell us the tps at 25%, 50%, 75%, 100% of your CPU resource
>
>
> I could, but it probably would not be useful for your documents or your
> queries.
>
> We have 22 million homework problems. Our queries are often hundreds of
> words long,
> because students copy and paste entire problems. After pre-processing, the
> average query
> is still 25 words.
>
> For load benchmarking, I use access logs from production. I typically
> gather over a half-million
> lines of log. Using production logs means that queries have the same
> statistical distribution
> as prod, so the cache hit rates are reasonable.
>
> Before each benchmark, I restart all the Solr instances to clear the
> caches. Then the first part
> of the query log is used to warm the caches, typically about 4000 queries.
>
> After that, the measured benchmark run starts. This uses JMeter with
> 100-500 threads. Each
> thread is configured with a constant throughput timer so a constant load is
> offered. Test run
> one or two hours. Recently, I ran a test with a rate of 1000
> requests/minute for one hour.
>
> During the benchmark, I monitor the CPU usage. Our systems are configured
> with enough RAM
> so that disk is not accessed for search indexes. If the CPU goes over
> 75-80%, there is congestion
> and queries will slow down. Also, if the run queue (load average) increases
> over the number of
> CPUs, there will be congestion.
>
> After the benchmark run, the JMeter log is analyzed to report response time
> percentiles for
> each Solr request handler.
>
>
> Sorry for being rude. But the ' results ' please, not the ' road to the
> results '
>
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
1234