Quantcast

What is largest reasonable setting for ramBufferSizeMB?

12 messages Options
What is largest reasonable setting for ramBufferSizeMB? – Hello all, At some point we will need to re-build an index that totals about 2 terrabytes in size (split over 10 shards). At our current index...
Burton-West, Tom wrote: > Hello all, > > At some point we will need to re-build an index that totals about 2 terrabytes in size (split...
Hi Tom, 32MB is very low, 320MB is medium, and I think you could go higher, just pick whichever garbage collector is good for throughput. I kn...
Thanks Otis, I don't know enough about Hadoop to understand the advantage of using Hadoop in this use case. How would using Hadoop differ from...
On Thu, Feb 18, 2010 at 8:52 AM, Otis Gospodnetic <otis_gospodnetic@...> wrote: > 32MB is very low, 320MB is medium, and I think you co...
Hi Tom, It wouldn't. I didn't see the mention of parallel indexing in the original email. :) Otis ---- Sematext :: http://sematext.com/ :...
I've run Lucene with heap sizes as large as 28GB of RAM (on a 32GB machine, 64bit, Linux) and a ramBufferSize of 3GB. While I haven't noticed th...
On Fri, Feb 19, 2010 at 5:03 AM, Glen Newton <glen.newton@...> wrote: > You may consider using LuSql[1] to create the indexes, if your s...
Hi Glen, I'd love to use LuSql, but our data is not in a db. Its 6-8TB of files containing OCR (one file per page for about 1.5 billion pages)...
Glen may be referring to LuSql indexing with multiple threads? Does/can DIH do that, too? Otis ---- Sematext :: http://sematext.com/ :: Sol...
Looks like multi-threaded support was added to the DIH recently: http://issues.apache.org/jira/browse/SOLR-1352 -Jay On Fri, Feb 19, 2010...
>>and a ramBufferSize of 3GB If you had actually used great than 2GB of it, you would have seen problems as an int overflowed - which...
Loading...