Lucene › Solr › Solr - User

What is largest reasonable setting for ramBufferSizeMB?

Classic

List

Threaded

12 messages Options

Tom Burton-West-2 What is largest reasonable setting for ramBufferSizeMB? – Hello all, At some point we will need to re-build an index that totals about 2 terrabytes in size (split over 10 shards). At our current index...

Mark Miller-3 Burton-West, Tom wrote: > Hello all, > > At some point we will need to re-build an index that totals about 2 terrabytes in size (split...

Otis Gospodnetic-2 Hi Tom, 32MB is very low, 320MB is medium, and I think you could go higher, just pick whichever garbage collector is good for throughput. I kn...

Tom Burton-West Thanks Otis, I don't know enough about Hadoop to understand the advantage of using Hadoop in this use case. How would using Hadoop differ from...

Yonik Seeley-2-2 On Thu, Feb 18, 2010 at 8:52 AM, Otis Gospodnetic <otis_gospodnetic@...> wrote: > 32MB is very low, 320MB is medium, and I think you co...

Otis Gospodnetic-2 Hi Tom, It wouldn't. I didn't see the mention of parallel indexing in the original email. :) Otis ---- Sematext :: http://sematext.com/ :...

Glen Newton I've run Lucene with heap sizes as large as 28GB of RAM (on a 32GB machine, 64bit, Linux) and a ramBufferSize of 3GB. While I haven't noticed th...

Yonik Seeley-2-2 On Fri, Feb 19, 2010 at 5:03 AM, Glen Newton <glen.newton@...> wrote: > You may consider using LuSql[1] to create the indexes, if your s...

Tom Burton-West Hi Glen, I'd love to use LuSql, but our data is not in a db. Its 6-8TB of files containing OCR (one file per page for about 1.5 billion pages)...

Otis Gospodnetic-2 Glen may be referring to LuSql indexing with multiple threads? Does/can DIH do that, too? Otis ---- Sematext :: http://sematext.com/ :: Sol...

Jay Hill Looks like multi-threaded support was added to the DIH recently: http://issues.apache.org/jira/browse/SOLR-1352 -Jay On Fri, Feb 19, 2010...

Mark Miller-3 >>and a ramBufferSize of 3GB If you had actually used great than 2GB of it, you would have seen problems as an int overflowed - which...