RamDirectory vs MemoryIndex vs MMapDirectory for In-Memory-Index

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

RamDirectory vs MemoryIndex vs MMapDirectory for In-Memory-Index

Matthias Müller-2
Hi,

Lucene provides different storage options for in-memory indexes. I
found three structures that would qualify for the task:

* RamDirectory (which I currently use for prototyping, but wonder if it
is the ideal choice for my task)
* MemoryIndex, which claims to have better performance and resource use
for small documents
* MMapDirectory which should outperform RamDirectory for huge indices
(what is "huge?")


My plan is to periodically index some properties (string codes, longs,
lat/lng points) of a larger database content with Lucene for quicker
lookups (compared to slow SQL queries).

What would be the most efficient (or intended) storage option for such
an index in terms of lookup speed and CPU/memory use? Below [1] is a
brief summary of the index contents and I hope these figures are
sufficient to get a recommendation. But I am also happy to study more
detailed documentation on the matter.

- Matthias

[1]: Summary of index contents and intended use
* Total documents: 500.000 - 1.000.000, may grow to 10.000.000 records
in  mid future.
* Document fields (all of them single value fields):
    * String (9x), usually 1-10 characters long, mostly recurring
values (5% distinct)
    * LongPoint (4x), two fields contain mostly distinct values, one
lostly recurring values (5-10% distinct), one field acts as a primary
key
    * LatLonPoint (1x), 30% distinct
* Refresh interval: 1..5 minutes (I currently create a fresh index
instance on each update and discard the old one)
* Most queries are range queries and exact matches on several
properties, sometimes I need to retrieve the property fields of a
single document based on a primary key value.

Reply | Threaded
Open this post in threaded view
|

Re: RamDirectory vs MemoryIndex vs MMapDirectory for In-Memory-Index

Dawid Weiss-2
Use MMapDirectory on a temporary location, Matthias. If you really
need in-memory indexes, a new Directory implementation is coming
(RAMDirectory will be deprecated, then removed), but the difference
compared to MMapDirectory is typically not worth the hassle. See this
issue for more discussion.

https://issues.apache.org/jira/browse/LUCENE-8438

Dawid
On Tue, Sep 25, 2018 at 10:44 AM Matthias Müller
<[hidden email]> wrote:

>
> Hi,
>
> Lucene provides different storage options for in-memory indexes. I
> found three structures that would qualify for the task:
>
> * RamDirectory (which I currently use for prototyping, but wonder if it
> is the ideal choice for my task)
> * MemoryIndex, which claims to have better performance and resource use
> for small documents
> * MMapDirectory which should outperform RamDirectory for huge indices
> (what is "huge?")
>
>
> My plan is to periodically index some properties (string codes, longs,
> lat/lng points) of a larger database content with Lucene for quicker
> lookups (compared to slow SQL queries).
>
> What would be the most efficient (or intended) storage option for such
> an index in terms of lookup speed and CPU/memory use? Below [1] is a
> brief summary of the index contents and I hope these figures are
> sufficient to get a recommendation. But I am also happy to study more
> detailed documentation on the matter.
>
> - Matthias
>
> [1]: Summary of index contents and intended use
> * Total documents: 500.000 - 1.000.000, may grow to 10.000.000 records
> in  mid future.
> * Document fields (all of them single value fields):
>     * String (9x), usually 1-10 characters long, mostly recurring
> values (5% distinct)
>     * LongPoint (4x), two fields contain mostly distinct values, one
> lostly recurring values (5-10% distinct), one field acts as a primary
> key
>     * LatLonPoint (1x), 30% distinct
> * Refresh interval: 1..5 minutes (I currently create a fresh index
> instance on each update and discard the old one)
> * Most queries are range queries and exact matches on several
> properties, sometimes I need to retrieve the property fields of a
> single document based on a primary key value.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RamDirectory vs MemoryIndex vs MMapDirectory for In-Memory-Index

Matthias Müller-2
Thanks Dawid, glad I asked!

Am Dienstag, den 25.09.2018, 10:46 +0200 schrieb Dawid Weiss:

> Use MMapDirectory on a temporary location, Matthias. If you really
> need in-memory indexes, a new Directory implementation is coming
> (RAMDirectory will be deprecated, then removed), but the difference
> compared to MMapDirectory is typically not worth the hassle. See this
> issue for more discussion.
>
> https://issues.apache.org/jira/browse/LUCENE-8438
>
> Dawid
> On Tue, Sep 25, 2018 at 10:44 AM Matthias Müller
> <[hidden email]> wrote:
> >
> > Hi,
> >
> > Lucene provides different storage options for in-memory indexes. I
> > found three structures that would qualify for the task:
> >
> > * RamDirectory (which I currently use for prototyping, but wonder
> > if it
> > is the ideal choice for my task)
> > * MemoryIndex, which claims to have better performance and resource
> > use
> > for small documents
> > * MMapDirectory which should outperform RamDirectory for huge
> > indices
> > (what is "huge?")
> >
> >
> > My plan is to periodically index some properties (string codes,
> > longs,
> > lat/lng points) of a larger database content with Lucene for
> > quicker
> > lookups (compared to slow SQL queries).
> >
> > What would be the most efficient (or intended) storage option for
> > such
> > an index in terms of lookup speed and CPU/memory use? Below [1] is
> > a
> > brief summary of the index contents and I hope these figures are
> > sufficient to get a recommendation. But I am also happy to study
> > more
> > detailed documentation on the matter.
> >
> > - Matthias
> >
> > [1]: Summary of index contents and intended use
> > * Total documents: 500.000 - 1.000.000, may grow to 10.000.000
> > records
> > in  mid future.
> > * Document fields (all of them single value fields):
> >     * String (9x), usually 1-10 characters long, mostly recurring
> > values (5% distinct)
> >     * LongPoint (4x), two fields contain mostly distinct values,
> > one
> > lostly recurring values (5-10% distinct), one field acts as a
> > primary
> > key
> >     * LatLonPoint (1x), 30% distinct
> > * Refresh interval: 1..5 minutes (I currently create a fresh index
> > instance on each update and discard the old one)
> > * Most queries are range queries and exact matches on several
> > properties, sometimes I need to retrieve the property fields of a
> > single document based on a primary key value.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]