RAMDirectory or Redis

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

RAMDirectory or Redis

Joe MA
Greetings,

Has anyone looked into using Redis or some other in-memory cache with Lucene?
It seems that ElasticSearch may do this.  Are there advantages to doing this versus, say, the RAMDirectory class?

Thanks in advance,
J


 



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RAMDirectory or Redis

Adrien Grand
Hi Joe,

Elasticsearch used to have an option to run with RAMDirectory but it
was removed long ago:
https://github.com/elastic/elasticsearch/pull/8536.

The recommended approach is to use MMapDirectory and to give
sufficient memory to the filesystem cache so that the most important
data-structures from the index fit in memory. For instance the
Elasticsearch docs recommend giving at most 50% of the system memory
to the JVM and I think Uwe goes even further and recommends giving at
most one third of the system memory to the JVM.
On Sun, Dec 2, 2018 at 10:22 AM Joe MA <[hidden email]> wrote:

>
> Greetings,
>
> Has anyone looked into using Redis or some other in-memory cache with Lucene?
> It seems that ElasticSearch may do this.  Are there advantages to doing this versus, say, the RAMDirectory class?
>
> Thanks in advance,
> J
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


--
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RAMDirectory or Redis

Arjen van der Meijden
In reply to this post by Joe MA
I doubt using Redis as directory-storage will be very good. I'd expect
it to have much more latency for reads and writes compared to any of
lucene's own directories. And Lucene probably won't like it if another
Lucene-instance changes that database.

It may be interesting as a result-level cache though, if you have a
relatively large amount of repeated queries.

With regards to RAMDirectory (and its successor the
ByteBuffersDirectory), it works well enough for our setup: we run an
in-memory copy of our regular sql database in an elastic search like
setup to support very fast responses with a form of faceted search which
would require relatively complex sql-statements (and many repeated
ones). Lucene is only used for textual search in that system.

We didn't want any trouble trying to sync whatever is on disk with the
database and/or having to purchase relatively expensive storage in the
servers. The total application uses about 6GB of RAM, of which somewhere
between 1/2 and 1/3 is for the Lucene directories (about 20-30).
We run an independent copy of that application on each of our 4
applications servers, so we can afford a relatively long startup time of
the application. In all it only takes about 4-5 minutes to load all data
and build in-memory lucene databases.

Our largest Lucene directory is about 400MB, and RAMDirectory worked
pretty good for this. We switched to ByteBuffersDirectory with 7.5, but
I actually didn't see much performance improvements or savings in memory.

If you do care about the rebuild time of your search engine, than just
using Lucene's recommended database and not trying to add additional
complexity with those in-memory databases is probably a good idea :)

Best regards,

Arjen

On 2-12-2018 10:22, Joe MA wrote:

> Greetings,
>
> Has anyone looked into using Redis or some other in-memory cache with Lucene?
> It seems that ElasticSearch may do this.  Are there advantages to doing this versus, say, the RAMDirectory class?
>
> Thanks in advance,
> J
>
>
>  
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RAMDirectory or Redis

Dawid Weiss-2
bq. We switched to ByteBuffersDirectory with 7.5, but
I actually didn't see much performance improvements or savings in memory.

Once the indexes are built I don't think there will be much of a
difference. The core problem with RAMDirectory was related to
synchronizations during merges/ file manipulations.

D.
On Sun, Dec 2, 2018 at 1:18 PM Arjen van der Meijden
<[hidden email]> wrote:

>
> I doubt using Redis as directory-storage will be very good. I'd expect
> it to have much more latency for reads and writes compared to any of
> lucene's own directories. And Lucene probably won't like it if another
> Lucene-instance changes that database.
>
> It may be interesting as a result-level cache though, if you have a
> relatively large amount of repeated queries.
>
> With regards to RAMDirectory (and its successor the
> ByteBuffersDirectory), it works well enough for our setup: we run an
> in-memory copy of our regular sql database in an elastic search like
> setup to support very fast responses with a form of faceted search which
> would require relatively complex sql-statements (and many repeated
> ones). Lucene is only used for textual search in that system.
>
> We didn't want any trouble trying to sync whatever is on disk with the
> database and/or having to purchase relatively expensive storage in the
> servers. The total application uses about 6GB of RAM, of which somewhere
> between 1/2 and 1/3 is for the Lucene directories (about 20-30).
> We run an independent copy of that application on each of our 4
> applications servers, so we can afford a relatively long startup time of
> the application. In all it only takes about 4-5 minutes to load all data
> and build in-memory lucene databases.
>
> Our largest Lucene directory is about 400MB, and RAMDirectory worked
> pretty good for this. We switched to ByteBuffersDirectory with 7.5, but
> I actually didn't see much performance improvements or savings in memory.
>
> If you do care about the rebuild time of your search engine, than just
> using Lucene's recommended database and not trying to add additional
> complexity with those in-memory databases is probably a good idea :)
>
> Best regards,
>
> Arjen
>
> On 2-12-2018 10:22, Joe MA wrote:
> > Greetings,
> >
> > Has anyone looked into using Redis or some other in-memory cache with Lucene?
> > It seems that ElasticSearch may do this.  Are there advantages to doing this versus, say, the RAMDirectory class?
> >
> > Thanks in advance,
> > J
> >
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]