[jira] [Commented] (LUCENE-8267) Remove memory codecs from the codebase

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] [Commented] (LUCENE-8267) Remove memory codecs from the codebase

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-8267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448272#comment-16448272 ]

Robert Muir commented on LUCENE-8267:

There are a lot of other alternatives to putting data in heap memory directly in the postings format.

The best (IMO) is for the user to use MMapDirectory.preload with the standard index format. This way it doesn't impact their java heap and they use supported index format. Users can also use RAMDirectory/FileSwitchDirectory to load specified files into heap.

Finally, users can use FSTPostingsFormat which will load *term dictionary only* into heap fst. This is way different than Memory/Direct which load not only terms but also postings lists and positions and stuff all into heap RAM.

So i don't really see any technical merit for your objection: there are many other ways to have a ram-resident terms dictionary, many of them better than the inefficient Memory/Direct formats.

> Remove memory codecs from the codebase
> --------------------------------------
>                 Key: LUCENE-8267
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8267
>             Project: Lucene - Core
>          Issue Type: Task
>            Reporter: Dawid Weiss
>            Priority: Major
> Memory codecs (MemoryPostings*, MemoryDocValues*) are part of random selection of codecs for tests and cause occasional OOMs when a test with huge data is selected. We don't use those memory codecs anywhere outside of tests, it has been suggested to just remove them to avoid maintenance costs and OOMs in tests. [1]
> [1] https://apache.markmail.org/thread/mj53os2ekyldsoy3

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]