[jira] Created: (LUCENE-2500) A Linux-specific Directory impl that bypasses the buffer cache

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-2500) A Linux-specific Directory impl that bypasses the buffer cache

JIRA jira@apache.org
A Linux-specific Directory impl that bypasses the buffer cache
--------------------------------------------------------------

                 Key: LUCENE-2500
                 URL: https://issues.apache.org/jira/browse/LUCENE-2500
             Project: Lucene - Java
          Issue Type: Improvement
          Components: contrib/*
            Reporter: Michael McCandless
            Assignee: Michael McCandless


I've been testing how we could prevent Lucene's merges from evicting
pages from the OS's buffer cache.  I tried fadvise/madvise (via JNI)
but (frustratingly), I could not get them to work (details at
http://chbits.blogspot.com/2010/06/lucene-and-fadvisemadvise.html).

The only thing that worked was to use Linux's O_DIRECT flag, which
forces all IO to bypass the buffer cache entirely... so I created a
Linux-specific Directory impl to do this.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2500) A Linux-specific Directory impl that bypasses the buffer cache

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-2500:
---------------------------------------

    Attachment: LUCENE-2500.patch

Attached patch, with new Directory impl DirectIOLinuxDirectory.

> A Linux-specific Directory impl that bypasses the buffer cache
> --------------------------------------------------------------
>
>                 Key: LUCENE-2500
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2500
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: contrib/*
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>         Attachments: LUCENE-2500.patch
>
>
> I've been testing how we could prevent Lucene's merges from evicting
> pages from the OS's buffer cache.  I tried fadvise/madvise (via JNI)
> but (frustratingly), I could not get them to work (details at
> http://chbits.blogspot.com/2010/06/lucene-and-fadvisemadvise.html).
> The only thing that worked was to use Linux's O_DIRECT flag, which
> forces all IO to bypass the buffer cache entirely... so I created a
> Linux-specific Directory impl to do this.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCENE-2500) A Linux-specific Directory impl that bypasses the buffer cache

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-2500.
----------------------------------------

    Fix Version/s: 3.1
                   4.0
       Resolution: Fixed

> A Linux-specific Directory impl that bypasses the buffer cache
> --------------------------------------------------------------
>
>                 Key: LUCENE-2500
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2500
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: contrib/*
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2500.patch
>
>
> I've been testing how we could prevent Lucene's merges from evicting
> pages from the OS's buffer cache.  I tried fadvise/madvise (via JNI)
> but (frustratingly), I could not get them to work (details at
> http://chbits.blogspot.com/2010/06/lucene-and-fadvisemadvise.html).
> The only thing that worked was to use Linux's O_DIRECT flag, which
> forces all IO to bypass the buffer cache entirely... so I created a
> Linux-specific Directory impl to do this.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2500) A Linux-specific Directory impl that bypasses the buffer cache

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882152#action_12882152 ]

Michael McCandless commented on LUCENE-2500:
--------------------------------------------

Uwe found this nice ant plugin: http://ant-contrib.sourceforge.net/cpptasks/index.html

It would allow us to [optionally] build the dynamic lib (JNI extension) from ant.

But I'm not going to have any time any time soon to try it... and I'm not good with The Ant.

> A Linux-specific Directory impl that bypasses the buffer cache
> --------------------------------------------------------------
>
>                 Key: LUCENE-2500
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2500
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: contrib/*
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2500.patch
>
>
> I've been testing how we could prevent Lucene's merges from evicting
> pages from the OS's buffer cache.  I tried fadvise/madvise (via JNI)
> but (frustratingly), I could not get them to work (details at
> http://chbits.blogspot.com/2010/06/lucene-and-fadvisemadvise.html).
> The only thing that worked was to use Linux's O_DIRECT flag, which
> forces all IO to bypass the buffer cache entirely... so I created a
> Linux-specific Directory impl to do this.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]