[jira] Created: (LUCY-75) CompoundFileWriter

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCY-75) CompoundFileWriter

Sebastian Nagel (Jira)
CompoundFileWriter
------------------

                 Key: LUCY-75
                 URL: https://issues.apache.org/jira/browse/LUCY-75
             Project: Lucy
          Issue Type: Sub-task
          Components: Core
            Reporter: Marvin Humphrey
            Assignee: Marvin Humphrey


Consolidate files within a subdirectory to save file descriptors.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCY-75) CompoundFileWriter

Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/LUCY-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marvin Humphrey updated LUCY-75:
--------------------------------

    Attachment: cf_writer_support.diff
                CompoundFileWriter.c
                CompoundFileWriter.bp

CompoundFileWriter consolidates files within a directory into a combinded
"cf.dat" file, storing their file metadata within a "cfmeta.json" file.
Nested subdirectories and files ending in ".json" are excluded.  

The ".json" exclusion policy is necessary because we want our "segmeta.json"
files to be visible when browsing the index; it causes no problems with file
descriptor limits because JSON files tend to be slurped rather than held open.

Each sub-file within the cf.dat file begins on an offset which is a multiple
of 8, allowing memory-mapped regions to be cast to any type of pointer without
memory alignment problems.

The Lucene version of CompoundFileWriter lives underneath
org.apache.lucene.index and operates on a logical index segment; in contrast,
this version lives under Lucy::Store and operates on any subdirectory,
regardless of what files the subdirectory contains.


> CompoundFileWriter
> ------------------
>
>                 Key: LUCY-75
>                 URL: https://issues.apache.org/jira/browse/LUCY-75
>             Project: Lucy
>          Issue Type: Sub-task
>          Components: Core
>            Reporter: Marvin Humphrey
>            Assignee: Marvin Humphrey
>         Attachments: cf_writer_support.diff, CompoundFileWriter.bp, CompoundFileWriter.c
>
>
> Consolidate files within a subdirectory to save file descriptors.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCY-75) CompoundFileWriter

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/LUCY-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marvin Humphrey updated LUCY-75:
--------------------------------

    Attachment: TestCompoundFileWriter.c
                TestCompoundFileWriter.bp
                112-cf_writer.t

> CompoundFileWriter
> ------------------
>
>                 Key: LUCY-75
>                 URL: https://issues.apache.org/jira/browse/LUCY-75
>             Project: Lucy
>          Issue Type: Sub-task
>          Components: Core
>            Reporter: Marvin Humphrey
>            Assignee: Marvin Humphrey
>         Attachments: 112-cf_writer.t, cf_writer_support.diff, CompoundFileWriter.bp, CompoundFileWriter.c, TestCompoundFileWriter.bp, TestCompoundFileWriter.c
>
>
> Consolidate files within a subdirectory to save file descriptors.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCY-75) CompoundFileWriter

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/LUCY-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marvin Humphrey resolved LUCY-75.
---------------------------------

    Resolution: Fixed

Committed as r884884.

> CompoundFileWriter
> ------------------
>
>                 Key: LUCY-75
>                 URL: https://issues.apache.org/jira/browse/LUCY-75
>             Project: Lucy
>          Issue Type: Sub-task
>          Components: Core
>            Reporter: Marvin Humphrey
>            Assignee: Marvin Humphrey
>         Attachments: 112-cf_writer.t, cf_writer_support.diff, CompoundFileWriter.bp, CompoundFileWriter.c, TestCompoundFileWriter.bp, TestCompoundFileWriter.c
>
>
> Consolidate files within a subdirectory to save file descriptors.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.