[jira] Created: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
-----------------------------------------------------------------------------------

                 Key: LUCENE-2047
                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Index
            Reporter: Michael McCandless
            Assignee: Michael McCandless
            Priority: Minor
             Fix For: 3.0


Spinoff from LUCENE-1526.

When deleteDocuments(Term) is called, we currently always buffer the
Term and only later, when it's time to flush deletes, resolve to
docIDs.  This is necessary because we don't in general hold
SegmentReaders open.

But, when IndexWriter is in NRT mode, we pool the readers, and so
deleting in the foreground is possible.

It's also beneficial, in that in can reduce the turnaround time when
reopening a new NRT reader by taking this resolution off the reopen
path.  And if multiple threads are used to do the deletion, then we
gain concurrency, vs reopen which is not concurrent when flushing the
deletes.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775115#action_12775115 ]

Jason Rutherglen commented on LUCENE-2047:
------------------------------------------

Is this going to be an option or default to true only when NRT is on?

Also, I can create a patch for this.

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.0
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775121#action_12775121 ]

Michael McCandless commented on LUCENE-2047:
--------------------------------------------

I think simply default to true.

Yes, please cons up a patch!

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.0
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-2047:
-------------------------------------

    Attachment: LUCENE-2047.patch

Deletes occur immediately if poolReader is true

I'm not sure updateDocument needs to delete immediately, as it's also writing a document, the deletes later would be lost in the noise.

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.0
>
>         Attachments: LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-2047:
---------------------------------------

    Fix Version/s:     (was: 3.0)
                   3.1

Pushing to 3.1...

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776203#action_12776203 ]

Michael McCandless commented on LUCENE-2047:
--------------------------------------------

Thanks Jason!

I would think we do want to delete live for updateDocument as well?  It's not clear the deletion will be in the noise (if it hits a disk seek, it won't).

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776674#action_12776674 ]

Jason Rutherglen commented on LUCENE-2047:
------------------------------------------

{quote}I would think we do want to delete live for
updateDocument as well? It's not clear the deletion will be in
the noise (if it hits a disk seek, it won't).{quote}

The delete shouldn't hit a disk seek because with poolReaders,
the deletes are applied and carried over in RAM. I'd assume the
overhead of flushing the segment would be the main consumption
of CPU and perhaps IO? Which with LUCENE-1313, is less of an
issue.

For consistency with deleteDocument, I'll add live deleting to
updateDocument.

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776691#action_12776691 ]

Michael McCandless commented on LUCENE-2047:
--------------------------------------------

We could very likely hit seeks resolving the term -> docID.  EG to consult the terms dict index (eg page fault), terms dict itself, and then to load the posting.  (Once we have pulsing, we can save that seek).

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776804#action_12776804 ]

Jason Rutherglen commented on LUCENE-2047:
------------------------------------------

bq. likely hit seeks resolving the term -> docID

Right, duh!  Hopefully we won't get into the delete by doc id discussion again.  

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-2047:
-------------------------------------

    Attachment: LUCENE-2047.patch

I added deleting live for updateDocument.

TestNRTReaderWithThreads and TestIndexWriterReader passes.



> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776865#action_12776865 ]

Jason Rutherglen commented on LUCENE-2047:
------------------------------------------

I'd like to remove the syncing in the deleteLive methods, however this causes other tangential exceptions.  

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776868#action_12776868 ]

Jason Rutherglen commented on LUCENE-2047:
------------------------------------------

Without syncing on the writer, we run into the SR's files being deleted out from under it while it's being obtained for deletion (I think).

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776877#action_12776877 ]

Jason Rutherglen commented on LUCENE-2047:
------------------------------------------

After thumbing a bit through the code, somehow, and this is a
bit too complex for me to venture a guess on, we'd need to
prevent the deletion of the SR's files we're deleting from, even
if that SR is no longer live. Which means possibly interacting
with the deletion policy, or some other method. It's a bit hairy
in the segment info transaction shuffling that goes on in IW.

I believe asyncing the live deletes is a good thing, as that way
we're taking advantage of concurrency. The possible expense is in
deleting from segments that are no longer live while the
deleting is occurring.

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777325#action_12777325 ]

Jason Rutherglen commented on LUCENE-2047:
------------------------------------------

I think there's a fundamental design issue here. What happens to
documents that need to be deleted but are still in the RAM
buffer? Because we're not buffering the deletes those wouldn't
be applied. Or would we still buffer the deletes, then only
apply them only to the new SR created from the RAM buffer when
flush or getReader is called?



> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777436#action_12777436 ]

Michael McCandless commented on LUCENE-2047:
--------------------------------------------

{quote}
we'd need to
prevent the deletion of the SR's files we're deleting from, even
if that SR is no longer live.
{quote}

It's strange that anything here is needed, because, when you check a
reader out from the pool, it's incRef'd, which should mean the files
need no protection.  Something strange is up... could it be that when
you checkout that reader to do deletions, it wasn't already open, and
then on trying to open it, its files were already deleted?  (In which
case, that segment has been merged away, and, the merge has committed,
ie already carried over all deletes, and so you should instead be
deleting against that merged segment).

So I think the sync(IW) is in fact necessary?  Note that the current
approach (deferring resolving term -> docIDs until flush time) aiso
sync(IW)'d, so we're not really changing that, here.  Though I agree
it would be nice to not have to sync(IW).  Really what we need to sync
on is "any merge that is merging this segment away and now wants to
commit".  That's actually a very narrow event so someday (separate
issue) if we could refine the sync'ing to that, it should be a good
net throughput improvement for updateDocument.

{quote}
What happens to
documents that need to be deleted but are still in the RAM
buffer?
{quote}

Ahh, yes.  We must still buffer for this case, and resolve these
deletes against the newly flushed segment.  I think we need a separate
buffer that tracks pending delete terms only against the RAM buffer?

Also, instead of actually setting the bits in SR's deletedDocs, I
think you should buffer the deleted docIDs into DW's
deletesInRAM.docIDs?  Ie, we do the resolution of Term/Query -> docID,
but buffer the docIDs we resolved to.  This is necessary for
correctness in exceptional situations, eg if you do a bunch of
updateDocuments, then DW hits an aborting exception (meaning its RAM
buffer may be corrupt) then DW currently discards the RAM buffer, but,
leaves previously flushed segments intact, so that if you then commit,
you have a consistent index.  Ie, in that situation, we don't want the
docs deleted by updateDocument calls to be committed to the index, so
we need to buffer them.


> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777638#action_12777638 ]

Jason Rutherglen commented on LUCENE-2047:
------------------------------------------

bq. It's strange that anything here is needed

I was obtaining the segment infos synced, had a small block of
unsynced code, then synced obtaining the sometimes defunct
readers. Fixed that part, then the errors went away!

bq. the sync(IW) is in fact necessary?

I'm hoping we can do the deletes unsynced, which will make this
patch a net performance gain because we're allowing multiple
threads to delete concurrently (whereas today we're performing
them synced at flush time, i.e. the current patch is merely
shifting the term/query lookup cost from flush to
deleteDocument).

bq. buffer the deleted docIDs into DW's deletesInRAM.docIDs

I'll need to step through this, as it's a little strange to me
how DW knows the doc id to cache for a particular SR, i.e. how
are they mapped to an SR? Oh there's the DW.remapDeletes method?
Hmm...

Couldn't we save off a per SR BV for the update doc rollback
case, merging the special updated doc BV into the SR's deletes
on successful flush, throwing them away on failure?  Memory is
less of a concern with the paged BV from the pending LUCENE-1526
patch.  On a delete by query with many hits, I'm concerned about
storing too many doc id Integers in BufferedDeletes.

Without syncing, new deletes could arrive, and we'd need to
queue them, and apply them to new segments, or newly merged
segments because we're not locking the segments. Otherwise some
deletes could be lost.

A possible solution is, deleteDocument would synchronously add
the delete query/term to a queue per SR and return.
Asynchronously (i.e. in background threads) the deletes could be
applied. Merging would aggregate the incoming SR's queued
deletes (as they haven't been applied yet) into the merged
reader's delete queue. On flush we'd wait for these queued
deletes to be applied.  After flush, the queues would be clear
and we'd start over.  And because the delete queue is per reader,
it would be thrown away with the closed reader.

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777773#action_12777773 ]

Jason Rutherglen commented on LUCENE-2047:
------------------------------------------

Also, if the per SR delete queue were implemented, we could expose
the callback, and allow users to delete by doc id, edit norms
(and in the future, update field caches) for a particular
IndexReader.  We'd pass the reader via a callback that resembles
IndexReaderWarmer, then deletes, norms updates, etc, could be
performed like they can be today with a non-readonly IR.

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777912#action_12777912 ]

Michael McCandless commented on LUCENE-2047:
--------------------------------------------

{quote}
I'm hoping we can do the deletes unsynced, which will make this
patch a net performance gain because we're allowing multiple
threads to delete concurrently (whereas today we're performing
them synced at flush time, i.e. the current patch is merely
shifting the term/query lookup cost from flush to
deleteDocument).
{quote}

Merely shifting the cost off the critical reopen path is already a
good step forward :)

But I agree, if we can also allow deletes to happen concurrently, that
would be fabulous.

Currently the buffered deleted docIDs in DW are only due to exceptions
hit while adding a document.  EG say for a given doc there was an
exception during analysis, after yielding 100 tokens.  At this point
DW's RAM buffer (postings) has already been changed to hold this
docID, and we can't easily undo that.  So we instead mark the docID as
immediately deleted.  These deleted docIDs can then carry in RAM for
some time, and get remapped when merges commit.

bq. On a delete by query with many hits, I'm concerned about storing too many doc id Integers in BufferedDeletes.

I agree, that's a risk if we are buffering an Integer for each deleted
docID, so we should avoid that.

We really just need an efficient IntSet.  Or, it's even possible a
data structure that does no deduping (eg a simple growable int[])
would be fine, since apps would tend not to delete the same docID over
and over.  Since we flush when deletes use too much RAM, we'd be
protected from silly apps...

Or maybe start with a growable int[], but cutover to BV once that's
too large?  This would protect the worst case of deleting by a single
query matching many docs in the index.

Whatever data structure it is, it should live under BufferedDeletes,
so the exception logic that already discards these deletes on hitting
an aborting exception will just continue to work.

bq. Memory is less of a concern with the paged BV from the pending LUCENE-1526 patch

I'm not sure we even want to switch to a paged BV for NRT in general
(just added comment to that effect), though I guess for just this case
(buffering deletes in IW in NRT mode) it could make sense?

{quote}
Couldn't we save off a per SR BV for the update doc rollback
case, merging the special updated doc BV into the SR's deletes
on successful flush, throwing them away on failure?
{quote}

I like that approach.  Because it means the normal case (no exception
is hit) is very fast (no merging of the two BufferedDeletes, on flush,
like we do today; and no docID remapping required), and only on
exception must we restore the previously saved deletes.

Another alternative would be to redefine the semantics of IW on
hitting an error.... right now, if you hit an aborting exception (one
that may have corrupted DW's internal RAM postings), you only lose
those docs buffered in DW at the time.  Any already flushed segments &
new deletions within this IW session are preserved.  So in theory if
you closed your IW, those changes would persist.

We could consider relaxing this, such that the entire session of IW is
rolled back, to the last commit/when-IW-was-opened, just like we now
do with OOME.

{quote}
A possible solution is, deleteDocument would synchronously add
the delete query/term to a queue per SR and return.
Asynchronously (i.e. in background threads) the deletes could be
applied.
{quote}
I'd prefer not to add further BG threads to IW, ie, take the app's
thread to perform the deletion.  If the app wants concurrency for
deletes, it can use multiple threads.

I believe we only have to sync on the merge committing its deletes,
right?  So we should make a separate lock for that?

And, on commit() we must also wait for all in-flight deletes to
finish.

Finally, when a new segment is flushed, we should resolve the buffered
Term/Query deletions against it, during the flush?

{quote}
Also, if the per SR delete queue were implemented, we could expose
the callback, and allow users to delete by doc id, edit norms
(and in the future, update field caches) for a particular
IndexReader. We'd pass the reader via a callback that resembles
IndexReaderWarmer, then deletes, norms updates, etc, could be
performed like they can be today with a non-readonly IR.
{quote}

I'd like to strongly decouple this discussion of extensibility, from
what we're doing in this issue.  I think it's a good idea, but let's
handle it separately -- maybe under LUCENE-2026 (refactoring IW,
which'd pull out the reader pool).  This issue should all be "under
the hood" improvements.



> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778335#action_12778335 ]

Michael McCandless commented on LUCENE-2047:
--------------------------------------------

Thinking more on this... I'm actually no longer convinced that this
change is worthwhile.

Net/net this will not improve the dps/qps throughput on a given fixed
hardware, because this is a zero sum game: the deletes must be
resolved one way or another.

Whether we do it in batch (as today), or incrementally/concurrently,
one at a time as they arrive, the same work must be done.  In fact,
batch should be less costly in practice since it clearly has temporal
locality in resolving terms -> postings, so on a machine whose IO
cache can't hold the entire index in RAM, bulk flushing should be
a win.

It's true that net latency of reopen will be reduced by being
incremental, but Lucene shouldn't aim to be able to reopen 100s of
times per second: I think that's a mis-feature (most apps don't need
it), and those that really do can and should use an approach like
Zoie.

Finally, one can always set the max buffered delete terms/docs to
something low, to achieve this same tradeoff.  It's true that won't
get you concurrent resolving of deleted Terms -> docIDs, but I bet in
practice that concurrency isn't necessary (ie the performance of a
single thread resolving all buffered deletes is plenty fast).

If the reopen time today is plenty fast, especially if you configure
your writer to flush often, then I don't think we need incremental
resolving of the deletions?


> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778452#action_12778452 ]

Jason Rutherglen commented on LUCENE-2047:
------------------------------------------

I want to replay how DW handle the updateDoc call to see if my
understanding is correct.

1: Analyzing hits an exception for a doc, it's doc id has
already been allocated so we mark it for deletion later (on
flush?) in BufferedDeletes.

2: RAM Buffer writing hits an exception, we've had updates which
marked deletes in current segments, however they haven't been
applied yet because they're stored in BufferedDeletes docids.
They're applied on successful flush.

Are these the two scenarios correct or am I completely off
target? If correct, isn't update doc already deleting in the
foreground?

bq. prefer not to add further BG threads

Maybe we can use 1.5's ReentrantReadWriteLock to effectively
allow multiple del/update doc calls to concurrently acquire the
read lock, and perform the deletes in the foreground. The write
lock could be acquired during commitDeletes, commit(), and after
a segment is flushed? I'm not sure it would be necessary to
acquire this write lock anytime segment infos is changed?

I think it's important to remove unnecessary global locks on
unitary operations (like deletes). We've had great results
removing these locks for isDeleted, (NIO)FSDirectory where we
didn't think there'd be an improvement, and there was. I think
this patch (or a follow on one that implements the shared lock
solution) could effectively increase throughput (for deleting
and updating), measurably.

{quote}Lucene shouldn't aim to be able to reopen 100s of times
per second{quote}

Reopening after every doc could be a valid case that I suspect
will come up again in the future. I don't think it's too hard to
support.

{quote} It's true that net latency of reopen will be reduced by
being incremental, but Lucene shouldn't aim to be able to reopen
100s of times per second: {quote}

Perhaps update/del throughput will increase because of the
shared lock which would makes the patch(s) worth implementing.

{quote} but I bet in practice that concurrency isn't necessary
(ie the performance of a single thread resolving all buffered
deletes is plenty fast). {quote}

We thought the same thing about the sync in FSDirectory, and it
turned out that in practice, NIOFSDir is an order of magnitude
faster on *nix machines. For NRT, every little bit of
concurrency will probably increase throughput. (i.e. most users
will have their indexes in IO cache and/or a ram dir, which
means we wouldn't be penalizing concurrency as we are today with
the global lock IW for del/up docs).

I'm going to go ahead and wrap up this patch, which will shift
deletion cost to the del/up methods (still synchronously). Then
create a separate patch that implements the shared lock
solution.

Exposing SRs for updates by the user can be done today, I'll
open a patch for this.

> IndexWriter should immediately resolve deleted docs to docID in near-real-time mode
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2047
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2047
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.1
>
>         Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs.  This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path.  And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12