[jira] Created: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
Provide a simple way to concurrently access a Lucene index from multiple threads
--------------------------------------------------------------------------------

                 Key: LUCENE-1026
                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
             Project: Lucene - Java
          Issue Type: New Feature
          Components: Index, Search
            Reporter: Mark Miller
            Priority: Minor


For building interactive indexes accessed through a network/internet (multiple threads).

This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).

This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.

Future revisions will be zipped.

Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated LUCENE-1026:
--------------------------------

    Attachment: TestIndexAccessor.java
                StopWatch.java

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated LUCENE-1026:
--------------------------------

    Attachment: SimpleSearchServer.java
                MultiIndexAccessor.java
                IndexAccessorFactory.java

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated LUCENE-1026:
--------------------------------

    Attachment: IndexAccessor.java
                DefaultMultiIndexAccessor.java
                DefaultIndexAccessor.java

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546288 ]

Shai Erera commented on LUCENE-1026:
------------------------------------

Hi
I've downloaded the code and tried to run the tests, but I think there are some problems:
1. The delete() method in the test attempts to delete the directory, and not the underlying files. So in effect it does not do anything.
2. Some of the tests that start new threads don't wait for them (by calling join()). That of course causes some Accessors to be removed (after you call closeAllAccessors()), while those threads are sill running.
I've fixed those issues in the test. I'd appreciate if you can take a look.

Also, in IndexAccessorFactory I've found some issues:
1. I guess you wanted to have it as a Singleton - so I defined a private default constructor to prevent applications from instantiating it.
2. I modified the code of createAccessor to first lookup if an accessor for that directory already exists. It should save the allocation of DefaultIndexAccessor.
3. I modified the implementation of other methods to access the HashMap of accessors more efficiently.

I'd appreciate if you can review my fixes. I'll attach them separately.

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shai Erera updated LUCENE-1026:
-------------------------------

    Attachment: IndexAccessorFactory.java

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shai Erera updated LUCENE-1026:
-------------------------------

    Attachment: shai-IndexAccessor.zip

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shai Erera updated LUCENE-1026:
-------------------------------

    Attachment:     (was: IndexAccessorFactory.java)

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546304 ]

Mark Miller commented on LUCENE-1026:
-------------------------------------

Hey Shai,

These fixes are great and I will incorporate them all.

I worked this up very quickly based on other less general code I am using. While I have not yet used this code for a project, it will be the framework that I migrate to for future projects. It should see much more development then. I am very eager to add some Searcher warming for one. Also, the tests where whipped together quite quickly. I appreciate your efforts at cleaning them up.

Buffing up the SearchServer code to production level will also be on my list.

Thanks for your improvements -- if you do any more work, keep me posted.

- Mark

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546592 ]

Shai Erera commented on LUCENE-1026:
------------------------------------

Hey Mark, few more questions:
1. Why is StopWatch needed? it seems like the only thing it does is count the time of execution of the code. That can be easily achieved by writing this:
long time = System.currentTimeMillis();
<code>
write any code here
</code>
time = System.currentTimeMillis() - time;
System.out.println('time: " + time + " ms");

Am I missing something?

2. In DefaultIndexAccessor there's this code:
  public int writersOut() {
    int writers;

    synchronized (this) {
      writers = this.writerUseCount;
    }

    return writers;
  }

I think it should be replaced with:
  public int writersOut() {
    return writerUseCount;
  }
No need to synchronize on *this* since you're only returning the value, not performing any modifications to it. The same goes for writingReadersOut, activeSearchers, isOpen.
What do you think?

3. open() -- is there a real need to throw an exception if someone called the accessor.open() twice?

I'll attach the modified test and DefaultIndexAccessor files for your review. I'd apprecite if you can answer the questions.

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shai Erera updated LUCENE-1026:
-------------------------------

    Attachment: shai-IndexAccessor-2.zip

Includes the changes to the files based on the comments I've recently posted

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546694 ]

Shai Erera commented on LUCENE-1026:
------------------------------------

Hey Mark,
I've cleaned up the code, added javadoc and organized the tests. Can you please review the changes - I think the code is even more simpler now.

1. I think we should remove StopWatch. Unless I'm missing something, it looks like it's doing nothing important, inefficiently :-).
2. What is the SimpleSearchServer? Is this like a sample program?

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shai Erera updated LUCENE-1026:
-------------------------------

    Attachment: shai-IndexAccessor3.zip

Based on my previous comments.

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546702 ]

Mark Miller commented on LUCENE-1026:
-------------------------------------

Hey Mark, few more questions:
1. Why is StopWatch needed?

StopWatch could certainly be taken out. I simply used it for timing...to show someone else that it doesn't take that long to get an accessor. I have no problem with removing it.

2. No need to synchronize on this since you're only returning the value, not performing any modifications to it.

Its possible this could be taken out. I doubt there would be any noticeable trouble if you tried to read that value while another thread was updating it.

3. open() - is there a real need to throw an exception if someone called the accessor.open() twice?

No real need I suppose. I believe this came in from the original code. I have no objections to taking it out.


SimpleSearchServer is an attempt to give an example that shows how this code can be used to create a very simple class that allows multi-threaded access to a Lucene index. I whipped it out extremely quickly, but its something I would like to build on. It can be nice to hide the IndexAccessor code behind a simple to use class that hides Lucene and a lot of complexity.

Thanks for your work on this!

- Mark

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546719 ]

Mark Miller commented on LUCENE-1026:
-------------------------------------

Quick comment:

You have the MultiSearcher closing its own IndexAccessor's now. Maybe I messed up copying in your changes, but this causes an Exception in the tests as the IndexFactory also closes all of the IndexAccessor's. (Really, it owns them)

It's probably a good idea for the MultiSearcher to have a close (clearing the two caches is not a bad idea), but I don't think the MultiSearcher should close IndexAccessors. It does not own those Accessors and other threads may be using them from the Factory. They are only kept around in the MultiSearcher so that it can release them later, back to the Factory.

Also, probably the Factory.close method should call the MultiSearcher close rather than having to get the MuliSearcher and call close separately?

Thoughts?

- Mark

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Shai Erera
Part of the changes I've done is to have close() not throw exception when it
is being called twice (I don't think it's bad to call it twice). Maybe you
didn't copy that part of the code?
Because the tests run on my machine ...

I agree on the last statement. I just didn't want to introduce too many
changes at once. But having factory.close() close the MultiSearchAccessor as
well is good. If you change that, then the MultiSearcher test can be changed
as well to not call close().

You are right about the comment on MultiIndexAccessor's close implementation
(not closing its IndexAccessor instances). I must say that its functionality
is not very clear, because of lack of documentation (that's why I didn't
complete the documentation for this class). Can you update the javadocs
while you're changing that?

Thanks,

Shai.


On Nov 29, 2007 3:25 PM, Mark Miller (JIRA) <[hidden email]> wrote:

>
>    [
> https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546719]
>
> Mark Miller commented on LUCENE-1026:
> -------------------------------------
>
> Quick comment:
>
> You have the MultiSearcher closing its own IndexAccessor's now. Maybe I
> messed up copying in your changes, but this causes an Exception in the tests
> as the IndexFactory also closes all of the IndexAccessor's. (Really, it owns
> them)
>
> It's probably a good idea for the MultiSearcher to have a close (clearing
> the two caches is not a bad idea), but I don't think the MultiSearcher
> should close IndexAccessors. It does not own those Accessors and other
> threads may be using them from the Factory. They are only kept around in the
> MultiSearcher so that it can release them later, back to the Factory.
>
> Also, probably the Factory.close method should call the MultiSearcher
> close rather than having to get the MuliSearcher and call close separately?
>
> Thoughts?
>
> - Mark
>
> > Provide a simple way to concurrently access a Lucene index from multiple
> threads
> >
> --------------------------------------------------------------------------------
> >
> >                 Key: LUCENE-1026
> >                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
> >             Project: Lucene - Java
> >          Issue Type: New Feature
> >          Components: Index, Search
> >            Reporter: Mark Miller
> >            Priority: Minor
> >         Attachments: DefaultIndexAccessor.java,
> DefaultMultiIndexAccessor.java, IndexAccessor.java,
> IndexAccessorFactory.java, MultiIndexAccessor.java,
> shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip,
> SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
> >
> >
> > For building interactive indexes accessed through a network/internet
> (multiple threads).
> > This builds upon the LuceneIndexAccessor patch. That patch was not very
> newbie friendly and did not properly handle MultiSearchers (or at the least
> made it easy to get into trouble).
> > This patch simplifies things and provides out of the box support for
> sharing the IndexAccessors across threads. There is also a simple test class
> and example SearchServer to get you started.
> > Future revisions will be zipped.
> > Works pretty solid as is, but could use the ability to warm new
> Searchers.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Regards,

Shai Erera
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Robert Engels
In reply to this post by Tim Allison (Jira)
FYI, any time one thread reads a value while another thread updates  
it it needs to be synchronized, or with current JVMs a volatile  
variable.

The Java Memory Model requires this.

Otherwise you can get a partial value (when the underlying value  
requires more than one memory access to retrieve).

On Nov 29, 2007, at 6:43 AM, Mark Miller (JIRA) wrote:

>
>     [ https://issues.apache.org/jira/browse/LUCENE-1026?
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel#action_12546702 ]
>
> Mark Miller commented on LUCENE-1026:
> -------------------------------------
>
> Hey Mark, few more questions:
> 1. Why is StopWatch needed?
>
> StopWatch could certainly be taken out. I simply used it for  
> timing...to show someone else that it doesn't take that long to get  
> an accessor. I have no problem with removing it.
>
> 2. No need to synchronize on this since you're only returning the  
> value, not performing any modifications to it.
>
> Its possible this could be taken out. I doubt there would be any  
> noticeable trouble if you tried to read that value while another  
> thread was updating it.
>
> 3. open() - is there a real need to throw an exception if someone  
> called the accessor.open() twice?
>
> No real need I suppose. I believe this came in from the original  
> code. I have no objections to taking it out.
>
>
> SimpleSearchServer is an attempt to give an example that shows how  
> this code can be used to create a very simple class that allows  
> multi-threaded access to a Lucene index. I whipped it out extremely  
> quickly, but its something I would like to build on. It can be nice  
> to hide the IndexAccessor code behind a simple to use class that  
> hides Lucene and a lot of complexity.
>
> Thanks for your work on this!
>
> - Mark
>
>> Provide a simple way to concurrently access a Lucene index from  
>> multiple threads
>> ---------------------------------------------------------------------
>> -----------
>>
>>                 Key: LUCENE-1026
>>                 URL: https://issues.apache.org/jira/browse/ 
>> LUCENE-1026
>>             Project: Lucene - Java
>>          Issue Type: New Feature
>>          Components: Index, Search
>>            Reporter: Mark Miller
>>            Priority: Minor
>>         Attachments: DefaultIndexAccessor.java,  
>> DefaultMultiIndexAccessor.java, IndexAccessor.java,  
>> IndexAccessorFactory.java, MultiIndexAccessor.java, shai-
>> IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-
>> IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java,  
>> TestIndexAccessor.java
>>
>>
>> For building interactive indexes accessed through a network/
>> internet (multiple threads).
>> This builds upon the LuceneIndexAccessor patch. That patch was not  
>> very newbie friendly and did not properly handle MultiSearchers  
>> (or at the least made it easy to get into trouble).
>> This patch simplifies things and provides out of the box support  
>> for sharing the IndexAccessors across threads. There is also a  
>> simple test class and example SearchServer to get you started.
>> Future revisions will be zipped.
>> Works pretty solid as is, but could use the ability to warm new  
>> Searchers.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Mark Miller-3
Thanks Robert. Ill keep the sync then. I only considered it possible as
the read is for reporting type purposes and so is not relied on for
functionality. Sounds like we better retain the sync anyway though.

Shai:

I have incorporated your code into mine. Looks great so far.

There is a rather large MultiIndexAccessor issue that I have to deal with though. It may turn out that MultiSearchers may not be able to be cached. Without passing information about Writer releases to the MultiIndexAccessor, the MultiIndexAccessor will hold onto old Searchers that it shouldn't (the cache never gets cleared). The simple solution is to take out the MultiSearcher caching and just make a new one on every request. This is not a huge deal as the underlying Searchers will be cached, but I liked avoiding the calls to getIndexAcessor if possible. Its prob best to drop the cache though. I think if this is done, your close method on MultiIndexAccessor can  be removed.

I have added to the tests to expose this issue.

I will post updated code later.

- Mark


robert engels wrote:

> FYI, any time one thread reads a value while another thread updates it
> it needs to be synchronized, or with current JVMs a volatile variable.
>
> The Java Memory Model requires this.
>
> Otherwise you can get a partial value (when the underlying value
> requires more than one memory access to retrieve).
>
> On Nov 29, 2007, at 6:43 AM, Mark Miller (JIRA) wrote:
>
>>
>>     [
>> https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546702 
>> ]
>>
>> Mark Miller commented on LUCENE-1026:
>> -------------------------------------
>>
>> Hey Mark, few more questions:
>> 1. Why is StopWatch needed?
>>
>> StopWatch could certainly be taken out. I simply used it for
>> timing...to show someone else that it doesn't take that long to get
>> an accessor. I have no problem with removing it.
>>
>> 2. No need to synchronize on this since you're only returning the
>> value, not performing any modifications to it.
>>
>> Its possible this could be taken out. I doubt there would be any
>> noticeable trouble if you tried to read that value while another
>> thread was updating it.
>>
>> 3. open() - is there a real need to throw an exception if someone
>> called the accessor.open() twice?
>>
>> No real need I suppose. I believe this came in from the original
>> code. I have no objections to taking it out.
>>
>>
>> SimpleSearchServer is an attempt to give an example that shows how
>> this code can be used to create a very simple class that allows
>> multi-threaded access to a Lucene index. I whipped it out extremely
>> quickly, but its something I would like to build on. It can be nice
>> to hide the IndexAccessor code behind a simple to use class that
>> hides Lucene and a lot of complexity.
>>
>> Thanks for your work on this!
>>
>> - Mark
>>
>>> Provide a simple way to concurrently access a Lucene index from
>>> multiple threads
>>> --------------------------------------------------------------------------------
>>>
>>>
>>>                 Key: LUCENE-1026
>>>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>>>             Project: Lucene - Java
>>>          Issue Type: New Feature
>>>          Components: Index, Search
>>>            Reporter: Mark Miller
>>>            Priority: Minor
>>>         Attachments: DefaultIndexAccessor.java,
>>> DefaultMultiIndexAccessor.java, IndexAccessor.java,
>>> IndexAccessorFactory.java, MultiIndexAccessor.java,
>>> shai-IndexAccessor-2.zip, shai-IndexAccessor.zip,
>>> shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java,
>>> TestIndexAccessor.java
>>>
>>>
>>> For building interactive indexes accessed through a network/internet
>>> (multiple threads).
>>> This builds upon the LuceneIndexAccessor patch. That patch was not
>>> very newbie friendly and did not properly handle MultiSearchers (or
>>> at the least made it easy to get into trouble).
>>> This patch simplifies things and provides out of the box support for
>>> sharing the IndexAccessors across threads. There is also a simple
>>> test class and example SearchServer to get you started.
>>> Future revisions will be zipped.
>>> Works pretty solid as is, but could use the ability to warm new
>>> Searchers.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated LUCENE-1026:
--------------------------------

    Attachment: IndexAccessor.zip

> Provide a simple way to concurrently access a Lucene index from multiple threads
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-1026
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index, Search
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessor.zip, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
>
>
> For building interactive indexes accessed through a network/internet (multiple threads).
> This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble).
> This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started.
> Future revisions will be zipped.
> Works pretty solid as is, but could use the ability to warm new Searchers.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads

Mark Miller-3
In reply to this post by Robert Engels
I've been thinking about this and I think the situation we are in is
okay. The variable is an int and so should be read in one memory access
(i would think?). So at worst, the result might be stale? Since this
read is just for informational type purposes, (eg its ok if this
particular method returns a stale value, by the time you read it its
likely to have changed anyway), I think its okay.

I think. Everything I read about the Java memory model, I quickly forget.

robert engels wrote:

> FYI, any time one thread reads a value while another thread updates it
> it needs to be synchronized, or with current JVMs a volatile variable.
>
> The Java Memory Model requires this.
>
> Otherwise you can get a partial value (when the underlying value
> requires more than one memory access to retrieve).
>
> On Nov 29, 2007, at 6:43 AM, Mark Miller (JIRA) wrote:
>
>>
>>     [
>> https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546702 
>> ]
>>
>> Mark Miller commented on LUCENE-1026:
>> -------------------------------------
>>
>> Hey Mark, few more questions:
>> 1. Why is StopWatch needed?
>>
>> StopWatch could certainly be taken out. I simply used it for
>> timing...to show someone else that it doesn't take that long to get
>> an accessor. I have no problem with removing it.
>>
>> 2. No need to synchronize on this since you're only returning the
>> value, not performing any modifications to it.
>>
>> Its possible this could be taken out. I doubt there would be any
>> noticeable trouble if you tried to read that value while another
>> thread was updating it.
>>
>> 3. open() - is there a real need to throw an exception if someone
>> called the accessor.open() twice?
>>
>> No real need I suppose. I believe this came in from the original
>> code. I have no objections to taking it out.
>>
>>
>> SimpleSearchServer is an attempt to give an example that shows how
>> this code can be used to create a very simple class that allows
>> multi-threaded access to a Lucene index. I whipped it out extremely
>> quickly, but its something I would like to build on. It can be nice
>> to hide the IndexAccessor code behind a simple to use class that
>> hides Lucene and a lot of complexity.
>>
>> Thanks for your work on this!
>>
>> - Mark
>>
>>> Provide a simple way to concurrently access a Lucene index from
>>> multiple threads
>>> --------------------------------------------------------------------------------
>>>
>>>
>>>                 Key: LUCENE-1026
>>>                 URL: https://issues.apache.org/jira/browse/LUCENE-1026
>>>             Project: Lucene - Java
>>>          Issue Type: New Feature
>>>          Components: Index, Search
>>>            Reporter: Mark Miller
>>>            Priority: Minor
>>>         Attachments: DefaultIndexAccessor.java,
>>> DefaultMultiIndexAccessor.java, IndexAccessor.java,
>>> IndexAccessorFactory.java, MultiIndexAccessor.java,
>>> shai-IndexAccessor-2.zip, shai-IndexAccessor.zip,
>>> shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java,
>>> TestIndexAccessor.java
>>>
>>>
>>> For building interactive indexes accessed through a network/internet
>>> (multiple threads).
>>> This builds upon the LuceneIndexAccessor patch. That patch was not
>>> very newbie friendly and did not properly handle MultiSearchers (or
>>> at the least made it easy to get into trouble).
>>> This patch simplifies things and provides out of the box support for
>>> sharing the IndexAccessors across threads. There is also a simple
>>> test class and example SearchServer to get you started.
>>> Future revisions will be zipped.
>>> Works pretty solid as is, but could use the ability to warm new
>>> Searchers.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12