[jira] Created: (NUTCH-581) DistributedSearch does not update search servers added to search-servers.txt on the fly

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (NUTCH-581) DistributedSearch does not update search servers added to search-servers.txt on the fly

JIRA jira@apache.org
DistributedSearch does not update search servers added to search-servers.txt on the fly
---------------------------------------------------------------------------------------

                 Key: NUTCH-581
                 URL: https://issues.apache.org/jira/browse/NUTCH-581
             Project: Nutch
          Issue Type: Improvement
          Components: searcher
    Affects Versions: 0.9.0
            Reporter: Rohan Mehta
            Priority: Trivial
             Fix For: 0.9.0


DistributedSearch client updates the search servers added to the search-servers.txt file on the fly.
This patch will updates the search servers on the fly and the client does not need a restart.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-581) DistributedSearch does not update search servers added to search-servers.txt on the fly

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/NUTCH-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohan Mehta updated NUTCH-581:
------------------------------

    Attachment: UpdateSearch.patch

> DistributedSearch does not update search servers added to search-servers.txt on the fly
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-581
>                 URL: https://issues.apache.org/jira/browse/NUTCH-581
>             Project: Nutch
>          Issue Type: Improvement
>          Components: searcher
>    Affects Versions: 0.9.0
>            Reporter: Rohan Mehta
>            Priority: Trivial
>             Fix For: 0.9.0
>
>         Attachments: UpdateSearch.patch
>
>
> DistributedSearch client updates the search servers added to the search-servers.txt file on the fly.
> This patch will updates the search servers on the fly and the client does not need a restart.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-581) DistributedSearch does not update search servers added to search-servers.txt on the fly

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/NUTCH-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547977 ]

Dennis Kubes commented on NUTCH-581:
------------------------------------

This is something that we are currently using at Visvo.  It is a simple patch but it allow adding and removing of search servers on the fly without having to shut down the search website.  I am +1 for this patch as it has been in production for us for a few months now.  If nobody objects I would like to commit this in the next day or so.

> DistributedSearch does not update search servers added to search-servers.txt on the fly
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-581
>                 URL: https://issues.apache.org/jira/browse/NUTCH-581
>             Project: Nutch
>          Issue Type: Improvement
>          Components: searcher
>    Affects Versions: 0.9.0
>            Reporter: Rohan Mehta
>            Priority: Trivial
>             Fix For: 0.9.0
>
>         Attachments: UpdateSearch.patch
>
>
> DistributedSearch client updates the search servers added to the search-servers.txt file on the fly.
> This patch will updates the search servers on the fly and the client does not need a restart.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-581) DistributedSearch does not update search servers added to search-servers.txt on the fly

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/NUTCH-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dennis Kubes updated NUTCH-581:
-------------------------------

    Priority: Minor  (was: Trivial)

> DistributedSearch does not update search servers added to search-servers.txt on the fly
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-581
>                 URL: https://issues.apache.org/jira/browse/NUTCH-581
>             Project: Nutch
>          Issue Type: Improvement
>          Components: searcher
>    Affects Versions: 0.9.0
>            Reporter: Rohan Mehta
>            Priority: Minor
>             Fix For: 0.9.0
>
>         Attachments: UpdateSearch.patch
>
>
> DistributedSearch client updates the search servers added to the search-servers.txt file on the fly.
> This patch will updates the search servers on the fly and the client does not need a restart.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-581) DistributedSearch does not update search servers added to search-servers.txt on the fly

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/NUTCH-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548045 ]

Andrzej Bialecki  commented on NUTCH-581:
-----------------------------------------

+1 as the first step ...

I think we should extend this support to include on-the-fly update in other scenarios (e.g. detecting updated indexes on search servers, requesting reload of backends from the frontend, etc).

Solr implements a good model for searcher reload. When an IndexSearcher needs to be reloaded, Solr prepares a new searcher in the background and warms it up by running the topN queries, and only when the new searcher is "warm" then it atomically switches the searchers, and closes the old one. IMHO we should eventually implement a model like this.

> DistributedSearch does not update search servers added to search-servers.txt on the fly
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-581
>                 URL: https://issues.apache.org/jira/browse/NUTCH-581
>             Project: Nutch
>          Issue Type: Improvement
>          Components: searcher
>    Affects Versions: 0.9.0
>            Reporter: Rohan Mehta
>            Priority: Minor
>             Fix For: 0.9.0
>
>         Attachments: UpdateSearch.patch
>
>
> DistributedSearch client updates the search servers added to the search-servers.txt file on the fly.
> This patch will updates the search servers on the fly and the client does not need a restart.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-581) DistributedSearch does not update search servers added to search-servers.txt on the fly

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/NUTCH-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548052 ]

Dennis Kubes commented on NUTCH-581:
------------------------------------

I was talking with Docagan about this very thing.  He has some ideas for a framework that would support monitoring and maintenance of search servers.  Similar to a hadoop or hbase model of master and slaves where you could add and remove search servers, change out indexes, etc.  Maybe we can all start designing this out together fairly soon.  I think it would be a very useful addition, especially for those of us managing large numbers of search servers.

> DistributedSearch does not update search servers added to search-servers.txt on the fly
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-581
>                 URL: https://issues.apache.org/jira/browse/NUTCH-581
>             Project: Nutch
>          Issue Type: Improvement
>          Components: searcher
>    Affects Versions: 0.9.0
>            Reporter: Rohan Mehta
>            Priority: Minor
>             Fix For: 0.9.0
>
>         Attachments: UpdateSearch.patch
>
>
> DistributedSearch client updates the search servers added to the search-servers.txt file on the fly.
> This patch will updates the search servers on the fly and the client does not need a restart.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-581) DistributedSearch does not update search servers added to search-servers.txt on the fly

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/NUTCH-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dennis Kubes updated NUTCH-581:
-------------------------------

    Attachment: NUTCH-581-2.patch

Update patch.  Passes all unit tests, including those on linux.  Renames DistributedSearchTest to TestDistributedSearch to be consistent with our current build file for unit tests.

> DistributedSearch does not update search servers added to search-servers.txt on the fly
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-581
>                 URL: https://issues.apache.org/jira/browse/NUTCH-581
>             Project: Nutch
>          Issue Type: Improvement
>          Components: searcher
>    Affects Versions: 0.9.0
>            Reporter: Rohan Mehta
>            Priority: Minor
>             Fix For: 0.9.0
>
>         Attachments: NUTCH-581-2.patch, UpdateSearch.patch
>
>
> DistributedSearch client updates the search servers added to the search-servers.txt file on the fly.
> This patch will updates the search servers on the fly and the client does not need a restart.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-581) DistributedSearch does not update search servers added to search-servers.txt on the fly

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/NUTCH-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548252 ]

Doğacan Güney commented on NUTCH-581:
-------------------------------------

This patch conflicts with my patch in NUTCH-442 (which I really want to commit sometime) but that's my problem :). So +1 from me.



> DistributedSearch does not update search servers added to search-servers.txt on the fly
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-581
>                 URL: https://issues.apache.org/jira/browse/NUTCH-581
>             Project: Nutch
>          Issue Type: Improvement
>          Components: searcher
>    Affects Versions: 0.9.0
>            Reporter: Rohan Mehta
>            Priority: Minor
>             Fix For: 0.9.0
>
>         Attachments: NUTCH-581-2.patch, UpdateSearch.patch
>
>
> DistributedSearch client updates the search servers added to the search-servers.txt file on the fly.
> This patch will updates the search servers on the fly and the client does not need a restart.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (NUTCH-581) DistributedSearch does not update search servers added to search-servers.txt on the fly

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/NUTCH-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dennis Kubes resolved NUTCH-581.
--------------------------------

    Resolution: Fixed

Patch committed.  This patch check modified time on search-servers.txt file and automatically reloads if changed.  This allows added and removing search servers on the fly.Thanks Rohan.

> DistributedSearch does not update search servers added to search-servers.txt on the fly
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-581
>                 URL: https://issues.apache.org/jira/browse/NUTCH-581
>             Project: Nutch
>          Issue Type: Improvement
>          Components: searcher
>    Affects Versions: 0.9.0
>            Reporter: Rohan Mehta
>            Priority: Minor
>             Fix For: 0.9.0
>
>         Attachments: NUTCH-581-2.patch, UpdateSearch.patch
>
>
> DistributedSearch client updates the search servers added to the search-servers.txt file on the fly.
> This patch will updates the search servers on the fly and the client does not need a restart.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-581) DistributedSearch does not update search servers added to search-servers.txt on the fly

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/NUTCH-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548541 ]

Hudson commented on NUTCH-581:
------------------------------

Integrated in Nutch-Nightly #285 (See [http://lucene.zones.apache.org:8080/hudson/job/Nutch-Nightly/285/])

> DistributedSearch does not update search servers added to search-servers.txt on the fly
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-581
>                 URL: https://issues.apache.org/jira/browse/NUTCH-581
>             Project: Nutch
>          Issue Type: Improvement
>          Components: searcher
>    Affects Versions: 0.9.0
>            Reporter: Rohan Mehta
>            Priority: Minor
>             Fix For: 0.9.0
>
>         Attachments: NUTCH-581-2.patch, UpdateSearch.patch
>
>
> DistributedSearch client updates the search servers added to the search-servers.txt file on the fly.
> This patch will updates the search servers on the fly and the client does not need a restart.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.