[jira] Created: (NUTCH-251) Administration GUI

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (NUTCH-251) Administration GUI

Hudson (Jira)
Administration GUI
------------------

         Key: NUTCH-251
         URL: http://issues.apache.org/jira/browse/NUTCH-251
     Project: Nutch
        Type: Improvement

    Versions: 0.8-dev    
    Reporter: Stefan Groschupf
    Priority: Minor
     Fix For: 0.8-dev


Having a web based administration interface would help to make nutch administration and management much more user friendly.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-251) Administration GUI

Hudson (Jira)
     [ http://issues.apache.org/jira/browse/NUTCH-251?page=all ]

Stefan Groschupf updated NUTCH-251:
-----------------------------------

    Attachment: hadoop_nutch_gui_v1.patch
                nutch_gui_v1.patch
                nutch_gui_plugins_v1.zip

This is a early preview patch of the nutch gui.
There are known issues, however it is a starting point from where we can continue building a solid administration user interface.


This patch introduce following functionalities:

+ web based administration gui via embed web container
+ gui is fully based  on the plugin system, so it is customizable and  extendable using plugins
+ all plugins can be internationalized  
+ introduce the concept of nutch instances, a mechanism to have separated configurable nutch deployments using the same code base. (e.g intranet search, webpage search)
+ plug able authentication, currently it comes with a default user  - password tuple based on the configuration but for example LDAP integration can be easily realized.  

The patch it comes with following plugins:
+ admin-listing
++ required by the web ui to show all deployed plugins as tabs on a webpage

+ admin-instance
++ lists all instances and allows to create a new instance

+ admin-configuration
++ configure a nutch instance (configuration will be written as nutch-site.xml to hdd)

+ admin-inject
++ inject urls in a crawlDb

+admin-system
++ shows status of system

+admin-job
++ shows  status of jobs

+ admin-crawldb-status
++ shows crawldb entries filtered by status or  shows the status of a given url  (usefully to check if a page was already fetched)

+admin-management
++ generate segment
++ fetch segment
++ parse segment (if required)
++ update crawldb
++ invert links
++ index segment
++ delete segment, parse, index etc.

+admin-scheduling
++ quartz based cron job management to run a time driven "generate - fetch - updatedb - invertlins - index" job


Known issues
+ require hadoop changes
+ local running jobs can not be stopped but distributed running jobs can be stopped
+ index searcher does not use index folders inside of segment folders as in nutch 0.7 but the gui place the index folder in the segment folder
++ searcher is unable to find indices
+ put to search does not work since searcher does not support dynamically adding of index folders
+ linkdb inverter does not update but overwrite a linkdb - this is a general nutch bug but affect the gui as well.
+ the nutch gui introduce locking by storing lock files in folders, this mechanism is ignored by the nutch command line tools.



It would be great if users can test the gui and reports bugs and help to improve the patch.
This is a very complex patch and it is difficult to stay in sync with the latest changes so in case we miss something
until generation this patch and the patch does not work as expected please don't blame us but give us some time and hints to fix the problems.


 help is welcome by following tasks:
+ fixing languages issues in java doc, api and bundle files
+ translate bundles in more languages (currently it comes with english and german bundles)
+ heavily test and find bugs and provide fixes :)
+ write help texts and documentation

How to:

+ checkout latest nutch sources

+ checkout hadoop sources
+ patch hadoop with the hadoop patch
+ build hadoop jar
+ remove old hadoop jar from nutch/lib
+ place new hadoop jar in nutch/lib


+ uncompress plugin zip file
+ place plugins in nutch/src/plugins (patch not possible since svn does not support binary patches)
+ patch nutch with nutch patch
+ start gui with bin/nutch gui <folderWhereYourInstanceDataWillBeStored)
+ point your browser to: http://localhost:50060/general/
+ username and password are "admin". ( can be changed in nutch-default.xml)
+ select the "default" instance or create a new instance.



Thanks to everybody that helped to get this implement and do the first beta tests, but specially to Marko hacking all jsp's!
I suggest to add this patch to a nutch 0.9 branch and add a gui component in the jira to go from there.
I really hope I didn't miss anything or upload the wrong files now. :-O

> Administration GUI
> ------------------
>
>          Key: NUTCH-251
>          URL: http://issues.apache.org/jira/browse/NUTCH-251
>      Project: Nutch
>         Type: Improvement

>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Minor
>      Fix For: 0.8-dev
>  Attachments: hadoop_nutch_gui_v1.patch, nutch_gui_plugins_v1.zip, nutch_gui_v1.patch
>
> Having a web based administration interface would help to make nutch administration and management much more user friendly.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-251) Administration GUI

Hudson (Jira)
In reply to this post by Hudson (Jira)
    [ http://issues.apache.org/jira/browse/NUTCH-251?page=comments#action_12375685 ]

Zaheed Haque commented on NUTCH-251:
------------------------------------

Great Job, Stefan and Marko! Just tried it looks very good. I am currently running 3 search site with 0.8-dev, using  automated shell scripts I can do most, but this patch will allow me to run things from home, some of my clients doesn't allow ssh access. So I hope this patch gets included, I have just voted for it :-)

Thanks again Stefan and Marko for your hard work!

Cheers

> Administration GUI
> ------------------
>
>          Key: NUTCH-251
>          URL: http://issues.apache.org/jira/browse/NUTCH-251
>      Project: Nutch
>         Type: Improvement

>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Minor
>      Fix For: 0.8-dev
>  Attachments: hadoop_nutch_gui_v1.patch, nutch_gui_plugins_v1.zip, nutch_gui_v1.patch
>
> Having a web based administration interface would help to make nutch administration and management much more user friendly.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Updated: (NUTCH-251) Administration GUI

Thomas Delnoij-3
In reply to this post by Hudson (Jira)
Stefan,

this patch looks very interesting. I would like to test it, but before
I have several questions.

1) I am using local filesystem - as you say that local running jobs
can not be stopped, does that imply that the scheduling is
dysfunctional as well on local filesystem

2) Do you think it makes sense to have a language bundle in Dutch -
personally I don't, because I never meet a Dutch developer who doesn't
speak English, but I could do it anyway.

3) What you like to operate as a filter for the bugs before we add
them to Jira, so that we don't post any known issues / duplicates.

Thanks for all the work.

Rgrds, Thomas Delnoij

On 4/22/06, Stefan Groschupf (JIRA) <[hidden email]> wrote:

>      [ http://issues.apache.org/jira/browse/NUTCH-251?page=all ]
>
> Stefan Groschupf updated NUTCH-251:
> -----------------------------------
>
>     Attachment: hadoop_nutch_gui_v1.patch
>                 nutch_gui_v1.patch
>                 nutch_gui_plugins_v1.zip
>
> This is a early preview patch of the nutch gui.
> There are known issues, however it is a starting point from where we can continue building a solid administration user interface.
>
>
> This patch introduce following functionalities:
>
> + web based administration gui via embed web container
> + gui is fully based  on the plugin system, so it is customizable and  extendable using plugins
> + all plugins can be internationalized
> + introduce the concept of nutch instances, a mechanism to have separated configurable nutch deployments using the same code base. (e.g intranet search, webpage search)
> + plug able authentication, currently it comes with a default user  - password tuple based on the configuration but for example LDAP integration can be easily realized.
>
> The patch it comes with following plugins:
> + admin-listing
> ++ required by the web ui to show all deployed plugins as tabs on a webpage
>
> + admin-instance
> ++ lists all instances and allows to create a new instance
>
> + admin-configuration
> ++ configure a nutch instance (configuration will be written as nutch-site.xml to hdd)
>
> + admin-inject
> ++ inject urls in a crawlDb
>
> +admin-system
> ++ shows status of system
>
> +admin-job
> ++ shows  status of jobs
>
> + admin-crawldb-status
> ++ shows crawldb entries filtered by status or  shows the status of a given url  (usefully to check if a page was already fetched)
>
> +admin-management
> ++ generate segment
> ++ fetch segment
> ++ parse segment (if required)
> ++ update crawldb
> ++ invert links
> ++ index segment
> ++ delete segment, parse, index etc.
>
> +admin-scheduling
> ++ quartz based cron job management to run a time driven "generate - fetch - updatedb - invertlins - index" job
>
>
> Known issues
> + require hadoop changes
> + local running jobs can not be stopped but distributed running jobs can be stopped
> + index searcher does not use index folders inside of segment folders as in nutch 0.7 but the gui place the index folder in the segment folder
> ++ searcher is unable to find indices
> + put to search does not work since searcher does not support dynamically adding of index folders
> + linkdb inverter does not update but overwrite a linkdb - this is a general nutch bug but affect the gui as well.
> + the nutch gui introduce locking by storing lock files in folders, this mechanism is ignored by the nutch command line tools.
>
>
>
> It would be great if users can test the gui and reports bugs and help to improve the patch.
> This is a very complex patch and it is difficult to stay in sync with the latest changes so in case we miss something
> until generation this patch and the patch does not work as expected please don't blame us but give us some time and hints to fix the problems.
>
>
>  help is welcome by following tasks:
> + fixing languages issues in java doc, api and bundle files
> + translate bundles in more languages (currently it comes with english and german bundles)
> + heavily test and find bugs and provide fixes :)
> + write help texts and documentation
>
> How to:
>
> + checkout latest nutch sources
>
> + checkout hadoop sources
> + patch hadoop with the hadoop patch
> + build hadoop jar
> + remove old hadoop jar from nutch/lib
> + place new hadoop jar in nutch/lib
>
>
> + uncompress plugin zip file
> + place plugins in nutch/src/plugins (patch not possible since svn does not support binary patches)
> + patch nutch with nutch patch
> + start gui with bin/nutch gui <folderWhereYourInstanceDataWillBeStored)
> + point your browser to: http://localhost:50060/general/
> + username and password are "admin". ( can be changed in nutch-default.xml)
> + select the "default" instance or create a new instance.
>
>
>
> Thanks to everybody that helped to get this implement and do the first beta tests, but specially to Marko hacking all jsp's!
> I suggest to add this patch to a nutch 0.9 branch and add a gui component in the jira to go from there.
> I really hope I didn't miss anything or upload the wrong files now. :-O
>
> > Administration GUI
> > ------------------
> >
> >          Key: NUTCH-251
> >          URL: http://issues.apache.org/jira/browse/NUTCH-251
> >      Project: Nutch
> >         Type: Improvement
>
> >     Versions: 0.8-dev
> >     Reporter: Stefan Groschupf
> >     Priority: Minor
> >      Fix For: 0.8-dev
> >  Attachments: hadoop_nutch_gui_v1.patch, nutch_gui_plugins_v1.zip, nutch_gui_v1.patch
> >
> > Having a web based administration interface would help to make nutch administration and management much more user friendly.
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators:
>    http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
>    http://www.atlassian.com/software/jira
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Updated: (NUTCH-251) Administration GUI

Stefan Groschupf-2
Hi Thomas,
> 1) I am using local filesystem - as you say that local running jobs
> can not be stopped, does that imply that the scheduling is
> dysfunctional as well on local filesystem

The gui fully works in a local file system and a local jobrunner,  
except of stopping a running job.
http://mail-archives.apache.org/mod_mbox/lucene-hadoop-dev/ 
200603.mbox/%[hidden email]%3E


>
> 2) Do you think it makes sense to have a language bundle in Dutch -
> personally I don't, because I never meet a Dutch developer who doesn't
> speak English, but I could do it anyway.

Getting as much as possible would increase the use friendliness.
However we may be need to test it and fix bugs before we start to  
translate more the ui in more languages.


>
> 3) What you like to operate as a filter for the bugs before we add
> them to Jira, so that we don't post any known issues / duplicates.

I created a component called administration gui, just attach bug  
reports under this component.
If we all use the jira than we will know what are known problems.

>
> Thanks for all the work.

Thanks for looking into it.

Stefan

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Updated: (NUTCH-251) Administration GUI

Thomas Delnoij-3
One more question: and where shall we post questions about the admin gui, on
the dev or the users list?

Rgrds, Thomas Delnoij

On 4/26/06, Stefan Groschupf <[hidden email]> wrote:

>
> Hi Thomas,
> > 1) I am using local filesystem - as you say that local running jobs
> > can not be stopped, does that imply that the scheduling is
> > dysfunctional as well on local filesystem
>
> The gui fully works in a local file system and a local jobrunner,
> except of stopping a running job.
> http://mail-archives.apache.org/mod_mbox/lucene-hadoop-dev/
> 200603.mbox/%[hidden email]%3E
>
>
> >
> > 2) Do you think it makes sense to have a language bundle in Dutch -
> > personally I don't, because I never meet a Dutch developer who doesn't
> > speak English, but I could do it anyway.
>
> Getting as much as possible would increase the use friendliness.
> However we may be need to test it and fix bugs before we start to
> translate more the ui in more languages.
>
>
> >
> > 3) What you like to operate as a filter for the bugs before we add
> > them to Jira, so that we don't post any known issues / duplicates.
>
> I created a component called administration gui, just attach bug
> reports under this component.
> If we all use the jira than we will know what are known problems.
>
> >
> > Thanks for all the work.
>
> Thanks for looking into it.
>
> Stefan
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Updated: (NUTCH-251) Administration GUI

Thomas Delnoij-3
Hi Stephan.

I am about to get started with the Admin GUI and was wondering if
these instructions are . still valid.

More in particular, is it still necessary to patch Hadoop, or has this
patch already been integrated?

Also do you know how the Nutch patches go with the latest revision?

Rgrds, Thomas Delnoij

How to:

+ checkout latest nutch sources

+ checkout hadoop sources
+ patch hadoop with the hadoop patch
+ build hadoop jar
+ remove old hadoop jar from nutch/lib
+ place new hadoop jar in nutch/lib


+ uncompress plugin zip file
+ place plugins in nutch/src/plugins (patch not possible since svn
does not support binary patches)
+ patch nutch with nutch patch
+ start gui with bin/nutch gui <folderWhereYourInstanceDataWillBeStored)
+ point your browser to: http://localhost:50060/general/
+ username and password are "admin". ( can be changed in nutch-default.xml)
+ select the "default" instance or create a new instance.
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Updated: (NUTCH-251) Administration GUI

Stefan Groschupf-2
Hi,

the easiest way is to download one of the binary distributions.
However as far I know the patches still work and need to be applied  
to both projects.
Stefan

Am 11.05.2006 um 08:38 schrieb TDLN:

> Hi Stephan.
>
> I am about to get started with the Admin GUI and was wondering if
> these instructions are . still valid.
>
> More in particular, is it still necessary to patch Hadoop, or has this
> patch already been integrated?
>
> Also do you know how the Nutch patches go with the latest revision?
>
> Rgrds, Thomas Delnoij
>
> How to:
>
> + checkout latest nutch sources
>
> + checkout hadoop sources
> + patch hadoop with the hadoop patch
> + build hadoop jar
> + remove old hadoop jar from nutch/lib
> + place new hadoop jar in nutch/lib
>
>
> + uncompress plugin zip file
> + place plugins in nutch/src/plugins (patch not possible since svn
> does not support binary patches)
> + patch nutch with nutch patch
> + start gui with bin/nutch gui  
> <folderWhereYourInstanceDataWillBeStored)
> + point your browser to: http://localhost:50060/general/
> + username and password are "admin". ( can be changed in nutch-
> default.xml)
> + select the "default" instance or create a new instance.
>

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Updated: (NUTCH-251) Administration GUI

Thomas Delnoij-3
I have my local changes, so I can't use the binary distribution.
Anyway, I will have a go at it and let you know.

Rgrds, Thomas



On 5/11/06, Stefan Groschupf <[hidden email]> wrote:

> Hi,
>
> the easiest way is to download one of the binary distributions.
> However as far I know the patches still work and need to be applied
> to both projects.
> Stefan
>
> Am 11.05.2006 um 08:38 schrieb TDLN:
>
> > Hi Stephan.
> >
> > I am about to get started with the Admin GUI and was wondering if
> > these instructions are . still valid.
> >
> > More in particular, is it still necessary to patch Hadoop, or has this
> > patch already been integrated?
> >
> > Also do you know how the Nutch patches go with the latest revision?
> >
> > Rgrds, Thomas Delnoij
> >
> > How to:
> >
> > + checkout latest nutch sources
> >
> > + checkout hadoop sources
> > + patch hadoop with the hadoop patch
> > + build hadoop jar
> > + remove old hadoop jar from nutch/lib
> > + place new hadoop jar in nutch/lib
> >
> >
> > + uncompress plugin zip file
> > + place plugins in nutch/src/plugins (patch not possible since svn
> > does not support binary patches)
> > + patch nutch with nutch patch
> > + start gui with bin/nutch gui
> > <folderWhereYourInstanceDataWillBeStored)
> > + point your browser to: http://localhost:50060/general/
> > + username and password are "admin". ( can be changed in nutch-
> > default.xml)
> > + select the "default" instance or create a new instance.
> >
>
>
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-251) Administration GUI

Hudson (Jira)
In reply to this post by Hudson (Jira)
    [ http://issues.apache.org/jira/browse/NUTCH-251?page=comments#action_12402297 ]

Thomas Delnoij commented on NUTCH-251:
--------------------------------------

I am getting compilation errors when I apply the patches and recompile Nutch. I think the Xalan Jars are missing. After I downloaded them, the admin plugins compile without errors. BTW: The jars are also missing from the binary distribution.

> Administration GUI
> ------------------
>
>          Key: NUTCH-251
>          URL: http://issues.apache.org/jira/browse/NUTCH-251
>      Project: Nutch
>         Type: Improvement

>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Minor
>      Fix For: 0.8-dev
>  Attachments: hadoop_nutch_gui_v1.patch, nutch_gui_plugins_v1.zip, nutch_gui_v1.patch
>
> Having a web based administration interface would help to make nutch administration and management much more user friendly.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-251) Administration GUI

Hudson (Jira)
In reply to this post by Hudson (Jira)
     [ http://issues.apache.org/jira/browse/NUTCH-251?page=all ]

Sami Siren updated NUTCH-251:
-----------------------------

    Fix Version/s: 0.9-dev
                       (was: 0.8-dev)

> Administration GUI
> ------------------
>
>                 Key: NUTCH-251
>                 URL: http://issues.apache.org/jira/browse/NUTCH-251
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 0.8-dev
>            Reporter: Stefan Groschupf
>            Priority: Minor
>             Fix For: 0.9-dev
>
>         Attachments: hadoop_nutch_gui_v1.patch, nutch_gui_plugins_v1.zip, nutch_gui_v1.patch
>
>
> Having a web based administration interface would help to make nutch administration and management much more user friendly.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira