Nutch-0.9 plugins, trouble with ant 1.6.5 and 1.7

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Nutch-0.9 plugins, trouble with ant 1.6.5 and 1.7

mbennett
Problems related to compiling:

1: Compiled plugin jars and files are not making it to nutch-0.9/plugin

2: ant package gives errors with both ant 1.6.5 and ant 1.7

3: even Apache example doesn't compile under nutch-0.9

 

Not sure if item 2 is the cause of item 1 or not, but was trying to fix it
just in case.  Problem 1 is more serious.

 

Problem 1:

 

When I use ant to compile plugins, they never make it from the build
directory.  I'm running ant with the default target in the main nutch-0.9
directory.

 

I did add my job to src/plugin/build.xml  And I do see files show up in
build.  They just never make it to the main plugin dir structure.

 

As another test, I've even tried temporarily removing a core plugin and ant
doesn't rebuild that either.

 

Attempted fix:

I wondered if maybe the default ant target wasn't supposed to create the
dirs and jars in the main plugin directory structure, and that maybe I
needed to run ant package.

 

I wondered if trying "ant package" would force it to migrate the files from
/build/ to /plugin/.  (is that correct??)

 

That led to problem 2, when I run "ant package" from the main dir I get
errors.  (I'm not sure I this was the correct fix or not..)

 

Problem 2:

I'm getting (different) errors with both ant 1.6.5 and 1.7.  I'm not sure if
either errors are related to the main problem.

 

Ant 1.6.5

ant package

BUILD FAILED

/home/mbennett/build_nutch/build.xml:442: The following error occurred while
executing this line:

/home/mbennett/build_nutch/build.xml:408: java.lang.ClassNotFoundException:
org.apache.tools.ant.taskdefs.optional.TraXLiaison

 

I saw comments that yum wasn't installing ant 1.7, so I did a manual install
of that.

 

Ant 1.7

ant package

BUILD FAILED

/home/mbennett/build_nutch/build.xml:61: Specify at least one source--a file
or resource collection.

 

build.xml line 61 has:

    <touch datetime="01/25/1971 2:00 pm">

      <fileset dir="${conf.dir}" includes="**/*.template"/>

    </touch>

I'm not an ant expert, but ant 1.6.5 didn't complain about this particular
line.

 

Problem 3:

This example doesn't compile, lots of changes needed.

http://wiki.apache.org/nutch/WritingPluginExample

It seems to be related to logging classes at a minimum.

I got part way through it.

 

The boost-urlpattern class seems way out of date too.  That's the one I've
been trying to port to 0.9

http://issues.apache.org/jira/browse/NUTCH-16

In addition to logging, it appears that the initialization of extensions has
changed, and how they find their current directory (to read config files
from, etc).

 

Stefan's Plugins doc on the Wiki also seems to be missing.

On this page:

http://wiki.apache.org/nutch/PluginCentral

The link to "Writing Plugins - by Stefan" goes to this URL:

http://wiki.media-style.com/display/nutchDocu/Write+a+plugin

 

I'm not whining, it's just that I was trying to find some working example in
nutch-0.9.

 

I'm really stuck trying to get any extension to work in 0.9  :-(

 

I would really appreciate any advice on any of this, or talking to anybody
who's been able to compile extensions in nutch-0.9.  I'm sure I'm doing
something "dumb".

Mark

 

 

 

 

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Nutch-0.9 plugins, trouble with ant 1.6.5 and 1.7

Tranquil
Hi,

try looking at me post titled :
"writing a new parse-exe plugin"

i've written a checklist there for writing a parse plugin that eventualy
worked for me.

also, ANT: i had a lot of problems with it... best thing to do is (i run
fedora):
1. uninstall current ver
2. download tar.gz of 1.6.5 f
3. install it (it's binary so just put it in /usr/share or somewhere)
4. update ANT_HOME in your bashrc (for the dir u put the ant in)
5. add ANT/BIN to PATH.

after that ANT should work properly, any problem compiling is not ant
related.
another thing to check is JAVA version, which in most cases is the problem.

install JAVA 1.6.2

http://blog.taragana.com/index.php/archive/how-to-install-jdk-6-java-se-6-tomcat-in-fedora-core-6-fedora-7-in-5-minutes/

hope his helps...

Eyal.



On Nov 10, 2007 3:29 AM, Mark Bennett <[hidden email]> wrote:

> Problems related to compiling:
>
> 1: Compiled plugin jars and files are not making it to nutch-0.9/plugin
>
> 2: ant package gives errors with both ant 1.6.5 and ant 1.7
>
> 3: even Apache example doesn't compile under nutch-0.9
>
>
>
> Not sure if item 2 is the cause of item 1 or not, but was trying to fix it
> just in case.  Problem 1 is more serious.
>
>
>
> Problem 1:
>
>
>
> When I use ant to compile plugins, they never make it from the build
> directory.  I'm running ant with the default target in the main nutch-0.9
> directory.
>
>
>
> I did add my job to src/plugin/build.xml  And I do see files show up in
> build.  They just never make it to the main plugin dir structure.
>
>
>
> As another test, I've even tried temporarily removing a core plugin and
> ant
> doesn't rebuild that either.
>
>
>
> Attempted fix:
>
> I wondered if maybe the default ant target wasn't supposed to create the
> dirs and jars in the main plugin directory structure, and that maybe I
> needed to run ant package.
>
>
>
> I wondered if trying "ant package" would force it to migrate the files
> from
> /build/ to /plugin/.  (is that correct??)
>
>
>
> That led to problem 2, when I run "ant package" from the main dir I get
> errors.  (I'm not sure I this was the correct fix or not..)
>
>
>
> Problem 2:
>
> I'm getting (different) errors with both ant 1.6.5 and 1.7.  I'm not sure
> if
> either errors are related to the main problem.
>
>
>
> Ant 1.6.5
>
> ant package
>
> BUILD FAILED
>
> /home/mbennett/build_nutch/build.xml:442: The following error occurred
> while
> executing this line:
>
> /home/mbennett/build_nutch/build.xml:408: java.lang.ClassNotFoundException
> :
> org.apache.tools.ant.taskdefs.optional.TraXLiaison
>
>
>
> I saw comments that yum wasn't installing ant 1.7, so I did a manual
> install
> of that.
>
>
>
> Ant 1.7
>
> ant package
>
> BUILD FAILED
>
> /home/mbennett/build_nutch/build.xml:61: Specify at least one source--a
> file
> or resource collection.
>
>
>
> build.xml line 61 has:
>
>    <touch datetime="01/25/1971 2:00 pm">
>
>      <fileset dir="${conf.dir}" includes="**/*.template"/>
>
>    </touch>
>
> I'm not an ant expert, but ant 1.6.5 didn't complain about this particular
> line.
>
>
>
> Problem 3:
>
> This example doesn't compile, lots of changes needed.
>
> http://wiki.apache.org/nutch/WritingPluginExample
>
> It seems to be related to logging classes at a minimum.
>
> I got part way through it.
>
>
>
> The boost-urlpattern class seems way out of date too.  That's the one I've
> been trying to port to 0.9
>
> http://issues.apache.org/jira/browse/NUTCH-16
>
> In addition to logging, it appears that the initialization of extensions
> has
> changed, and how they find their current directory (to read config files
> from, etc).
>
>
>
> Stefan's Plugins doc on the Wiki also seems to be missing.
>
> On this page:
>
> http://wiki.apache.org/nutch/PluginCentral
>
> The link to "Writing Plugins - by Stefan" goes to this URL:
>
> http://wiki.media-style.com/display/nutchDocu/Write+a+plugin
>
>
>
> I'm not whining, it's just that I was trying to find some working example
> in
> nutch-0.9.
>
>
>
> I'm really stuck trying to get any extension to work in 0.9  :-(
>
>
>
> I would really appreciate any advice on any of this, or talking to anybody
> who's been able to compile extensions in nutch-0.9.  I'm sure I'm doing
> something "dumb".
>
> Mark
>
>
>
>
>
>
>
>
>
>
>
>
>
>


--
Eyal Edri
Reply | Threaded
Open this post in threaded view
|

RE: Nutch-0.9 plugins, trouble with ant 1.6.5 and 1.7

mbennett
Hello,

Thank you!  Very good reading.

I notice in your post you say:
>    8. copied $NUTCH_HOME/build/plugins/parse-exe/parse-exe.jar to
>   $NUTCH_HOME/plugins/parse-exe

So ant is not intended to automatically copy compiled plugins from build to
their final destination?

I've had to do this manually as well, but I assumed it was something that
was "broken", not "by design" ?

With ant doing so many other things, compiling, building wars, etc, I had
assumed it would also migrate the files to their proper destination in
NUTCH_HOME/plugins

-----Original Message-----
From: eyal edri [mailto:[hidden email]]
Sent: Saturday, November 10, 2007 5:09 AM
To: [hidden email]
Subject: Re: Nutch-0.9 plugins, trouble with ant 1.6.5 and 1.7

Hi,

try looking at me post titled :
"writing a new parse-exe plugin"

i've written a checklist there for writing a parse plugin that eventualy
worked for me.

also, ANT: i had a lot of problems with it... best thing to do is (i run
fedora):
1. uninstall current ver
2. download tar.gz of 1.6.5 f
3. install it (it's binary so just put it in /usr/share or somewhere)
4. update ANT_HOME in your bashrc (for the dir u put the ant in)
5. add ANT/BIN to PATH.

after that ANT should work properly, any problem compiling is not ant
related.
another thing to check is JAVA version, which in most cases is the problem.

install JAVA 1.6.2

http://blog.taragana.com/index.php/archive/how-to-install-jdk-6-java-se-6-to
mcat-in-fedora-core-6-fedora-7-in-5-minutes/

hope his helps...

Eyal.



On Nov 10, 2007 3:29 AM, Mark Bennett <[hidden email]> wrote:

> Problems related to compiling:
>
> 1: Compiled plugin jars and files are not making it to nutch-0.9/plugin
>
> 2: ant package gives errors with both ant 1.6.5 and ant 1.7
>
> 3: even Apache example doesn't compile under nutch-0.9
>
>
>
> Not sure if item 2 is the cause of item 1 or not, but was trying to fix it
> just in case.  Problem 1 is more serious.
>
>
>
> Problem 1:
>
>
>
> When I use ant to compile plugins, they never make it from the build
> directory.  I'm running ant with the default target in the main nutch-0.9
> directory.
>
>
>
> I did add my job to src/plugin/build.xml  And I do see files show up in
> build.  They just never make it to the main plugin dir structure.
>
>
>
> As another test, I've even tried temporarily removing a core plugin and
> ant
> doesn't rebuild that either.
>
>
>
> Attempted fix:
>
> I wondered if maybe the default ant target wasn't supposed to create the
> dirs and jars in the main plugin directory structure, and that maybe I
> needed to run ant package.
>
>
>
> I wondered if trying "ant package" would force it to migrate the files
> from
> /build/ to /plugin/.  (is that correct??)
>
>
>
> That led to problem 2, when I run "ant package" from the main dir I get
> errors.  (I'm not sure I this was the correct fix or not..)
>
>
>
> Problem 2:
>
> I'm getting (different) errors with both ant 1.6.5 and 1.7.  I'm not sure
> if
> either errors are related to the main problem.
>
>
>
> Ant 1.6.5
>
> ant package
>
> BUILD FAILED
>
> /home/mbennett/build_nutch/build.xml:442: The following error occurred
> while
> executing this line:
>
> /home/mbennett/build_nutch/build.xml:408: java.lang.ClassNotFoundException
> :
> org.apache.tools.ant.taskdefs.optional.TraXLiaison
>
>
>
> I saw comments that yum wasn't installing ant 1.7, so I did a manual
> install
> of that.
>
>
>
> Ant 1.7
>
> ant package
>
> BUILD FAILED
>
> /home/mbennett/build_nutch/build.xml:61: Specify at least one source--a
> file
> or resource collection.
>
>
>
> build.xml line 61 has:
>
>    <touch datetime="01/25/1971 2:00 pm">
>
>      <fileset dir="${conf.dir}" includes="**/*.template"/>
>
>    </touch>
>
> I'm not an ant expert, but ant 1.6.5 didn't complain about this particular
> line.
>
>
>
> Problem 3:
>
> This example doesn't compile, lots of changes needed.
>
> http://wiki.apache.org/nutch/WritingPluginExample
>
> It seems to be related to logging classes at a minimum.
>
> I got part way through it.
>
>
>
> The boost-urlpattern class seems way out of date too.  That's the one I've
> been trying to port to 0.9
>
> http://issues.apache.org/jira/browse/NUTCH-16
>
> In addition to logging, it appears that the initialization of extensions
> has
> changed, and how they find their current directory (to read config files
> from, etc).
>
>
>
> Stefan's Plugins doc on the Wiki also seems to be missing.
>
> On this page:
>
> http://wiki.apache.org/nutch/PluginCentral
>
> The link to "Writing Plugins - by Stefan" goes to this URL:
>
> http://wiki.media-style.com/display/nutchDocu/Write+a+plugin
>
>
>
> I'm not whining, it's just that I was trying to find some working example
> in
> nutch-0.9.
>
>
>
> I'm really stuck trying to get any extension to work in 0.9  :-(
>
>
>
> I would really appreciate any advice on any of this, or talking to anybody
> who's been able to compile extensions in nutch-0.9.  I'm sure I'm doing
> something "dumb".
>
> Mark
>
>
>
>
>
>
>
>
>
>
>
>
>
>


--
Eyal Edri

Reply | Threaded
Open this post in threaded view
|

Re: Nutch-0.9 plugins, trouble with ant 1.6.5 and 1.7

Tranquil
I can't agree with you more!

I raised a query on the subject but no one replied...

mybe now someone will raise the gluve and give an answer to it...

the most wierd thing is, i've written a new class under the java/utils lib
(where LogUtil.java is) and ant knows to complie it properly without the
need of coping anyting...

it's just the plugins that has this issue..

i'm sure there is a setting in build.xml that can handle this... we just
need to find it

anyone??



On Nov 10, 2007 7:36 PM, Mark Bennett <[hidden email]> wrote:

> Hello,
>
> Thank you!  Very good reading.
>
> I notice in your post you say:
> >    8. copied $NUTCH_HOME/build/plugins/parse-exe/parse-exe.jar to
> >   $NUTCH_HOME/plugins/parse-exe
>
> So ant is not intended to automatically copy compiled plugins from build
> to
> their final destination?
>
> I've had to do this manually as well, but I assumed it was something that
> was "broken", not "by design" ?
>
> With ant doing so many other things, compiling, building wars, etc, I had
> assumed it would also migrate the files to their proper destination in
> NUTCH_HOME/plugins
>
> -----Original Message-----
> From: eyal edri [mailto:[hidden email]]
> Sent: Saturday, November 10, 2007 5:09 AM
> To: [hidden email]
> Subject: Re: Nutch-0.9 plugins, trouble with ant 1.6.5 and 1.7
>
> Hi,
>
> try looking at me post titled :
> "writing a new parse-exe plugin"
>
> i've written a checklist there for writing a parse plugin that eventualy
> worked for me.
>
> also, ANT: i had a lot of problems with it... best thing to do is (i run
> fedora):
> 1. uninstall current ver
> 2. download tar.gz of 1.6.5 f
> 3. install it (it's binary so just put it in /usr/share or somewhere)
> 4. update ANT_HOME in your bashrc (for the dir u put the ant in)
> 5. add ANT/BIN to PATH.
>
> after that ANT should work properly, any problem compiling is not ant
> related.
> another thing to check is JAVA version, which in most cases is the
> problem.
>
> install JAVA 1.6.2
>
>
> http://blog.taragana.com/index.php/archive/how-to-install-jdk-6-java-se-6-to
> mcat-in-fedora-core-6-fedora-7-in-5-minutes/
>
> hope his helps...
>
> Eyal.
>
>
>
> On Nov 10, 2007 3:29 AM, Mark Bennett <[hidden email]> wrote:
>
> > Problems related to compiling:
> >
> > 1: Compiled plugin jars and files are not making it to nutch-0.9/plugin
> >
> > 2: ant package gives errors with both ant 1.6.5 and ant 1.7
> >
> > 3: even Apache example doesn't compile under nutch-0.9
> >
> >
> >
> > Not sure if item 2 is the cause of item 1 or not, but was trying to fix
> it
> > just in case.  Problem 1 is more serious.
> >
> >
> >
> > Problem 1:
> >
> >
> >
> > When I use ant to compile plugins, they never make it from the build
> > directory.  I'm running ant with the default target in the main
> nutch-0.9
> > directory.
> >
> >
> >
> > I did add my job to src/plugin/build.xml  And I do see files show up in
> > build.  They just never make it to the main plugin dir structure.
> >
> >
> >
> > As another test, I've even tried temporarily removing a core plugin and
> > ant
> > doesn't rebuild that either.
> >
> >
> >
> > Attempted fix:
> >
> > I wondered if maybe the default ant target wasn't supposed to create the
> > dirs and jars in the main plugin directory structure, and that maybe I
> > needed to run ant package.
> >
> >
> >
> > I wondered if trying "ant package" would force it to migrate the files
> > from
> > /build/ to /plugin/.  (is that correct??)
> >
> >
> >
> > That led to problem 2, when I run "ant package" from the main dir I get
> > errors.  (I'm not sure I this was the correct fix or not..)
> >
> >
> >
> > Problem 2:
> >
> > I'm getting (different) errors with both ant 1.6.5 and 1.7.  I'm not
> sure
> > if
> > either errors are related to the main problem.
> >
> >
> >
> > Ant 1.6.5
> >
> > ant package
> >
> > BUILD FAILED
> >
> > /home/mbennett/build_nutch/build.xml:442: The following error occurred
> > while
> > executing this line:
> >
> > /home/mbennett/build_nutch/build.xml:408:
> java.lang.ClassNotFoundException
> > :
> > org.apache.tools.ant.taskdefs.optional.TraXLiaison
> >
> >
> >
> > I saw comments that yum wasn't installing ant 1.7, so I did a manual
> > install
> > of that.
> >
> >
> >
> > Ant 1.7
> >
> > ant package
> >
> > BUILD FAILED
> >
> > /home/mbennett/build_nutch/build.xml:61: Specify at least one source--a
> > file
> > or resource collection.
> >
> >
> >
> > build.xml line 61 has:
> >
> >    <touch datetime="01/25/1971 2:00 pm">
> >
> >      <fileset dir="${conf.dir}" includes="**/*.template"/>
> >
> >    </touch>
> >
> > I'm not an ant expert, but ant 1.6.5 didn't complain about this
> particular
> > line.
> >
> >
> >
> > Problem 3:
> >
> > This example doesn't compile, lots of changes needed.
> >
> > http://wiki.apache.org/nutch/WritingPluginExample
> >
> > It seems to be related to logging classes at a minimum.
> >
> > I got part way through it.
> >
> >
> >
> > The boost-urlpattern class seems way out of date too.  That's the one
> I've
> > been trying to port to 0.9
> >
> > http://issues.apache.org/jira/browse/NUTCH-16
> >
> > In addition to logging, it appears that the initialization of extensions
> > has
> > changed, and how they find their current directory (to read config files
> > from, etc).
> >
> >
> >
> > Stefan's Plugins doc on the Wiki also seems to be missing.
> >
> > On this page:
> >
> > http://wiki.apache.org/nutch/PluginCentral
> >
> > The link to "Writing Plugins - by Stefan" goes to this URL:
> >
> > http://wiki.media-style.com/display/nutchDocu/Write+a+plugin
> >
> >
> >
> > I'm not whining, it's just that I was trying to find some working
> example
> > in
> > nutch-0.9.
> >
> >
> >
> > I'm really stuck trying to get any extension to work in 0.9  :-(
> >
> >
> >
> > I would really appreciate any advice on any of this, or talking to
> anybody
> > who's been able to compile extensions in nutch-0.9.  I'm sure I'm doing
> > something "dumb".
> >
> > Mark
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
> --
> Eyal Edri
>
>


--
Eyal Edri