How to extract specified information from html?

classic Classic list List threaded Threaded
5 messages Options
jqq
Reply | Threaded
Open this post in threaded view
|

How to extract specified information from html?

jqq
Hi,
    I have seen the http://wiki.apache.org/nutch/WritingPluginExample, but I
don't understand clearly.
    I  need extract specified infromation  in specified web site in nucth.
   Firstly,I determine a URL set.
  Secondly,I determine that the current page URL was contained the URL set.
  Lastly,I extract infromation according to  regular expression and save it.

For example:a.html
   <span class="title">behavioral<font color=red>disease</font>(N76.8)
</span>
   extraction result:DiseaseName: behavioral disease,ID=N76.8

How should I do?

Thanks a lot.
jqq
Reply | Threaded
Open this post in threaded view
|

Re: How to extract specified information from html?

jqq
Should I implement HtmlParseFilter? If it is,How to invoke my method in
filter() of  HtmlParseFilter?

Thanks.


2007/10/31, zhao xiuwen <[hidden email]>:

>
> Hi,
>     I have seen the http://wiki.apache.org/nutch/WritingPluginExample, but
> I don't understand clearly.
>     I  need extract specified infromation  in specified web site in nucth.
>    Firstly,I determine a URL set.
>   Secondly,I determine that the current page URL was contained the URL
> set.
>   Lastly,I extract infromation according to  regular expression and
> save it.
>
> For example:a.html
>    <span class="title">behavioral<font color=red>disease</font>(N76.8)
> </span>
>    extraction result:DiseaseName: behavioral disease,ID=N76.8
>
> How should I do?
>
> Thanks a lot.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: How to extract specified information from html?

Adam Lofts
Hi,

On 31/10/2007, zhao xiuwen <[hidden email]> wrote:
>
> Should I implement HtmlParseFilter?


Yes

If it is,How to invoke my method in
> filter() of  HtmlParseFilter?


Load your plugin in the nutch config and filter() will be called for every
html file that you crawl.

Best,
Adam
Reply | Threaded
Open this post in threaded view
|

Re: How to extract specified information from html?

wuqi-2
In reply to this post by jqq
Try to take a look at HtmlParser.java in parse-html plugin...You can develop your own HtmlParser by modifying the implementation of  function

public Parse getParse(Content content) {
 Step1: get HTML sourcecode through content.
  String htmlCode= content.toString( );

Step2:  Check the Html Source code one by one with a Regular Expression to find the structured data you want..

Step3: Keep the data extracted ,to database or anyting elses;
 

}

----- Original Message -----
From: "zhao xiuwen" <[hidden email]>
To: <[hidden email]>
Sent: Thursday, November 01, 2007 12:12 AM
Subject: Re: How to extract specified information from html?


> Should I implement HtmlParseFilter? If it is,How to invoke my method in
> filter() of  HtmlParseFilter?
>
> Thanks.
>
>
> 2007/10/31, zhao xiuwen <[hidden email]>:
>>
>> Hi,
>>     I have seen the http://wiki.apache.org/nutch/WritingPluginExample, but
>> I don't understand clearly.
>>     I  need extract specified infromation  in specified web site in nucth.
>>    Firstly,I determine a URL set.
>>   Secondly,I determine that the current page URL was contained the URL
>> set.
>>   Lastly,I extract infromation according to  regular expression and
>> save it.
>>
>> For example:a.html
>>    <span class="title">behavioral<font color=red>disease</font>(N76.8)
>> </span>
>>    extraction result:DiseaseName: behavioral disease,ID=N76.8
>>
>> How should I do?
>>
>> Thanks a lot.
>>
>>
>
jqq
Reply | Threaded
Open this post in threaded view
|

Re: How to extract specified information from html?

jqq
Thanks.

2007/11/3, qi wu <[hidden email]>:

>
> Try to take a look at HtmlParser.java in parse-html plugin...You can
> develop your own HtmlParser by modifying the implementation of  function
>
> public Parse getParse(Content content) {
> Step1: get HTML sourcecode through content.
> String htmlCode= content.toString( );
>
> Step2:  Check the Html Source code one by one with a Regular Expression to
> find the structured data you want..
>
> Step3: Keep the data extracted ,to database or anyting elses;
>
>
> }
>
> ----- Original Message -----
> From: "zhao xiuwen" <[hidden email]>
> To: <[hidden email]>
> Sent: Thursday, November 01, 2007 12:12 AM
> Subject: Re: How to extract specified information from html?
>
>
> > Should I implement HtmlParseFilter? If it is,How to invoke my method in
> > filter() of  HtmlParseFilter?
> >
> > Thanks.
> >
> >
> > 2007/10/31, zhao xiuwen <[hidden email]>:
> >>
> >> Hi,
> >>     I have seen the http://wiki.apache.org/nutch/WritingPluginExample,
> but
> >> I don't understand clearly.
> >>     I  need extract specified infromation  in specified web site in
> nucth.
> >>    Firstly,I determine a URL set.
> >>   Secondly,I determine that the current page URL was contained the URL
> >> set.
> >>   Lastly,I extract infromation according to  regular expression and
> >> save it.
> >>
> >> For example:a.html
> >>    <span class="title">behavioral<font color=red>disease</font>(N76.8)
> >> </span>
> >>    extraction result:DiseaseName: behavioral disease,ID=N76.8
> >>
> >> How should I do?
> >>
> >> Thanks a lot.
> >>
> >>
> >