XmlInputFortmat ?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

XmlInputFortmat ?

Jack.Tang
Hi

I am going to feed nutch-0.8-dev crawler with seeds in xml format. And
I have read nutch TextInputFormat/InputFormatBase. It seems now nutch
breaks the plain text files into chars and parses on them. My question
is how to support XmlInputFormat, in my eye, xml format is not
character-based but blocke-based.

Thanks

/Jack

--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
Reply | Threaded
Open this post in threaded view
|

Re: XmlInputFortmat ?

Doug Cutting-2
Jack Tang wrote:
> I am going to feed nutch-0.8-dev crawler with seeds in xml format. And
> I have read nutch TextInputFormat/InputFormatBase. It seems now nutch
> breaks the plain text files into chars and parses on them. My question
> is how to support XmlInputFormat, in my eye, xml format is not
> character-based but blocke-based.

To make splitting efficient (since it is performed on the single master
node) TextInputFormat splits files by byte positions.  Then, when the
split is processed, on a node, input is synchronized to lines.  In
particular, the first line in the split is the first starting after the
start of the split (unless the split start position is zero, in which
case the first line in the split is the first line in the file) and the
last line in the split is the line containing the split's end position.

SequenceFiles are handled similarly.

XML would be hard to handle in this way, since one cannot seek to an
arbitrary position in an XML file and begin processing it.  So the
simplest way to implement an XML input format would be to disable file
splitting: each split should contain an entire file.

Doug