document support for file system crawling

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

document support for file system crawling

Bruno-2
Hi there,

browsing through the message thread I tried to find a trail addressing file system crawls. I want to implement an enterprise search over a networked filesystem, crawling all sorts of documents, such as html, doc, ppt and pdf. Nutch provides plugins enabling it to read proprietary formats.
Is there support for the same functionality in solr?

Bruno
Reply | Threaded
Open this post in threaded view
|

Re: document support for file system crawling

Erik Hatcher

On Aug 30, 2006, at 2:42 AM, Bruno wrote:
> browsing through the message thread I tried to find a trail  
> addressing file
> system crawls. I want to implement an enterprise search over a  
> networked
> filesystem, crawling all sorts of documents, such as html, doc, ppt  
> and pdf.
> Nutch provides plugins enabling it to read proprietary formats.
> Is there support for the same functionality in solr?

No.  Solr is strictly a search server that takes plain text for the  
fields of documents added to it.  The client is responsible parsing  
the text out of these types of documents.  You could borrow the  
document parsing pieces from Lucene's contrib and Nutch and glue them  
together into your client that speaks to Solr, or perhaps Solr isn't  
the right approach for your needs?   It certainly is possible to add  
these capabilities into Solr, but it would be awkward to have to  
stream binary data into XML documents such that Solr could parse them  
on the server side.

        Erik


Reply | Threaded
Open this post in threaded view
|

Re: document support for file system crawling

Chris Hostetter-3

: the text out of these types of documents.  You could borrow the
: document parsing pieces from Lucene's contrib and Nutch and glue them
: together into your client that speaks to Solr, or perhaps Solr isn't
: the right approach for your needs?   It certainly is possible to add
: these capabilities into Solr, but it would be awkward to have to
: stream binary data into XML documents such that Solr could parse them
: on the server side.

Agreed.  Solr's focus is in indexing "Structured Data".  The support for
dynamic fields certainly allows you do deal with complex structured data,
and somewhat heterogeneous structured data -- but it's still structured
data.  If your goal is to do a lot of crawling of disparat physical
documents, extract the text, and build a "path,title,content" index
then Nutch is probably your best bet.


-Hoss