solr plug-in - external document store

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

solr plug-in - external document store

Ben Incani-2
Hi Solr Developers,

I would like to implement an external document storage system using
Solr.

Currently I am building lucene indexes with Solr and creating a binary
field, which is used to store a document.  I would like to extend this
implementation by storing the document in an external container system
(independent of Solr) using a simple file system mechanism or a more
robust solution like Jackrabbit.

Presently I am using Solr plug-ins to create additional field types,
which implement the binary storage system, but would like to know what
would be the best method to extend Solr to redirect insertions (add
requests) to an external storage system?

Regards,

Ben
Reply | Threaded
Open this post in threaded view
|

Re: solr plug-in - external document store

Ryan McKinley
You will either want to do something with a custom RequestHandler, or
plug into the UpdateRequestProcessor framework
  http://wiki.apache.org/solr/UpdateRequestProcessor

On the search/retrieval side, you will probably want to implement a
SearchComponent to fill in the stored fields:
  http://wiki.apache.org/solr/SearchComponent  (wiki page *way* out of date)

Both of these are 1.3-dev features....

ryan


Ben Incani wrote:

> Hi Solr Developers,
>
> I would like to implement an external document storage system using
> Solr.
>
> Currently I am building lucene indexes with Solr and creating a binary
> field, which is used to store a document.  I would like to extend this
> implementation by storing the document in an external container system
> (independent of Solr) using a simple file system mechanism or a more
> robust solution like Jackrabbit.
>
> Presently I am using Solr plug-ins to create additional field types,
> which implement the binary storage system, but would like to know what
> would be the best method to extend Solr to redirect insertions (add
> requests) to an external storage system?
>
> Regards,
>
> Ben
>

Reply | Threaded
Open this post in threaded view
|

RE: solr plug-in - external document store

Ben Incani-2
Thanks Ryan,

I have created an UpdateHandler that adds documents to a store, based on
a custom field type.

Now I would like to retrieve the document directly via a solr request,
should this be done via a custom queryResponseWriter?

-Ben


> -----Original Message-----
> From: Ryan McKinley [mailto:[hidden email]]
> Sent: Friday, 18 January 2008 12:37 PM
> To: [hidden email]
> Subject: Re: solr plug-in - external document store
>
> You will either want to do something with a custom
> RequestHandler, or plug into the UpdateRequestProcessor framework
>   http://wiki.apache.org/solr/UpdateRequestProcessor
>
> On the search/retrieval side, you will probably want to
> implement a SearchComponent to fill in the stored fields:
>   http://wiki.apache.org/solr/SearchComponent  (wiki page
> *way* out of date)
>
> Both of these are 1.3-dev features....
>
> ryan
>
>
> Ben Incani wrote:
> > Hi Solr Developers,
> >
> > I would like to implement an external document storage system using
> > Solr.
> >
> > Currently I am building lucene indexes with Solr and
> creating a binary
> > field, which is used to store a document.  I would like to
> extend this
> > implementation by storing the document in an external
> container system
> > (independent of Solr) using a simple file system mechanism
> or a more
> > robust solution like Jackrabbit.
> >
> > Presently I am using Solr plug-ins to create additional
> field types,
> > which implement the binary storage system, but would like
> to know what
> > would be the best method to extend Solr to redirect insertions (add
> > requests) to an external storage system?
> >
> > Regards,
> >
> > Ben
> >
>
>
Reply | Threaded
Open this post in threaded view
|

RE: solr plug-in - external document store

Ben Incani-2
ok, I now have a ExternalStoreRequestHandler that extends
RequestHandlerBase.

how do include extra fields in documents that get rendered by the
SolrQueryResponse?

do I need to recreate the DocList?
 

> -----Original Message-----
> From: Ben Incani [mailto:[hidden email]]
> Sent: Thursday, 24 January 2008 4:52 PM
> To: [hidden email]
> Subject: RE: solr plug-in - external document store
>
> Thanks Ryan,
>
> I have created an UpdateHandler that adds documents to a
> store, based on a custom field type.
>
> Now I would like to retrieve the document directly via a solr
> request, should this be done via a custom queryResponseWriter?
>
> -Ben
>
>
> > -----Original Message-----
> > From: Ryan McKinley [mailto:[hidden email]]
> > Sent: Friday, 18 January 2008 12:37 PM
> > To: [hidden email]
> > Subject: Re: solr plug-in - external document store
> >
> > You will either want to do something with a custom
> RequestHandler, or
> > plug into the UpdateRequestProcessor framework
> >   http://wiki.apache.org/solr/UpdateRequestProcessor
> >
> > On the search/retrieval side, you will probably want to implement a
> > SearchComponent to fill in the stored fields:
> >   http://wiki.apache.org/solr/SearchComponent  (wiki page
> > *way* out of date)
> >
> > Both of these are 1.3-dev features....
> >
> > ryan
> >
> >
> > Ben Incani wrote:
> > > Hi Solr Developers,
> > >
> > > I would like to implement an external document storage
> system using
> > > Solr.
> > >
> > > Currently I am building lucene indexes with Solr and
> > creating a binary
> > > field, which is used to store a document.  I would like to
> > extend this
> > > implementation by storing the document in an external
> > container system
> > > (independent of Solr) using a simple file system mechanism
> > or a more
> > > robust solution like Jackrabbit.
> > >
> > > Presently I am using Solr plug-ins to create additional
> > field types,
> > > which implement the binary storage system, but would like
> > to know what
> > > would be the best method to extend Solr to redirect
> insertions (add
> > > requests) to an external storage system?
> > >
> > > Regards,
> > >
> > > Ben
> > >
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

RE: solr plug-in - external document store

hossman

: how do include extra fields in documents that get rendered by the
: SolrQueryResponse?
:
: do I need to recreate the DocList?

i'm kind of out of hte loop on this, but i'm pretty sure there is an open
issue relating to trying to abstract this so that plugins can "mask" a
DocList to add external fields (and things like it) and then leverage any
ResponseWriter ... but i'm not sure off the top of my head which issue is
dealing with this (there's a lot of "big" issues going on right now that
all might relate: distributed search being the one i suspect is most
involved)

oh wait ... did you see this from Ryan earlier in the thread...

: > > On the search/retrieval side, you will probably want to implement a
: > > SearchComponent to fill in the stored fields:
: > >   http://wiki.apache.org/solr/SearchComponent  (wiki page
: > > *way* out of date)

...i think he's suggesting a Component that comes after the query which
knows how to fetch your external data and then just addds fields directly
to the Document objects (i'm not sure i'm a fan of that idea
... there's the documentCache to consider ... i suppose if you
*always* set the values you don't have to worry about old data in the
documentCache)



-Hoss