Database Import RequestHandler

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Database Import RequestHandler

Noble Paul നോബിള്‍  नोब्ळ्
hi,
We have a project where we need to index a part of our data in the database
in Solr . We are  writing a RequestHandler in Solr which can do this . Think
of it as an advanced form of SqlUpload Plugin (SOLR-103).

The way it works is as follows.

   - Provide a configuration file (xml) to the Handler which takes in the
   necessary SQL queries and mappings to a solr schema
      - It also takes in a properties file for the data source
      configuraution
      - Given the configuration it can also generate the solr
   schema.xml
   - It is registered as a RequestHandler which can take two commands
   do-full-import, do-delta-import
      -  do-full-import - dumps all the data from the Database into
      the index (based on the SQL query in configuration)
      - do-delta-import - dumps all the data that has changed since
      last import. (We assume a modified-timestamp column in tables)
   - It provides a admin page
   - where we can schedule it to be run automatically at regular
      intervals
      - It shows the status of the Handler (idle, full-import,
      delta-import)

We wish to know if this can be contributed back to SOLR because we assume
this is a very common use case.
--Noble
Reply | Threaded
Open this post in threaded view
|

Re: Database Import RequestHandler

Amit Sudharshan
Greetings

We are working on something very similar here. It would be great to collaborate on it with you.

-Amit
Sent via BlackBerry from T-Mobile

-----Original Message-----
From: "Noble Paul   " <[hidden email]>

Date: Thu, 31 Jan 2008 12:32:06
To:[hidden email]
Subject: Database Import RequestHandler


hi,
We have a project where we need to index a part of our data in the database
in Solr . We are  writing a RequestHandler in Solr which can do this . Think
of it as an advanced form of SqlUpload Plugin (SOLR-103).

The way it works is as follows.

   - Provide a configuration file (xml) to the Handler which takes in the
   necessary SQL queries and mappings to a solr schema
      - It also takes in a properties file for the data source
      configuraution
      - Given the configuration it can also generate the solr
   schema.xml
   - It is registered as a RequestHandler which can take two commands
   do-full-import, do-delta-import
      -  do-full-import - dumps all the data from the Database into
      the index (based on the SQL query in configuration)
      - do-delta-import - dumps all the data that has changed since
      last import. (We assume a modified-timestamp column in tables)
   - It provides a admin page
   - where we can schedule it to be run automatically at regular
      intervals
      - It shows the status of the Handler (idle, full-import,
      delta-import)

We wish to know if this can be contributed back to SOLR because we assume
this is a very common use case.
--Noble

Reply | Threaded
Open this post in threaded view
|

Re: Database Import RequestHandler

Ryan McKinley
In reply to this post by Noble Paul നോബിള്‍ नोब्ळ्
>
> We wish to know if this can be contributed back to SOLR because we assume
> this is a very common use case.
> --Noble
>

YES! of course.  At a minimum, it can reside in a JIRA issue while the
(often tedius) details get sorted.  Having something in JIRA -- even
incomplete -- helps others substantially.

ryan