Does Solr support integration with the Compass framework?

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Does Solr support integration with the Compass framework?

Jochen Franke
Currently I'm investigating different Lucene based
search technologies.

For the indexing of our object model my favorite
is Compass because of the Object/Search Engine Mapping
capabilities.
At the same time Solr offers serveral nice features
like faceted search and caching.

Has anybody integrated or tried to integrate
Solr with Compass already and can share experiences.

Thanks,
   Jochen

Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

Yonik Seeley-2
One could do a very loose coupling by just pointing Solr at the index
created by Compass, and send a commit command to solr whenever you
want a new view of the index.

-Yonik

On 1/10/07, Jochen Franke <[hidden email]> wrote:

> Currently I'm investigating different Lucene based
> search technologies.
>
> For the indexing of our object model my favorite
> is Compass because of the Object/Search Engine Mapping
> capabilities.
> At the same time Solr offers serveral nice features
> like faceted search and caching.
>
> Has anybody integrated or tried to integrate
> Solr with Compass already and can share experiences.
>
> Thanks,
>    Jochen
Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

Graham O'Regan-2
doesn't compass use multiple indexes?

have a read of the "direct lucene" box on

http://www.opensymphony.com/compass/versions/1.1M3/html/introduction.html#i-use-lucene

would that prevent the two being used together? i'd be interested in
getting the two working together as well, it'd be great to have the
compass api to create the indexes and use solr to expose them over http.

Yonik Seeley wrote:

> One could do a very loose coupling by just pointing Solr at the index
> created by Compass, and send a commit command to solr whenever you
> want a new view of the index.
>
> -Yonik
>
> On 1/10/07, Jochen Franke <[hidden email]> wrote:
>> Currently I'm investigating different Lucene based
>> search technologies.
>>
>> For the indexing of our object model my favorite
>> is Compass because of the Object/Search Engine Mapping
>> capabilities.
>> At the same time Solr offers serveral nice features
>> like faceted search and caching.
>>
>> Has anybody integrated or tried to integrate
>> Solr with Compass already and can share experiences.
>>
>> Thanks,
>>    Jochen
>
Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

grahamoregan
In reply to this post by Yonik Seeley-2
doesn't compass use multiple indexes?

have a read of the "direct lucene" box on

http://www.opensymphony.com/compass/versions/1.1M3/html/introduction.html#i-use-lucene 


would that prevent the two being used together? i'd be interested in
getting the two working together as well, it'd be great to have the
compass api to create the indexes and use solr to expose them over http.

Yonik Seeley wrote:

> One could do a very loose coupling by just pointing Solr at the index
> created by Compass, and send a commit command to solr whenever you
> want a new view of the index.
>
> -Yonik
>
> On 1/10/07, Jochen Franke <[hidden email]> wrote:
>> Currently I'm investigating different Lucene based
>> search technologies.
>>
>> For the indexing of our object model my favorite
>> is Compass because of the Object/Search Engine Mapping
>> capabilities.
>> At the same time Solr offers serveral nice features
>> like faceted search and caching.
>>
>> Has anybody integrated or tried to integrate
>> Solr with Compass already and can share experiences.
>>
>> Thanks,
>>    Jochen
>

Reply | Threaded
Open this post in threaded view
|

solr boost

prosen2012
I am looking for documentation on how to boost a document when adding via
http post but am not finding any. Anyone have a pointer on this?

Thanks

Reply | Threaded
Open this post in threaded view
|

Re: solr boost

Yonik Seeley-2
On 1/12/07, Phil Rosen <[hidden email]> wrote:
> I am looking for documentation on how to boost a document when adding via
> http post but am not finding any. Anyone have a pointer on this?

http://wiki.apache.org/solr/UpdateXmlMessages

<add>
  <doc boost="2.5">
     [...]

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

Chris Hostetter-3
In reply to this post by grahamoregan

: doesn't compass use multiple indexes?

I don't relaly know anything about Compass, but if it's managing seperate
index partitions itself then yes it would certainly be hard to try and
point a Solr instance at it's index (since you'd just have ot pick one)

Solr was designed with teh assumption that it would *own* the index and be
bale to manage everything itself -- butthe master/slave model built in to
it makes it easy to configure a 'slave" whose "master" isn't actaully
another Solr index -- with the constraint that it still expects the index
to be a single FSDirectory.

The topic of using a MultiReader (whose primary usage is opening
multi-segment indexes) to search multiple standalone indexes in seperate
directories has come up on the lucene-java lists before ... i've never
tried it myself but if:
  a) it really is that easy
  b) there is a desire among the Solr user base for Solr to support
     searching of indexes not created by Solr (ie: Compass)

...then perhaps we could add two new options:
  1) a way to configure solr so that it knows it should treat the index as
read only (this might be as simple as not registering any UpdateHandlers)
  2) a hook to register a factory Solr would use when it wants/needs to
open a new IndexReader ... the default implimentation being the current
code that opens an FSDIrectory in ${dataDir}/index ... but another more
complex implimentation could be configured to take in a list of
directories and open a MultiReader across all of them. (or even an
implimentation that used a pure RAMDirectory



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

Ryan McKinley
In reply to this post by grahamoregan
>
> doesn't compass use multiple indexes?
>

yes, compass creates an index for each 'type' it handles.  I 'think'
you could point solr to it as long as you know that limitation.

I started using compass a few months back.  It is an amazing system:
with almost no effort, it just works.  BUT the showstopper (for me)
was that you could not easily update the index from multiple machines.
 Compass lets you put the lucene indexes in SQL with JDBC, but this
felt wrong....  then i found solr, and it solves most things.

I am working on a hibernate-solr bridge that will behave like the
compass Hibernate3GpsDevice.  It gets a callback from hibernate when
an object is stored, checks if it is 'SolrDocumentable' and sends it
to solr using the client library from:
  http://issues.apache.org/jira/browse/SOLR-20  (solr-client.zip)

If your interested, i can send you my initial version...  when i'm
further along, i'll try to post it to solr/client/java

ryan
Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

kkrugler
>>doesn't compass use multiple indexes?
>>
>
>yes, compass creates an index for each 'type' it handles.  I 'think'
>you could point solr to it as long as you know that limitation.
>
>I started using compass a few months back.  It is an amazing system:
>with almost no effort, it just works.  BUT the showstopper (for me)
>was that you could not easily update the index from multiple machines.
>Compass lets you put the lucene indexes in SQL with JDBC, but this
>felt wrong....  then i found solr, and it solves most things.
>
>I am working on a hibernate-solr bridge that will behave like the
>compass Hibernate3GpsDevice.  It gets a callback from hibernate when
>an object is stored, checks if it is 'SolrDocumentable' and sends it
>to solr using the client library from:
>  http://issues.apache.org/jira/browse/SOLR-20  (solr-client.zip)
>
>If your interested, i can send you my initial version...  when i'm
>further along, i'll try to post it to solr/client/java

That would be great - we're also facing the same issue of rolling our
own code to keep a Solr index in sync with a MySQL DB that we access
via Hibernate.

Thanks!

-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"
Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

Marios Skounakis-2
In reply to this post by Ryan McKinley
Hi all,

> >
> >I am working on a hibernate-solr bridge that will behave like the
> >compass Hibernate3GpsDevice.  It gets a callback from hibernate when
> >an object is stored, checks if it is 'SolrDocumentable' and sends it
> >to solr using the client library from:
> >  http://issues.apache.org/jira/browse/SOLR-20  (solr-client.zip)
> >
> >If your interested, i can send you my initial version...  when i'm
> >further along, i'll try to post it to solr/client/java
>
> That would be great - we're also facing the same issue of rolling our
> own code to keep a Solr index in sync with a MySQL DB that we access
> via Hibernate.

I wonder whether people who try to keep a Solr (or Lucene) index in
sync with a database are at all worried about index update failures.

Propagating the update from the DB to the index is one thing, and
relatively easy to implement. But how do you handle failures to update
either the index or the DB since you cannot enforce transactional
semantics over both updates? Or do index update failures occur so
infrequently that you do not worry about it?

Marios
Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

Graham O'Regan-2
compass provides a transaction manager for lucene indexes so you can
incorporate an index update and database update in a single transaction
or roll-back if either fails. thats why it would be interesting to see
the two working together.

Marios Skounakis wrote:

> Hi all,
>
>> >
>> >I am working on a hibernate-solr bridge that will behave like the
>> >compass Hibernate3GpsDevice.  It gets a callback from hibernate when
>> >an object is stored, checks if it is 'SolrDocumentable' and sends it
>> >to solr using the client library from:
>> >  http://issues.apache.org/jira/browse/SOLR-20  (solr-client.zip)
>> >
>> >If your interested, i can send you my initial version...  when i'm
>> >further along, i'll try to post it to solr/client/java
>>
>> That would be great - we're also facing the same issue of rolling our
>> own code to keep a Solr index in sync with a MySQL DB that we access
>> via Hibernate.
>
> I wonder whether people who try to keep a Solr (or Lucene) index in
> sync with a database are at all worried about index update failures.
>
> Propagating the update from the DB to the index is one thing, and
> relatively easy to implement. But how do you handle failures to update
> either the index or the DB since you cannot enforce transactional
> semantics over both updates? Or do index update failures occur so
> infrequently that you do not worry about it?
>
> Marios
>
Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

Marios Skounakis-2
Does compass store the lucene index in a database? If this is the
case, it is fairly straightforward to understand how this happens.

If the index is still in disk files how does it provide transactional
semantics? Would you care to give a high-level overview?

TIA
Marios

On 1/15/07, Graham O'Regan <[hidden email]> wrote:

> compass provides a transaction manager for lucene indexes so you can
> incorporate an index update and database update in a single transaction
> or roll-back if either fails. thats why it would be interesting to see
> the two working together.
>
> Marios Skounakis wrote:
> > Hi all,
> >
> >> >
> >> >I am working on a hibernate-solr bridge that will behave like the
> >> >compass Hibernate3GpsDevice.  It gets a callback from hibernate when
> >> >an object is stored, checks if it is 'SolrDocumentable' and sends it
> >> >to solr using the client library from:
> >> >  http://issues.apache.org/jira/browse/SOLR-20  (solr-client.zip)
> >> >
> >> >If your interested, i can send you my initial version...  when i'm
> >> >further along, i'll try to post it to solr/client/java
> >>
> >> That would be great - we're also facing the same issue of rolling our
> >> own code to keep a Solr index in sync with a MySQL DB that we access
> >> via Hibernate.
> >
> > I wonder whether people who try to keep a Solr (or Lucene) index in
> > sync with a database are at all worried about index update failures.
> >
> > Propagating the update from the DB to the index is one thing, and
> > relatively easy to implement. But how do you handle failures to update
> > either the index or the DB since you cannot enforce transactional
> > semantics over both updates? Or do index update failures occur so
> > infrequently that you do not worry about it?
> >
> > Marios
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

Lukáš Vlček
In reply to this post by Ryan McKinley
Ryan,

Could you be more specific on your statement?

On 1/12/07, Ryan McKinley <[hidden email]> wrote:
>
> I started using compass a few months back.  It is an amazing system:
> with almost no effort, it just works.  BUT the showstopper (for me)
> was that you could not easily update the index from multiple machines.
> Compass lets you put the lucene indexes in SQL with JDBC, but this
> felt wrong....  then i found solr, and it solves most things.
>


What exactly do you mean by "you could not easily update the index from
multiple machines"?

Could you describe you problem in more details (and possible workaround if
you found any) please?

Thanks,
Lukas
Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

Graham O'Regan-2
In reply to this post by Marios Skounakis-2
Hi Marios,

It can store the index in a database, but I wouldn't want to use that
route myself. Here is a quick link to the docs which provides an
over-view of the transactional features;

http://www.opensymphony.com/compass/versions/1.1M3/html/core-searchengine.html#core-searchengine-transaction

HTH,

Graham

Marios Skounakis wrote:

> Does compass store the lucene index in a database? If this is the
> case, it is fairly straightforward to understand how this happens.
>
> If the index is still in disk files how does it provide transactional
> semantics? Would you care to give a high-level overview?
>
> TIA
> Marios
>
> On 1/15/07, Graham O'Regan <[hidden email]> wrote:
>> compass provides a transaction manager for lucene indexes so you can
>> incorporate an index update and database update in a single transaction
>> or roll-back if either fails. thats why it would be interesting to see
>> the two working together.
>>
>> Marios Skounakis wrote:
>> > Hi all,
>> >
>> >> >
>> >> >I am working on a hibernate-solr bridge that will behave like the
>> >> >compass Hibernate3GpsDevice.  It gets a callback from hibernate when
>> >> >an object is stored, checks if it is 'SolrDocumentable' and sends it
>> >> >to solr using the client library from:
>> >> >  http://issues.apache.org/jira/browse/SOLR-20  (solr-client.zip)
>> >> >
>> >> >If your interested, i can send you my initial version...  when i'm
>> >> >further along, i'll try to post it to solr/client/java
>> >>
>> >> That would be great - we're also facing the same issue of rolling our
>> >> own code to keep a Solr index in sync with a MySQL DB that we access
>> >> via Hibernate.
>> >
>> > I wonder whether people who try to keep a Solr (or Lucene) index in
>> > sync with a database are at all worried about index update failures.
>> >
>> > Propagating the update from the DB to the index is one thing, and
>> > relatively easy to implement. But how do you handle failures to update
>> > either the index or the DB since you cannot enforce transactional
>> > semantics over both updates? Or do index update failures occur so
>> > infrequently that you do not worry about it?
>> >
>> > Marios
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Does Solr support integration with the Compass framework?

Ryan McKinley
In reply to this post by Lukáš Vlček
On 1/15/07, Lukas Vlcek <[hidden email]> wrote:

> Ryan,
>
> Could you be more specific on your statement?
>
> On 1/12/07, Ryan McKinley <[hidden email]> wrote:
> >
> > I started using compass a few months back.  It is an amazing system:
> > with almost no effort, it just works.  BUT the showstopper (for me)
> > was that you could not easily update the index from multiple machines.
> > Compass lets you put the lucene indexes in SQL with JDBC, but this
> > felt wrong....  then i found solr, and it solves most things.
> >
>
>
> What exactly do you mean by "you could not easily update the index from
> multiple machines"?
>

Consider a standard load balanced web setup with three machines:
DB1 - running mysql
WEB1 - webapp talking to DB1
WEB2 - webapp talking to DB1
...

In compass, the lucene index is stored on a disk - WEB1 writes its
lucene index on WEB1.  So for changes that WEB1 makes, WEB2 does not
see them (without adding more logic)

To solve this problem, compass is able to write its index into SQL.
WEB1 & WEB2 can write the lucene index in DB1.  But the performance is
not great and it seems to be something people discourage (tho i have
not tried it)

Also, take a look at:
http://forums.opensymphony.com/thread.jspa?messageID=100071

> Could you describe you problem in more details (and possible workaround if
> you found any) please?
>

workaround?  I'm now using solr :)

Otherwise, consider:

* Try the JDBC store:
http://www.opensymphony.com/compass/versions/1.1M3/html/core-connection.html#core-connection-jdbc

* If you are ok with WEB1 & WEB2 being slightly out of sync for new
content, you could us them normally and periodically call index() on
the hibernate GPS device.  This will synchronize whatever is stored in
hibernate with the lucene index