Thank you...

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Thank you...

Cass Costello
...to everyone who helps make Lucene and Solr such fantastic tools.

I'm the Platform Architect for a leading online event ticket
after-marketplace (think eBay for tickets), and we've just completed a 12
month project to rewrite the Browse and Search components of our
customer-facing site.  Both rely heavily on Solr and a smattering of custom
Lucene API calls throughout the code base.  As of this morning (2 weeks
after our final-stage push to production), we've reduced our hardware needs
over 50% and have improved page response times by 30-60% as compared to the
old architecture (a mix of very old CF code for the browse pieces, and a
couple GSAs for the seach pages).  More importantly, we've established new
patterns and flex points for future growth, and I have utmost confidence
that we'll be leveraging the Lucene/Solr combo for years to come as we grow.
The devs are proud, the execs are happy, our customers have noticed the
speed improvements, and we all learned many positive lessons along the way.

These forums and various online resources (the Solr wiki first and foremost)
have been tremendously helpful.   And so, thank you, all, for pointing the
way to a simpler, faster, better solution for our needs.

Cass Costello

--
A complex system that works is invariably found to have evolved from a
simple system that works.
  - John Gaule
Reply | Threaded
Open this post in threaded view
|

Re: Thank you...

Cass Costello
Heh - it used to be in my sig ... my bad.

Thanks, all. :)

http://www.stubhub.com


On 3/20/07, bruce <[hidden email]> wrote:

>
> hey cass...
>
> anyway you could let us know the site/app that we're powering!!!
>
> always good to see what's going on in the world!
>
> thanks
>
>
> -----Original Message-----
> From: Cass Costello [mailto:[hidden email]]
> Sent: Tuesday, March 20, 2007 12:58 PM
> To: [hidden email]; [hidden email]
> Subject: Thank you...
>
>
> ...to everyone who helps make Lucene and Solr such fantastic tools.
>
> I'm the Platform Architect for a leading online event ticket
> after-marketplace (think eBay for tickets), and we've just completed a 12
> month project to rewrite the Browse and Search components of our
> customer-facing site.  Both rely heavily on Solr and a smattering of
> custom
> Lucene API calls throughout the code base.  As of this morning (2 weeks
> after our final-stage push to production), we've reduced our hardware
> needs
> over 50% and have improved page response times by 30-60% as compared to
> the
> old architecture (a mix of very old CF code for the browse pieces, and a
> couple GSAs for the seach pages).  More importantly, we've established new
> patterns and flex points for future growth, and I have utmost confidence
> that we'll be leveraging the Lucene/Solr combo for years to come as we
> grow.
> The devs are proud, the execs are happy, our customers have noticed the
> speed improvements, and we all learned many positive lessons along the
> way.
>
> These forums and various online resources (the Solr wiki first and
> foremost)
> have been tremendously helpful.   And so, thank you, all, for pointing the
> way to a simpler, faster, better solution for our needs.
>
> Cass Costello
>
> --
> A complex system that works is invariably found to have evolved from a
> simple system that works.
>   - John Gaule
>
>


--
A complex system that works is invariably found to have evolved from a
simple system that works.
  - John Gaule
Reply | Threaded
Open this post in threaded view
|

Re: Thank you...

Yonik Seeley-2
On 3/20/07, Cass Costello <[hidden email]> wrote:
> Heh - it used to be in my sig ... my bad.
>
> Thanks, all. :)
>
> http://www.stubhub.com

I tried it out... looks great!
Is the faceting stuff done with Solr too?
If so is it the Solr built-in faceting, or a custom query handler?  If
the latter, can you think of ways to improve the built in faceting
that would have made things easier?

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Thank you...

Cass Costello
You all rock.  I'm clearing the semi-official legal hurdle with my CTO and
our head counsel to full (or something close to full) disclosure of some of
the architectural details, so stay tuned for as much as I'm allowed to share
(and btw, for any of you that live/work/vacation in the SF Bay area, I'd be
happy to discuss things over a beer :)

In the meantime, yes, much of the faceting you see on the Browse and Search
pages is direct Solr faceting.  We made *very* liberal use of
faceting-specific fields in our event and ontology documents.

Similarly, we rely on the standard query handler for almost all the Solr
calls.  Our business logic is modelled in a CoR pattern, and we did write
one command to encapsulate SolrCore/Request/Response use for our highest
throughput needs - the win there was direct building into our predominant
xml object model (we use XOM) via a QueryResponseXOMBuilder.build(
solrRequest, solrResponse ) call.

I definitely need clearance before sharing hard perf numbers (anyone working
for TicketMaster on these lists? - funny, I used to use "eBay" in lines like
that - see below) but I will share the fact that our response times and q/s
numbers were very good out-of-the-box.  We've an Ehcache-based distributed
caching framework that we threw in front of the Solr servers, and our final
total throughput for the Solr queries utilized to support Browse runs into
the 100's of requests per sec, per Solr server.  Our Search (true free-text
searches) numbers are lower, but not by that much.

This might be the coolest (for me, anyway) result of the entire endeavor...
StubHub was recently acquired by eBay.  I've been working closely with a
couple of their architects over the past few weeks, and we've done a bit of
theoretical, comparative analysis between our search/browse platform and
theirs (which is completely home grown), and things stack up favorably for
the Solr/Lucene approach, esp. when considering the *vast* difference in
hardware allocation per-request.  Again, if I'm allowed, more details to
follow.

>>...I've taken the liberty of adding stubhub to the "Powered By" Solr
page...

Awesome.  I can't say it enough... thank you all.





On 3/21/07, Yonik Seeley <[hidden email]> wrote:

>
> On 3/20/07, Cass Costello <[hidden email]> wrote:
> > Heh - it used to be in my sig ... my bad.
> >
> > Thanks, all. :)
> >
> > http://www.stubhub.com
>
> I tried it out... looks great!
> Is the faceting stuff done with Solr too?
> If so is it the Solr built-in faceting, or a custom query handler?  If
> the latter, can you think of ways to improve the built in faceting
> that would have made things easier?
>
> -Yonik
>



--
A complex system that works is invariably found to have evolved from a
simple system that works.
  - John Gaule
Reply | Threaded
Open this post in threaded view
|

Re: Thank you...

Otis Gospodnetic-2
In reply to this post by Cass Costello
Thanks for sharing the info, Cass.  Is eBay still using Texis? (this used to be obvious from eBay's URLs a few years ago).  I used Texis with their Vortex script before Lucene was born.

Otis
 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share

----- Original Message ----
From: Cass Costello <[hidden email]>
To: [hidden email]
Cc: [hidden email]
Sent: Wednesday, March 21, 2007 3:59:49 PM
Subject: Re: Thank you...

You all rock.  I'm clearing the semi-official legal hurdle with my CTO and
our head counsel to full (or something close to full) disclosure of some of
the architectural details, so stay tuned for as much as I'm allowed to share
(and btw, for any of you that live/work/vacation in the SF Bay area, I'd be
happy to discuss things over a beer :)

In the meantime, yes, much of the faceting you see on the Browse and Search
pages is direct Solr faceting.  We made *very* liberal use of
faceting-specific fields in our event and ontology documents.

Similarly, we rely on the standard query handler for almost all the Solr
calls.  Our business logic is modelled in a CoR pattern, and we did write
one command to encapsulate SolrCore/Request/Response use for our highest
throughput needs - the win there was direct building into our predominant
xml object model (we use XOM) via a QueryResponseXOMBuilder.build(
solrRequest, solrResponse ) call.

I definitely need clearance before sharing hard perf numbers (anyone working
for TicketMaster on these lists? - funny, I used to use "eBay" in lines like
that - see below) but I will share the fact that our response times and q/s
numbers were very good out-of-the-box.  We've an Ehcache-based distributed
caching framework that we threw in front of the Solr servers, and our final
total throughput for the Solr queries utilized to support Browse runs into
the 100's of requests per sec, per Solr server.  Our Search (true free-text
searches) numbers are lower, but not by that much.

This might be the coolest (for me, anyway) result of the entire endeavor...
StubHub was recently acquired by eBay.  I've been working closely with a
couple of their architects over the past few weeks, and we've done a bit of
theoretical, comparative analysis between our search/browse platform and
theirs (which is completely home grown), and things stack up favorably for
the Solr/Lucene approach, esp. when considering the *vast* difference in
hardware allocation per-request.  Again, if I'm allowed, more details to
follow.

>>...I've taken the liberty of adding stubhub to the "Powered By" Solr
page...

Awesome.  I can't say it enough... thank you all.





On 3/21/07, Yonik Seeley <[hidden email]> wrote:

>
> On 3/20/07, Cass Costello <[hidden email]> wrote:
> > Heh - it used to be in my sig ... my bad.
> >
> > Thanks, all. :)
> >
> > http://www.stubhub.com
>
> I tried it out... looks great!
> Is the faceting stuff done with Solr too?
> If so is it the Solr built-in faceting, or a custom query handler?  If
> the latter, can you think of ways to improve the built in faceting
> that would have made things easier?
>
> -Yonik
>



--
A complex system that works is invariably found to have evolved from a
simple system that works.
  - John Gaule



Reply | Threaded
Open this post in threaded view
|

Re: Thank you...

Kevin Lewandowski
> Thanks for sharing the info, Cass.  Is eBay still using Texis? (this used to be obvious from eBay's URLs a few years ago).  I used Texis with their Vortex script before Lucene was born.

I'd guess no. I read a PDF about ebay's architecture a few months ago
and it said all of the search stuff was custom.
Reply | Threaded
Open this post in threaded view
|

Re: Thank you...

Cass Costello
I haven't heard anyone say "texis" in any discussion since the aquisition.
I'll ask about it specifically next week.

On 3/22/07, Kevin Lewandowski <[hidden email]> wrote:
>
> > Thanks for sharing the info, Cass.  Is eBay still using Texis? (this
> used to be obvious from eBay's URLs a few years ago).  I used Texis with
> their Vortex script before Lucene was born.
>
> I'd guess no. I read a PDF about ebay's architecture a few months ago
> and it said all of the search stuff was custom.
>



--
A complex system that works is invariably found to have evolved from a
simple system that works.
  - John Gaule