Streaming results of analysis to shards ... possible?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Streaming results of analysis to shards ... possible?

Cass Costello
Hello all,

Our application involves a high index write rate - anywhere from a few
dozen to many thousands of docs per sec.  The write rate is frequently
higher than the read rate (though not always), and our index must be
as fresh as possible (we'd like search results to be no more than a
couple of seconds out of date). We're considering many approaches to
achieving our desired TCO.

We've noted that the indexing process can be quite costly.  Our latest
POC shards the total index over N machines which effectively
distributes the indexing load and keeps refresh and and search
response times decent, but to maintain performance during peak write
rates, we've had to make N a much larger number than we'd like.

One idea we're floating would be to do all the analysis centrally,
perhaps on N/4 machines, and then stream the raw tokens and data
directly to the read "slaves," who would (hopefully) need to do
nothing more than manage segments and readers.

We have some very rough math that makes the approach compelling, but
before diving in wholesale, we thought we'd ask if anyone else has
taken a similar approach.   Thoughts?


Cass Costello

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]