[OT] Real Time Open source solutions for aggregation and stream processing

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[OT] Real Time Open source solutions for aggregation and stream processing

Ian Holsman (Lists)
firstly, my apologies for the off-topic message,
but I thought most people on this list would be knowledgeable and
interested in this kind of thing.

We are looking to find a open source, scalable solution to do RT
aggregation and stream processing (similar to what the 'hop' project
http://code.google.com/p/hop/ set out to do) for large(ish) click-stream
logs.

My first thought was something like esper, but in our testing it kind of
hits the wall at around 10,000 rules per JVM.

I was wondering if any of you guys had some experiences in this area,
and what your favorite toolsets are around this.

currently we are using cassandra and redis with home grown software to
do the aggregation, but I'd love to use a common package if there is one.

and again.. apologies for the off-topic message and the x-posting.

regards
Ian