Using Lucene Index for Business Intelligence / Analytics

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Using Lucene Index for Business Intelligence / Analytics

saurabhd


                                I don't have a lot of experience with reporting tools and how data is stored by high priced tools which use OLAP and other similar storage types but we needed a solution for drill down reports and searching within large application logs / web server logs.

So, we decided to build a reporting / analytics solution using Lucene. The product is almost ready and we are doing closed beta testing for now for reporting & searching in Web Server logs with our small set of beta customers. I am not here to promote the product or drop the URL, but wanted to get an opinion from this group on using Lucene as the data store for Business Intelligence / Analytics.

If implemented correctly, is there any reason why a Lucene based data store cannot compete with the high priced reporting / data analytics products for drill down reports? I know about a lot of Lucene advantages which I can talk about with customers, but what can be the downside of using Lucene / competitive advantages in other products which we should be aware of?

Thanks
Saurabh Dani


Reply | Threaded
Open this post in threaded view
|

Re: Using Lucene Index for Business Intelligence / Analytics

Tomi N/A
On 8/31/06, Saurabh Dani <[hidden email]> wrote:
>
>
>                                 I don't have a lot of experience with reporting tools and how data is stored by high priced tools which use OLAP and other similar storage types but we needed a solution for drill down reports and searching within large application logs / web server logs.
>
> So, we decided to build a reporting / analytics solution using Lucene. The product is almost ready and we are doing closed beta testing for now for reporting & searching in Web Server logs with our small set of beta customers. I am not here to promote the product or drop the URL, but wanted to get an opinion from this group on using Lucene as the data store for Business Intelligence / Analytics.
>
> If implemented correctly, is there any reason why a Lucene based data store cannot compete with the high priced reporting / data analytics products for drill down reports? I know about a lot of Lucene advantages which I can talk about with customers, but what can be the downside of using Lucene / competitive advantages in other products which we should be aware of?

It doesn't sound promising to me.
From my own (true, not too rich) experience, search engine has as much
to do with business inteligence as e.g. text processing.
I mean, yes, you can probably draw an analogy of an OLAP drill-down to
searching withing search results, but that's about it...reporting and
analysis tools are fundamentaly different from a search engine in that
they work with highly structured data and leverage that structure to
do various calculations the results of which are then presented to the
user. A search engine, on the other hand, is basically a tool designed
to handle tons of unstructured or badly structured (i.e. structured
for the human reader) data.
In conclusion, you can do great things with a search engine, but BI in
general or reporting in particular is not one of them. Take a look at
bizgres, pentaho (mondrian, jrubik/jpivot, kettle, weka), and
jasperreports to get a better idea of what I'm talking about.

Just another opinion, anyway...

t.n.a.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]