Database data storage question

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Database data storage question

McGibbney, Lewis John
Hello List,

I am gathering information on the above topic as I intend to integrate a database to store fetched data. I would like community input of any experiences using different database implementations before doing so. E.g. comparison between HBase & MySQL etc.

Thank you

Lewis


Glasgow Caledonian University is a registered Scottish charity, number SC021474

Winner: Times Higher Education's Widening Participation Initiative of the Year 2009 and Herald Society's Education Initiative of the Year 2009
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
Reply | Threaded
Open this post in threaded view
|

Re: Database data storage question

Otis Gospodnetic-2-2
There are lots of factors to consider, so one can't give a good general answer,
but:

Nutch already uses HBase (trunk), so that's +1 for HBase.  HBase makes it easy
to scale and has built-in replication thanks to being built on top of HDFS.

Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/



----- Original Message ----

> From: "McGibbney, Lewis John" <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Fri, January 14, 2011 8:00:50 AM
> Subject: Database data storage question
>
> Hello List,
>
> I am gathering information on the above topic as I intend to  integrate a
>database to store fetched data. I would like community input of any  experiences
>using different database implementations before doing so. E.g.  comparison
>between HBase & MySQL etc.
>
> Thank  you
>
> Lewis
>
>
> Glasgow Caledonian University is a registered  Scottish charity, number
>SC021474
>
> Winner: Times Higher Education's  Widening Participation Initiative of the Year
>2009 and Herald Society's  Education Initiative of the Year 2009
>http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>l
>
Reply | Threaded
Open this post in threaded view
|

RE: Database data storage question

McGibbney, Lewis John
Hi Otis,

Thank you for this. From reaading various posts on this list and the roadmap for Nutch 2.0 I had gathered that using HBase was probably the most supported option within the community.

Lewis

________________________________________
From: Otis Gospodnetic [[hidden email]]
Sent: 16 January 2011 10:45
To: [hidden email]
Subject: Re: Database data storage question

There are lots of factors to consider, so one can't give a good general answer,
but:

Nutch already uses HBase (trunk), so that's +1 for HBase.  HBase makes it easy
to scale and has built-in replication thanks to being built on top of HDFS.

Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/



----- Original Message ----

> From: "McGibbney, Lewis John" <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Fri, January 14, 2011 8:00:50 AM
> Subject: Database data storage question
>
> Hello List,
>
> I am gathering information on the above topic as I intend to  integrate a
>database to store fetched data. I would like community input of any  experiences
>using different database implementations before doing so. E.g.  comparison
>between HBase & MySQL etc.
>
> Thank  you
>
> Lewis
>
>
> Glasgow Caledonian University is a registered  Scottish charity, number
>SC021474
>
> Winner: Times Higher Education's  Widening Participation Initiative of the Year
>2009 and Herald Society's  Education Initiative of the Year 2009
>http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>l
>

Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems

Glasgow Caledonian University is a registered Scottish charity, number SC021474

Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
Reply | Threaded
Open this post in threaded view
|

Re: Database data storage question

Alexis
Dear Otis and Lewis,

According to the few tests I made. I feel MySQL has the best
performance, compared to HSQL and HBase. HSQL is slower and takes up
so much disk space. HBase uses more resources. Under HBase, I couldn't
get the Fetch job to complete when holding 5000 pages buffered in
memory, without having my laptop getting extremely slow. It finally
worked with a flushing frequency to the store of 2500 pages. Under
MySQL, it worked out smoothly with a 10000 value.

NoSQL technology scales better, but for a "reasonable" volume MySQL
will do the job fine and faster.

It would be nice to test Cassandra as Gora backend. Write operations
are allegedly faster that Hbase. Haven't tried yet.

Alexis


On Sun, Jan 16, 2011 at 12:57 PM, McGibbney, Lewis John
<[hidden email]> wrote:

> Hi Otis,
>
> Thank you for this. From reaading various posts on this list and the roadmap for Nutch 2.0 I had gathered that using HBase was probably the most supported option within the community.
>
> Lewis
>
> ________________________________________
> From: Otis Gospodnetic [[hidden email]]
> Sent: 16 January 2011 10:45
> To: [hidden email]
> Subject: Re: Database data storage question
>
> There are lots of factors to consider, so one can't give a good general answer,
> but:
>
> Nutch already uses HBase (trunk), so that's +1 for HBase.  HBase makes it easy
> to scale and has built-in replication thanks to being built on top of HDFS.
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
>> From: "McGibbney, Lewis John" <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Fri, January 14, 2011 8:00:50 AM
>> Subject: Database data storage question
>>
>> Hello List,
>>
>> I am gathering information on the above topic as I intend to  integrate a
>>database to store fetched data. I would like community input of any  experiences
>>using different database implementations before doing so. E.g.  comparison
>>between HBase & MySQL etc.
>>
>> Thank  you
>>
>> Lewis
>>
>>
>> Glasgow Caledonian University is a registered  Scottish charity, number
>>SC021474
>>
>> Winner: Times Higher Education's  Widening Participation Initiative of the Year
>>2009 and Herald Society's  Education Initiative of the Year 2009
>>http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>>l
>>
>
> Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems
>
> Glasgow Caledonian University is a registered Scottish charity, number SC021474
>
> Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Database data storage question

Andrzej Białecki-2
On 1/16/11 7:40 AM, Alexis wrote:
> Dear Otis and Lewis,
>
> According to the few tests I made. I feel MySQL has the best
> performance, compared to HSQL and HBase. HSQL is slower and takes up
> so much disk space. HBase uses more resources. Under HBase, I couldn't
> get the Fetch job to complete when holding 5000 pages buffered in
> memory, without having my laptop getting extremely slow. It finally
> worked with a flushing frequency to the store of 2500 pages. Under
> MySQL, it worked out smoothly with a 10000 value.

...and this is of course nowhere near the level of scalability that 1.x
releases had, as they would easily crawl a hundred million pages.
There's a lot of remaining work on Gora and its integration with Nutch
that affects this situation.

Eventually I expect HBase will be the best choice for large scale
crawling, with MySQL backend suitable for small to medium scale, and
HSQL being used only for tests or really small crawls < 1000 pages.

>
> NoSQL technology scales better, but for a "reasonable" volume MySQL
> will do the job fine and faster.
>
> It would be nice to test Cassandra as Gora backend. Write operations
> are allegedly faster that Hbase. Haven't tried yet.

There are some concurrency limitations in the Cassandra client - OTOH
that's maybe where Gora needs to improve.

--
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Reply | Threaded
Open this post in threaded view
|

Re: Database data storage question

Alexis
>> NoSQL technology scales better, but for a "reasonable" volume MySQL
>> will do the job fine and faster.

Sorry it was not working that well in my tests with Gora code as is
and MySQL backend, because of the broad SELECT statement. The issue is
described here:
http://techvineyard.blogspot.com/2010/12/build-nutch-20.html#Gora

and reported in the JIRA: https://issues.apache.org/jira/browse/GORA-23

Alexis