[jira] Created: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org
Implement the LzoCodec with support for the lzo compression algorithms
----------------------------------------------------------------------

                 Key: HADOOP-851
                 URL: https://issues.apache.org/jira/browse/HADOOP-851
             Project: Hadoop
          Issue Type: New Feature
          Components: io
            Reporter: Arun C Murthy
         Assigned To: Arun C Murthy
             Fix For: 0.10.0


lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html

It should be a good value-add for hadoop...


--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-851:
---------------------------------

    Attachment: HADOOP-851_1_20070103.patch

> Implement the LzoCodec with support for the lzo compression algorithms
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-851
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>             Fix For: 0.10.0
>
>         Attachments: HADOOP-851_1_20070103.patch
>
>
> lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html
> It should be a good value-add for hadoop...

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462016 ]

Arun C Murthy commented on HADOOP-851:
--------------------------------------

lzo patch for review... appreciate any feedback.

> Implement the LzoCodec with support for the lzo compression algorithms
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-851
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>             Fix For: 0.10.0
>
>         Attachments: HADOOP-851_1_20070103.patch
>
>
> lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html
> It should be a good value-add for hadoop...

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-851:
---------------------------------

    Fix Version/s:     (was: 0.10.0)
           Status: Patch Available  (was: Open)

> Implement the LzoCodec with support for the lzo compression algorithms
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-851
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>         Attachments: HADOOP-851_1_20070103.patch
>
>
> lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html
> It should be a good value-add for hadoop...

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462594 ]

Hadoop QA commented on HADOOP-851:
----------------------------------

+1, because http://issues.apache.org/jira/secure/attachment/12348214/HADOOP-851_1_20070103.patch applied and successfully tested against trunk revision r493146.

> Implement the LzoCodec with support for the lzo compression algorithms
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-851
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>         Attachments: HADOOP-851_1_20070103.patch
>
>
> lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html
> It should be a good value-add for hadoop...

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462601 ]

Doug Cutting commented on HADOOP-851:
-------------------------------------

I can't build this on Ubuntu Dapper, which only has packages for liblzo1.  Ubuntu Edgy has lzo2 support.  I guess this gives me an excuse to upgrade to Edgy...

http://packages.ubuntu.com/cgi-bin/search_packages.pl?keywords=lzo&searchon=names&subword=1&version=all&release=all

> Implement the LzoCodec with support for the lzo compression algorithms
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-851
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>         Attachments: HADOOP-851_1_20070103.patch
>
>
> lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html
> It should be a good value-add for hadoop...

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-851:
--------------------------------

    Status: Open  (was: Patch Available)

I am able to build this without problem on Ubuntu Edgy.

Since liblzo2 isn't normally installed, I'm hesitant to include this in libhadoop.so, since I think libhadoop.so would then no longer link unless folks have liblzo2 installed.  (This is different from zlib, which is normally installed on most systems.)  It's a pain to build a separate library, but I don't see an alternative.

Also, can you please add a unit test?  This should be a no-op when the native code isn't available.  This could be a simple addition to TestSequenceFile#testSequenceFile, having it test another codec.

> Implement the LzoCodec with support for the lzo compression algorithms
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-851
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>         Attachments: HADOOP-851_1_20070103.patch
>
>
> lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html
> It should be a good value-add for hadoop...

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12463461 ]

Arun C Murthy commented on HADOOP-851:
--------------------------------------

Doug, I didn't realise this could be a problem... I took a look at the lzo ChangeLog (http://www.oberhumer.com/opensource/lzo/lzonews.php), saw that lzo2 has been around for almost 2 years now (released in May 2005) and assumed lzo2 is reasonably common.

The way libhadoop.so is structured now we don't mandate lzo2 or zlib, thus people can still use native-zlib without installing lzo2 or vice-versa and libhadoop.so will link fine. However if they need the lzo codec, they are forced to install lzo2 (as opposed to lzo1). Does that address your concern?

Sure, I'll also add the test case.

> Implement the LzoCodec with support for the lzo compression algorithms
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-851
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>         Attachments: HADOOP-851_1_20070103.patch
>
>
> lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html
> It should be a good value-add for hadoop...

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12463622 ]

Doug Cutting commented on HADOOP-851:
-------------------------------------

> lzo2 has been around for almost 2 years now

I'm okay with lzo2.  It's probably best to start with the more recent release.

> people can still use native-zlib without installing lzo2 or vice-versa and libhadoop.so will link fine

Perfect!  That's exactly what I was concerned about.

So once we add a test I can commit this.  Thanks!

> Implement the LzoCodec with support for the lzo compression algorithms
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-851
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>         Attachments: HADOOP-851_1_20070103.patch
>
>
> lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html
> It should be a good value-add for hadoop...

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-851:
---------------------------------

    Attachment: HADOOP-851_20070110_2.patch

> Implement the LzoCodec with support for the lzo compression algorithms
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-851
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>         Attachments: HADOOP-851_1_20070103.patch, HADOOP-851_20070110_2.patch
>
>
> lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html
> It should be a good value-add for hadoop...

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-851:
---------------------------------

    Fix Version/s: 0.10.1
           Status: Patch Available  (was: Open)

> That's exactly what I was concerned about.
Great! This is something we were very careful about during HADOOP-538 and that's why we have the dlopen/dlsym stuff (http://issues.apache.org/jira/browse/HADOOP-538#action_12446647).

I've added the test cases and here is the new patch...

> Implement the LzoCodec with support for the lzo compression algorithms
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-851
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>             Fix For: 0.10.1
>
>         Attachments: HADOOP-851_1_20070103.patch, HADOOP-851_20070110_2.patch
>
>
> lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html
> It should be a good value-add for hadoop...

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-851) Implement the LzoCodec with support for the lzo compression algorithms

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-851:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Arun!

> Implement the LzoCodec with support for the lzo compression algorithms
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-851
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>             Fix For: 0.10.1
>
>         Attachments: HADOOP-851_1_20070103.patch, HADOOP-851_20070110_2.patch
>
>
> lzo is clearly one the best compression libraries out there: ... http://compression.ca/act/act-summary.html
> It should be a good value-add for hadoop...

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira