Fwd: way to add custom udf jar in hadoop 2.x version

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: way to add custom udf jar in hadoop 2.x version

Ted Yu-3
Forwarding Niels' question to hive mailing list.

On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <[hidden email]> wrote:

Thanks for the pointer.
This seems to work for functions. Is there something similar for CREATE EXTERNAL TABLE ??

Niels

On Dec 31, 2014 8:13 AM, "Ted Yu" <[hidden email]> wrote:
Have you seen this thread ?

On Dec 30, 2014, at 10:56 PM, reena upadhyay <[hidden email]> wrote:

Hi,

I am using hadoop 2.4.0 version. I have created custom udf jar. I am trying to execute a simple select udf query using java hive jdbc client program. When hive execute the query using map reduce job, then the query execution get fails because the mapper is not able to locate the udf class.
So I wanted to add the udf jar in hadoop environment permanently. Please suggest me a way to add this external jar for single node and multi node hadoop cluster.

PS: I am using hive 0.13.1 version and I already have this custom udf jar added in HIVE_HOME/lib directory.


Thanks

Reply | Threaded
Open this post in threaded view
|

Re: way to add custom udf jar in hadoop 2.x version

Yakubovich, Alexey

Another advice: insert your ADD JAR commands in your $HOME/.hiverc file and start hive. (<a href="http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr&#43;0h3SMDw4zHTpYo5B1B4iob05BPW8LS&#43;dAEH595qZidjEQ@mail.gmail.com%3E">http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@...%3E)



From: Ted Yu <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, December 31, 2014 at 8:25 AM
To: "[hidden email]" <[hidden email]>
Subject: Fwd: way to add custom udf jar in hadoop 2.x version

Forwarding Niels' question to hive mailing list.

On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <[hidden email]> wrote:

Thanks for the pointer.
This seems to work for functions. Is there something similar for CREATE EXTERNAL TABLE ??

Niels

On Dec 31, 2014 8:13 AM, "Ted Yu" <[hidden email]> wrote:
Have you seen this thread ?
<a href="http://search-hadoop.com/m/8er9TcALc/Hive&#43;udf&#43;custom&#43;jar&amp;subj=Best&#43;way&#43;to&#43;add&#43;custom&#43;UDF&#43;jar&#43;in&#43;HiveServer2" target="_blank">http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2

On Dec 30, 2014, at 10:56 PM, reena upadhyay <[hidden email]> wrote:

Hi,

I am using hadoop 2.4.0 version. I have created custom udf jar. I am trying to execute a simple select udf query using java hive jdbc client program. When hive execute the query using map reduce job, then the query execution get fails because the mapper is not able to locate the udf class.
So I wanted to add the udf jar in hadoop environment permanently. Please suggest me a way to add this external jar for single node and multi node hadoop cluster.

PS: I am using hive 0.13.1 version and I already have this custom udf jar added in HIVE_HOME/lib directory.


Thanks

This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: way to add custom udf jar in hadoop 2.x version

Niels Basjes
Hi,

These options:
HIVE_HOME/auxlib
- ADD JAR commands in your $HOME/.hiverc file

either require IT operations to put my JAR on all nodes OR I cannot share it, Only works on the commandline and it won't work in HUE/Beeswax.

Now "Permanent Functions":

What these "Permanent Functions" do is:
1) put the jar on the cluster without IT operations putting the jar on all nodes
2) the jar is used transparently for everyone who want to use the function.

I am writing a deserializer [1] (Not finished yet: https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that should make existing files query-able as an external table in Hive.

Question is: Is there something similar for CREATE EXTERNAL TABLE ??

Something like 

CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
    ...
    STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)] 
    [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];


Is this something for which there is already a JIRA (couldn't find it)?
If not; Should I create one? (I.e. do you think this would make sense for others?)

Niels Basjes


On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <[hidden email]> wrote:

Another advice: insert your ADD JAR commands in your $HOME/.hiverc file and start hive. (http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@...%3E)



From: Ted Yu <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, December 31, 2014 at 8:25 AM
To: "[hidden email]" <[hidden email]>
Subject: Fwd: way to add custom udf jar in hadoop 2.x version

Forwarding Niels' question to hive mailing list.

On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <[hidden email]> wrote:

Thanks for the pointer.
This seems to work for functions. Is there something similar for CREATE EXTERNAL TABLE ??

Niels

On Dec 31, 2014 8:13 AM, "Ted Yu" <[hidden email]> wrote:
Have you seen this thread ?

On Dec 30, 2014, at 10:56 PM, reena upadhyay <[hidden email]> wrote:

Hi,

I am using hadoop 2.4.0 version. I have created custom udf jar. I am trying to execute a simple select udf query using java hive jdbc client program. When hive execute the query using map reduce job, then the query execution get fails because the mapper is not able to locate the udf class.
So I wanted to add the udf jar in hadoop environment permanently. Please suggest me a way to add this external jar for single node and multi node hadoop cluster.

PS: I am using hive 0.13.1 version and I already have this custom udf jar added in HIVE_HOME/lib directory.


Thanks

This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.



--
Best regards / Met vriendelijke groeten,

Niels Basjes
Reply | Threaded
Open this post in threaded view
|

Re: way to add custom udf jar in hadoop 2.x version

Niels Basjes
I created https://issues.apache.org/jira/browse/HIVE-9252 for this improvement.

On Sun, Jan 4, 2015 at 5:16 PM, Niels Basjes <[hidden email]> wrote:
Hi,

These options:
HIVE_HOME/auxlib
- ADD JAR commands in your $HOME/.hiverc file

either require IT operations to put my JAR on all nodes OR I cannot share it, Only works on the commandline and it won't work in HUE/Beeswax.

Now "Permanent Functions":

What these "Permanent Functions" do is:
1) put the jar on the cluster without IT operations putting the jar on all nodes
2) the jar is used transparently for everyone who want to use the function.

I am writing a deserializer [1] (Not finished yet: https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that should make existing files query-able as an external table in Hive.

Question is: Is there something similar for CREATE EXTERNAL TABLE ??

Something like 

CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
    ...
    STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)] 
    [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];


Is this something for which there is already a JIRA (couldn't find it)?
If not; Should I create one? (I.e. do you think this would make sense for others?)

Niels Basjes


On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <[hidden email]> wrote:

Another advice: insert your ADD JAR commands in your $HOME/.hiverc file and start hive. (http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@...%3E)



From: Ted Yu <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, December 31, 2014 at 8:25 AM
To: "[hidden email]" <[hidden email]>
Subject: Fwd: way to add custom udf jar in hadoop 2.x version

Forwarding Niels' question to hive mailing list.

On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <[hidden email]> wrote:

Thanks for the pointer.
This seems to work for functions. Is there something similar for CREATE EXTERNAL TABLE ??

Niels

On Dec 31, 2014 8:13 AM, "Ted Yu" <[hidden email]> wrote:
Have you seen this thread ?

On Dec 30, 2014, at 10:56 PM, reena upadhyay <[hidden email]> wrote:

Hi,

I am using hadoop 2.4.0 version. I have created custom udf jar. I am trying to execute a simple select udf query using java hive jdbc client program. When hive execute the query using map reduce job, then the query execution get fails because the mapper is not able to locate the udf class.
So I wanted to add the udf jar in hadoop environment permanently. Please suggest me a way to add this external jar for single node and multi node hadoop cluster.

PS: I am using hive 0.13.1 version and I already have this custom udf jar added in HIVE_HOME/lib directory.


Thanks

This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.



--
Best regards / Met vriendelijke groeten,

Niels Basjes



--
Best regards / Met vriendelijke groeten,

Niels Basjes