some questions about hadoop

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

some questions about hadoop

王学超
hi, everyone!
I'm new,I am not familiar with hadoop,so I have some following
questions,please help me. thanks
1、Are there 3rd party technologies and included in hadoop or dependencies?
2、How is hadoop's high-availability handled? Built-in or requires
separate HA manager?
3、How is security handled?
4、How is error logging handled?
Reply | Threaded
Open this post in threaded view
|

Re: some questions about hadoop

Khalil Honsali
As far as my basic understanding goes:
1 - 3rd party - or so technologies are located in the folder hadoop-xx/lib ,
for example apache's commons-logging technologie ...
4 -  ... which is used by hadoop for logging into the log files specified in
the configuration files. log files are usually in the hadoop-xx/logs/
directory and there are two types of logging: INFO (information messages)
and ERROR. you can tune logging schemes via hadoop-xx/conf/commons-
logging.properties or hadoop-xx/conf/log4j.properties

On 12/12/2007, 王学超 <[hidden email]> wrote:

>
> hi, everyone!
> I'm new,I am not familiar with hadoop,so I have some following
> questions,please help me. thanks
> 1、Are there 3rd party technologies and included in hadoop or dependencies?
> 2、How is hadoop's high-availability handled? Built-in or requires
> separate HA manager?
> 3、How is security handled?
> 4、How is error logging handled?
>



--
---------------------------------------------------------
شهر مبارك كريم
كل عام و أنتم بخير
---------------------------------------------------------
Honsali Khalil − 本査理 カリル
Academic>Japan>NIT>Grad. Sc. Eng.>Dept. CS>Matsuo&Tsumura Lab.
http://www.matlab.nitech.ac.jp/~k-hon/<http://www.matlab.nitech.ac.jp/%7Ek-hon/>
+81 (zero-)eight-zero 5134 8119
[hidden email] (instant reply mail)
Reply | Threaded
Open this post in threaded view
|

Re: some questions about hadoop

王学超
thank you ,Khalil Honsali
Can others  can give me some other suggestion?
I am eagerly waiting for your reply.


2007/12/12, Khalil Honsali <[hidden email]>:

>
> As far as my basic understanding goes:
> 1 - 3rd party - or so technologies are located in the folder hadoop-xx/lib
> ,
> for example apache's commons-logging technologie ...
> 4 -  ... which is used by hadoop for logging into the log files specified
> in
> the configuration files. log files are usually in the hadoop-xx/logs/
> directory and there are two types of logging: INFO (information messages)
> and ERROR. you can tune logging schemes via hadoop-xx/conf/commons-
> logging.properties or hadoop-xx/conf/log4j.properties
>
> On 12/12/2007, 王学超 <[hidden email]> wrote:
> >
> > hi, everyone!
> > I'm new,I am not familiar with hadoop,so I have some following
> > questions,please help me. thanks
> > 1、Are there 3rd party technologies and included in hadoop or
> dependencies?
> > 2、How is hadoop's high-availability handled? Built-in or requires
> > separate HA manager?
> > 3、How is security handled?
> > 4、How is error logging handled?
> >
>
>
>
> --
> ---------------------------------------------------------
> شهر مبارك كريم
> كل عام و أنتم بخير
> ---------------------------------------------------------
> Honsali Khalil − 本査理 カリル
> Academic>Japan>NIT>Grad. Sc. Eng.>Dept. CS>Matsuo&Tsumura Lab.
> http://www.matlab.nitech.ac.jp/~k-hon/<
> http://www.matlab.nitech.ac.jp/%7Ek-hon/>
> +81 (zero-)eight-zero 5134 8119
> [hidden email] (instant reply mail)
>



--
                     此致
敬礼
Reply | Threaded
Open this post in threaded view
|

Re: some questions about hadoop

Ted Dunning-3


3.  There is currently no security.  Weak user level security will appear
soon (but you will still be able to lie about who you are).  Stronger
security is in the works, but you should expect to protect a Hadoop cluster
from the outside.

2.  High availability is inherent in hadoop's map-reduce system and storage
system.  You have to use external tools to achieve full HA capabilities for
the central name node.


On 12/12/07 12:21 AM, "王学超" <[hidden email]> wrote:

> thank you ,Khalil Honsali
Can others  can give me some other suggestion?
I am
> eagerly waiting for your reply.


2007/12/12, Khalil Honsali

> <[hidden email]>:
>
> As far as my basic understanding goes:
> 1 - 3rd
> party - or so technologies are located in the folder hadoop-xx/lib
> ,
> for
> example apache's commons-logging technologie ...
> 4 -  ... which is used by
> hadoop for logging into the log files specified
> in
> the configuration
> files. log files are usually in the hadoop-xx/logs/
> directory and there are
> two types of logging: INFO (information messages)
> and ERROR. you can tune
> logging schemes via hadoop-xx/conf/commons-
> logging.properties or
> hadoop-xx/conf/log4j.properties
>
> On 12/12/2007, 王学超 <[hidden email]>
> wrote:
> >
> > hi, everyone!
> > I'm new,I am not familiar with hadoop,so I
> have some following
> > questions,please help me. thanks
> > 1、Are there 3rd
> party technologies and included in hadoop or
> dependencies?
> > 2、How is
> hadoop's high-availability handled? Built-in or requires
> > separate HA
> manager?
> > 3、How is security handled?
> > 4、How is error logging handled?
>
> >
>
>
>
> --
> ---------------------------------------------------------
> شهر
> مبارك كريم
> كل عام و أنتم بخير
>
> ---------------------------------------------------------
> Honsali Khalil − 本
> 査理 カリル
> Academic>Japan>NIT>Grad. Sc. Eng.>Dept. CS>Matsuo&Tsumura Lab.
>
> http://www.matlab.nitech.ac.jp/~k-hon/<
>
> http://www.matlab.nitech.ac.jp/%7Ek-hon/>
> +81 (zero-)eight-zero 5134 8119
>
> [hidden email] (instant reply mail)
>



--
                     此致

> 礼


Reply | Threaded
Open this post in threaded view
|

Re: some questions about hadoop

Allen Wittenauer-3



On 12/12/07 9:16 AM, "Ted Dunning" <[hidden email]> wrote:
> 3.  There is currently no security.  Weak user level security will appear
> soon (but you will still be able to lie about who you are).  Stronger
> security is in the works, but you should expect to protect a Hadoop cluster
> from the outside.

    It is probably worth pointing out that the approach we took to
protecting the HDFS was to firewall off the fs.default.name machine/port and
50075 (datanode.something.or.other) while also having users use another port
that pointed to an apache instance that wrapped the dfs browser stuff. Hosts
that can connect to the HDFS is then strictly controlled, including limited
root access, etc.

    [Yes, this is painful.]