C API for HDFS

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

C API for HDFS

alakshman
How do I get the documentation for the C API ? Also when I try to use the
hdfsConnect() function what do I pass as argument ? Should I pass in the
host name of the master and the port it is listening on ? I mean in the
configuration file for Hadoop I have the fs.default.name set to
"hdfs://dev030.sctm.com:9000". So do I pass in "hdfs://dev030.sctm.com" and
9000 as args. Please advice.

Thanks
A

This is the exception I get when I connect with "hdfs://<host>", port.

Exception in thread "main" java.io.IOException: No FileSystem for scheme:
hdfs
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
Call to org.apache.hadoop.fs.FileSystem::get(URI, Configuration) failed!
Reply | Threaded
Open this post in threaded view
|

Re: C API for HDFS

Arun C Murthy-2
On Wed, Jun 13, 2007 at 10:43:09AM -0700, Phantom wrote:
>How do I get the documentation for the C API ?

Sorry, I should get to put up a twiki on this one...

$ ant doc-libhdfs

which generates doxygen docs from src/c++/libhdfs/hdfs.h

Doug, is there an easy way for us to put up the doxygen-docs, similar to the java api docs?

>Also when I try to use the
>hdfsConnect() function what do I pass as argument ? Should I pass in the
>host name of the master and the port it is listening on ? I mean in the
>configuration file for Hadoop I have the fs.default.name set to
>"hdfs://dev030.sctm.com:9000". So do I pass in "hdfs://dev030.sctm.com" and
>9000 as args. Please advice.
>

Take a look at src/c++/libhdfs/hdfs_test.c for some examples... passing 'default' & 0 will pick up values from hadoop-site.xml

hth,
Arun

>Thanks
>A
>
>This is the exception I get when I connect with "hdfs://<host>", port.
>
>Exception in thread "main" java.io.IOException: No FileSystem for scheme:
>hdfs
>       at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
>Call to org.apache.hadoop.fs.FileSystem::get(URI, Configuration) failed!
Reply | Threaded
Open this post in threaded view
|

Re: C API for HDFS

alakshman
How does it know where to pick up hadoop-site.xml from ? Should I be
defining some env variable or passing this info into the program somehow ?

Thanks
Avinash

On 6/13/07, Arun C Murthy <[hidden email]> wrote:

>
> On Wed, Jun 13, 2007 at 10:43:09AM -0700, Phantom wrote:
> >How do I get the documentation for the C API ?
>
> Sorry, I should get to put up a twiki on this one...
>
> $ ant doc-libhdfs
>
> which generates doxygen docs from src/c++/libhdfs/hdfs.h
>
> Doug, is there an easy way for us to put up the doxygen-docs, similar to
> the java api docs?
>
> >Also when I try to use the
> >hdfsConnect() function what do I pass as argument ? Should I pass in the
> >host name of the master and the port it is listening on ? I mean in the
> >configuration file for Hadoop I have the fs.default.name set to
> >"hdfs://dev030.sctm.com:9000". So do I pass in "hdfs://dev030.sctm.com"
> and
> >9000 as args. Please advice.
> >
>
> Take a look at src/c++/libhdfs/hdfs_test.c for some examples... passing
> 'default' & 0 will pick up values from hadoop-site.xml
>
> hth,
> Arun
>
> >Thanks
> >A
> >
> >This is the exception I get when I connect with "hdfs://<host>", port.
> >
> >Exception in thread "main" java.io.IOException: No FileSystem for scheme:
> >hdfs
> >       at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
> >Call to org.apache.hadoop.fs.FileSystem::get(URI, Configuration) failed!
>
Reply | Threaded
Open this post in threaded view
|

Re: C API for HDFS

alakshman
I ran the hdfs_test and this is error I am getting. What am I missing ?

07/06/13 11:42:13 DEBUG conf.Configuration: java.io.IOException: config()
        at org.apache.hadoop.conf.Configuration.<init>(Configuration.java
:93)

Exception in thread "main" java.io.IOException: No FileSystem for scheme:
hdfs
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
        at org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:119)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:91)
Call to org.apache.hadoop.fs.FileSystem::get failed!
Oops! Failed to connect to hdfs!

Thanks
A

On 6/13/07, Phantom <[hidden email]> wrote:

>
> How does it know where to pick up hadoop-site.xml from ? Should I be
> defining some env variable or passing this info into the program somehow ?
>
> Thanks
> Avinash
>
> On 6/13/07, Arun C Murthy <[hidden email]> wrote:
> >
> > On Wed, Jun 13, 2007 at 10:43:09AM -0700, Phantom wrote:
> > >How do I get the documentation for the C API ?
> >
> > Sorry, I should get to put up a twiki on this one...
> >
> > $ ant doc-libhdfs
> >
> > which generates doxygen docs from src/c++/libhdfs/hdfs.h
> >
> > Doug, is there an easy way for us to put up the doxygen-docs, similar to
> > the java api docs?
> >
> > >Also when I try to use the
> > >hdfsConnect() function what do I pass as argument ? Should I pass in
> > the
> > >host name of the master and the port it is listening on ? I mean in the
> > >configuration file for Hadoop I have the fs.default.name set to
> > >"hdfs://dev030.sctm.com:9000". So do I pass in "hdfs://dev030.sctm.com"
> > and
> > >9000 as args. Please advice.
> > >
> >
> > Take a look at src/c++/libhdfs/hdfs_test.c for some examples... passing
> > 'default' & 0 will pick up values from hadoop-site.xml
> >
> > hth,
> > Arun
> >
> > >Thanks
> > >A
> > >
> > >This is the exception I get when I connect with "hdfs://<host>", port.
> > >
> > >Exception in thread "main" java.io.IOException: No FileSystem for
> > scheme:
> > >hdfs
> > >       at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
> > >Call to org.apache.hadoop.fs.FileSystem::get(URI, Configuration)
> > failed!
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: C API for HDFS

Arun C Murthy-2
In reply to this post by alakshman
On Wed, Jun 13, 2007 at 11:37:57AM -0700, Phantom wrote:
>How does it know where to pick up hadoop-site.xml from ? Should I be
>defining some env variable or passing this info into the program somehow ?
>

It relies on the environment variable 'CLASSPATH'; i.e. just append your conf directory (which contains the preferred hadoop-site.xml) to the CLASSPATH.

Arun

>Thanks
>Avinash
>
>On 6/13/07, Arun C Murthy <[hidden email]> wrote:
>>
>>On Wed, Jun 13, 2007 at 10:43:09AM -0700, Phantom wrote:
>>>How do I get the documentation for the C API ?
>>
>>Sorry, I should get to put up a twiki on this one...
>>
>>$ ant doc-libhdfs
>>
>>which generates doxygen docs from src/c++/libhdfs/hdfs.h
>>
>>Doug, is there an easy way for us to put up the doxygen-docs, similar to
>>the java api docs?
>>
>>>Also when I try to use the
>>>hdfsConnect() function what do I pass as argument ? Should I pass in the
>>>host name of the master and the port it is listening on ? I mean in the
>>>configuration file for Hadoop I have the fs.default.name set to
>>>"hdfs://dev030.sctm.com:9000". So do I pass in "hdfs://dev030.sctm.com"
>>and
>>>9000 as args. Please advice.
>>>
>>
>>Take a look at src/c++/libhdfs/hdfs_test.c for some examples... passing
>>'default' & 0 will pick up values from hadoop-site.xml
>>
>>hth,
>>Arun
>>
>>>Thanks
>>>A
>>>
>>>This is the exception I get when I connect with "hdfs://<host>", port.
>>>
>>>Exception in thread "main" java.io.IOException: No FileSystem for scheme:
>>>hdfs
>>>       at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
>>>Call to org.apache.hadoop.fs.FileSystem::get(URI, Configuration) failed!
>>