Using the HDFS C API

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Using the HDFS C API

alakshman
I tried running the hdfs_test from a machine which is not part of the Hadoop
cluster. Could someone please tell me what I am doing wrong (error shown
below)?
I get the following error :

07/06/20 20:16:43 DEBUG conf.Configuration: java.io.IOException: config()
        at org.apache.hadoop.conf.Configuration.<init>(Configuration.java
:93)
        at org.apache.hadoop.dfs.FSConstants.<clinit>(FSConstants.java:120)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(
DFSClient.java:1039)
        at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:318)
        at
org.apache.hadoop.dfs.DistributedFileSystem$RawDistributedFileSystem.create(
DistributedFileSystem.java:143)
        at org.apache.hadoop.fs.ChecksumFileSystem$FSOutputSummer.<init>(
ChecksumFileSystem.java:374)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(
ChecksumFileSystem.java:449)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:360)

Exception in thread "main" java.io.IOException: No valid local directories
in property: dfs.client.buffer.dir
        at org.apache.hadoop.conf.Configuration.getFile(Configuration.java
:410)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(
DFSClient.java:1100)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(
DFSClient.java:1067)
        at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:318)
        at
org.apache.hadoop.dfs.DistributedFileSystem$RawDistributedFileSystem.create(
DistributedFileSystem.java:143)
        at org.apache.hadoop.fs.ChecksumFileSystem$FSOutputSummer.<init>(
ChecksumFileSystem.java:374)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(
ChecksumFileSystem.java:449)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:360)
Call to org.apache.hadoop.fs.FileSystem::create((Lorg/apache/hadoop/fs/Path;ZISJ)Lorg/apache/hadoop/fs/FSDataOutputStream;)
failed!
Failed to open /tmp/testfile.txt for writing!
Reply | Threaded
Open this post in threaded view
|

HBase from Map-Reduce update?

James Kennedy-6
 From Wiki:

"We need easy interfaces to MapReduce jobs, so they can scan tables. We
have been contacted by Vuk Ercegovac
<vercego AT SPAMFREE us DOT ibm DOT com> of IBM Almaden Research who
expressed an interest in working on an HBase interface to Hadoop
map/reduce."

Does anyone know what the status of this is?
Reply | Threaded
Open this post in threaded view
|

Re: HBase from Map-Reduce update?

hank williams
I have been meaning to ask a similar question, but perhaps a bit more
broadly about the status and anticipated timeline for hbase. I am
curious if the effort is purely individual or if there is any
corporate push (for example from powerset) or if it is just a personal
project.

Also, as I understand it from the wiki, hbase works well on a single
machine but scaling is being worked out. Is there any sense of when an
anticipated (or desired) target for a scalable version?

Regards
Hank

On 6/21/07, James Kennedy <[hidden email]> wrote:

>  From Wiki:
>
> "We need easy interfaces to MapReduce jobs, so they can scan tables. We
> have been contacted by Vuk Ercegovac
> <vercego AT SPAMFREE us DOT ibm DOT com> of IBM Almaden Research who
> expressed an interest in working on an HBase interface to Hadoop
> map/reduce."
>
> Does anyone know what the status of this is?
>
Reply | Threaded
Open this post in threaded view
|

Re: HBase from Map-Reduce update?

stack-3
hank williams wrote:
> I have been meaning to ask a similar question, but perhaps a bit more
> broadly about the status and anticipated timeline for hbase. I am
> curious if the effort is purely individual or if there is any
> corporate push (for example from powerset) or if it is just a personal
> project.
Regards the state of HBase, I just made updates to the HBase home page
news section and edited the tail of the architecture page where it talks
of current status.  JIRA is -- or should be -- the best source for what
is currently being worked on.

Broadly, from my point of view:

+ I am able to start up a cluster of 4 nodes and run simple sustained
loads without the cluster falling over.
+ I'm currently working on loading it with a bunch of web pages with
columns to hold original content, parsed content, etc. It looks like
there is sufficient functionality currently implemented for me to do
this (we'll see).
+ Its slow (See the PerformanceEvaluation page in wiki).
+ Its fragile (E.g HADOOP-1516)

Regards timeline, Powerset where both Jim Kellerman and I work would
love to have a working HBase as soon as possible (smile).

Jim and I are both working on HBase near full-time.  Others are also
making good contributions.

But, IMO, HBase is a big job and its going to take a load of work by a
bunch of folks making it robust at scale.

> Also, as I understand it from the wiki, hbase works well on a single
> machine but scaling is being worked out. Is there any sense of when an
> anticipated (or desired) target for a scalable version?
Adding nodes to a HBase cluster currently makes it so that the cluster
can carry more load and handle more requests.  In a sense, with caveats
-- some of which have been mentioned above while others are yet to be
discovered -- HBase scales now.

James Kennedy wrote:
> "We need easy interfaces to MapReduce jobs, so they can scan tables.
> We have been contacted by Vuk Ercegovac <vercego AT SPAMFREE us DOT
> ibm DOT com> of IBM Almaden Research who expressed an interest in
> working on an HBase interface to Hadoop map/reduce."
>
> Does anyone know what the status of this is?
I do not know of any work that has been done on the above.  I made an
issue for it (HADOOP-1519).

St.Ack


>
> Regards
> Hank
>
> On 6/21/07, James Kennedy <[hidden email]> wrote:
>>  From Wiki:
>>
>> "We need easy interfaces to MapReduce jobs, so they can scan tables. We
>> have been contacted by Vuk Ercegovac
>> <vercego AT SPAMFREE us DOT ibm DOT com> of IBM Almaden Research who
>> expressed an interest in working on an HBase interface to Hadoop
>> map/reduce."
>>
>> Does anyone know what the status of this is?
>>