Are these configuration parameters deprecated?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Are these configuration parameters deprecated?

Tianyin Xu
Hi,

I'm very confused by some of the MapReduce configuration parameters
which appear in the latest version of mapred-default.xml.
http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
in code but

/* mapreduce/util/ConfigUtil.java */
 55       new DeprecationDelta("mapred.temp.dir",
 56         MRConfig.TEMP_DIR),

My interpretation is that it's renamed into "mapred.temp.dir".
However, when I grep the new name, I still cannot find any code except
some testing ones in
./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java

From the semantics, this should be a "must-to-have" parameter for MR jobs...

Also, many parameters are like this. So I'm really confused.

Am I missing something? Are these parameters deprecated?

Thanks a lot!
Tianyin
Reply | Threaded
Open this post in threaded view
|

Re: Are these configuration parameters deprecated?

Niels Basjes
A while ago I found a similar problem; 
I was wondering why do a lot of tools like the hdfs, hbase shell, pig and many other complain at startup about deprecated parameters.
It turns out that these deprecated names are still in *-default.xml files and in various other places in the code base.

Perhaps an issue indicating that the use of the deprecated parameters should be removed from the main code base is in order here.

Niels Basjes

On Fri, Nov 14, 2014 at 9:22 PM, Tianyin Xu <[hidden email]> wrote:
Hi,

I'm very confused by some of the MapReduce configuration parameters
which appear in the latest version of mapred-default.xml.
http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
in code but

/* mapreduce/util/ConfigUtil.java */
 55       new DeprecationDelta("mapred.temp.dir",
 56         MRConfig.TEMP_DIR),

My interpretation is that it's renamed into "mapred.temp.dir".
However, when I grep the new name, I still cannot find any code except
some testing ones in
./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java

From the semantics, this should be a "must-to-have" parameter for MR jobs...

Also, many parameters are like this. So I'm really confused.

Am I missing something? Are these parameters deprecated?

Thanks a lot!
Tianyin



--
Best regards / Met vriendelijke groeten,

Niels Basjes
Reply | Threaded
Open this post in threaded view
|

Re: Are these configuration parameters deprecated?

Tianyin Xu
So, these parameters are indeed deprecated?

Then how do I find the replaced one? Let's just say
"mapreduce.task.tmp.dir," how which one should I set to have the same
behavior?

@Niels, yes, exactly, Hadoop in particular has a high churn of
configuration renaming which is pretty annoying. Not even the
deprecated parameters are not complained, but also any invalid
parameters are like this.

~t


On Fri, Nov 14, 2014 at 1:03 PM, Niels Basjes <[hidden email]> wrote:

> A while ago I found a similar problem;
> I was wondering why do a lot of tools like the hdfs, hbase shell, pig and
> many other complain at startup about deprecated parameters.
> It turns out that these deprecated names are still in *-default.xml files
> and in various other places in the code base.
>
> Perhaps an issue indicating that the use of the deprecated parameters should
> be removed from the main code base is in order here.
>
> Niels Basjes
>
> On Fri, Nov 14, 2014 at 9:22 PM, Tianyin Xu <[hidden email]> wrote:
>>
>> Hi,
>>
>> I'm very confused by some of the MapReduce configuration parameters
>> which appear in the latest version of mapred-default.xml.
>>
>> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>>
>> Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
>> in code but
>>
>> /* mapreduce/util/ConfigUtil.java */
>>  55       new DeprecationDelta("mapred.temp.dir",
>>  56         MRConfig.TEMP_DIR),
>>
>> My interpretation is that it's renamed into "mapred.temp.dir".
>> However, when I grep the new name, I still cannot find any code except
>> some testing ones in
>>
>> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java
>>
>> From the semantics, this should be a "must-to-have" parameter for MR
>> jobs...
>>
>> Also, many parameters are like this. So I'm really confused.
>>
>> Am I missing something? Are these parameters deprecated?
>>
>> Thanks a lot!
>> Tianyin
>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes



--
Tianyin XU,
http://cseweb.ucsd.edu/~tixu/