Reducer to output only json

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Reducer to output only json

Chengi Liu
Hi, 

 I have the following redcuer class

public static class TokenCounterReducer
    extends Reducer<Text, Text, Text, Text> {
    public void reduce(Text key, Iterable<Text> values, Context context)
        throws IOException, InterruptedException {
   
    //String[] fields = s.split("\t", -1)
    JSONObject jsn = new JSONObject();
        int sum = 0;
        for (Text value : values) {
        String[] vals = value.toString().split("\t");
        String[] targetNodes = vals[0].toString().split(",",-1);
        jsn.put("source",vals[1] );
        jsn.put("target",targetNodes);
            //sum += value.get();
        }
       // context.write(key, new Text(sum));
    }
}

I want to save that json to hdfs?

It was very trivial in hadoop streaming.. but how do i do it in hadoop java?
Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Reducer to output only json

Mohammad Tariq
If you need to save the JSON as it is then you could implement OutputFormat to create you custom outputformat that'll allow you to write the data as per your wish.

Warm Regards,
Tariq


On Tue, Jun 4, 2013 at 11:39 PM, Chengi Liu <[hidden email]> wrote:
Hi, 

 I have the following redcuer class

public static class TokenCounterReducer
    extends Reducer<Text, Text, Text, Text> {
    public void reduce(Text key, Iterable<Text> values, Context context)
        throws IOException, InterruptedException {
   
    //String[] fields = s.split("\t", -1)
    JSONObject jsn = new JSONObject();
        int sum = 0;
        for (Text value : values) {
        String[] vals = value.toString().split("\t");
        String[] targetNodes = vals[0].toString().split(",",-1);
        jsn.put("source",vals[1] );
        jsn.put("target",targetNodes);
            //sum += value.get();
        }
       // context.write(key, new Text(sum));
    }
}

I want to save that json to hdfs?

It was very trivial in hadoop streaming.. but how do i do it in hadoop java?
Thanks

Reply | Threaded
Open this post in threaded view
|

Re: Reducer to output only json

Niels Basjes
In reply to this post by Chengi Liu

Have you tried something like this (i do not have a pc here to check this code)

context.write(NullWritable, new Text(jsn.toString()));

On Jun 4, 2013 8:10 PM, "Chengi Liu" <[hidden email]> wrote:
Hi, 

 I have the following redcuer class

public static class TokenCounterReducer
    extends Reducer<Text, Text, Text, Text> {
    public void reduce(Text key, Iterable<Text> values, Context context)
        throws IOException, InterruptedException {
   
    //String[] fields = s.split("\t", -1)
    JSONObject jsn = new JSONObject();
        int sum = 0;
        for (Text value : values) {
        String[] vals = value.toString().split("\t");
        String[] targetNodes = vals[0].toString().split(",",-1);
        jsn.put("source",vals[1] );
        jsn.put("target",targetNodes);
            //sum += value.get();
        }
       // context.write(key, new Text(sum));
    }
}

I want to save that json to hdfs?

It was very trivial in hadoop streaming.. but how do i do it in hadoop java?
Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Reducer to output only json

Mohammad Tariq
Yes...This should do the trick.

Warm Regards,
Tariq


On Wed, Jun 5, 2013 at 1:38 AM, Niels Basjes <[hidden email]> wrote:

Have you tried something like this (i do not have a pc here to check this code)

context.write(NullWritable, new Text(jsn.toString()));

On Jun 4, 2013 8:10 PM, "Chengi Liu" <[hidden email]> wrote:
Hi, 

 I have the following redcuer class

public static class TokenCounterReducer
    extends Reducer<Text, Text, Text, Text> {
    public void reduce(Text key, Iterable<Text> values, Context context)
        throws IOException, InterruptedException {
   
    //String[] fields = s.split("\t", -1)
    JSONObject jsn = new JSONObject();
        int sum = 0;
        for (Text value : values) {
        String[] vals = value.toString().split("\t");
        String[] targetNodes = vals[0].toString().split(",",-1);
        jsn.put("source",vals[1] );
        jsn.put("target",targetNodes);
            //sum += value.get();
        }
       // context.write(key, new Text(sum));
    }
}

I want to save that json to hdfs?

It was very trivial in hadoop streaming.. but how do i do it in hadoop java?
Thanks

Reply | Threaded
Open this post in threaded view
|

Re: Reducer to output only json

Shahab Yunus
Chengi,

You can also see this for pointers:
http://java.dzone.com/articles/hadoop-practice

Regards,
Shahab


On Tue, Jun 4, 2013 at 4:15 PM, Mohammad Tariq <[hidden email]> wrote:
Yes...This should do the trick.

Warm Regards,
Tariq


On Wed, Jun 5, 2013 at 1:38 AM, Niels Basjes <[hidden email]> wrote:

Have you tried something like this (i do not have a pc here to check this code)

context.write(NullWritable, new Text(jsn.toString()));

On Jun 4, 2013 8:10 PM, "Chengi Liu" <[hidden email]> wrote:
Hi, 

 I have the following redcuer class

public static class TokenCounterReducer
    extends Reducer<Text, Text, Text, Text> {
    public void reduce(Text key, Iterable<Text> values, Context context)
        throws IOException, InterruptedException {
   
    //String[] fields = s.split("\t", -1)
    JSONObject jsn = new JSONObject();
        int sum = 0;
        for (Text value : values) {
        String[] vals = value.toString().split("\t");
        String[] targetNodes = vals[0].toString().split(",",-1);
        jsn.put("source",vals[1] );
        jsn.put("target",targetNodes);
            //sum += value.get();
        }
       // context.write(key, new Text(sum));
    }
}

I want to save that json to hdfs?

It was very trivial in hadoop streaming.. but how do i do it in hadoop java?
Thanks