Pig reduce :: NullPointerException

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Pig reduce :: NullPointerException

Bhupesh Bansal-3
I am trying to use SubGroup and Flatten constructs, I get this Null pointer exception

Queries:

table1 = Load 'datafile' using PigStorage(' ') as (sid, eid, mid, pid, did, fid);
table2  = Load 'datafile2' using PigStorage(' ') as (src_d, dest_d, score_d);

table3 = COGROUP table1 by did INNER, table2 by src_d INNER;
table4 = FOREACH table3 GENERATE FLATTEN(table1), FLATTEN(table2);
store table4 'table4.txt'

Error in File : /task_0002_r_000006_1/stderr/part-000000

main: Executing driver on (100310770, {(100310770, 42611676, 101703187, 54, 61661798, 579874705)[0], (100310770, 492859788, 301826791, 16573, 61661798, 579874705)[0], (100310770, 91343671, 68965026, 26215, 790, 1705)[0], (100310770,
138450842, 112796472, 29238, 61661798, 579874705)[0]}, {(100310770, 138450842, 112796472, 29238, 61661798, 579874705)[1], (100310770, 492859788, 301826791, 16573, 61661798, 579874705)[1], (100310770, 91343671, 68965026, 26215, 790, 1
705)[1], (100310770, 42611676, 101703187, 54, 61661798, 579874705)[1]})
java.lang.NullPointerException
        at com.yahoo.pig.impl.eval.EvalItemList.eval(EvalItemList.java:292)
        at com.yahoo.pig.impl.eval.EvalItemList.access$000(EvalItemList.java:22)
        at com.yahoo.pig.impl.eval.EvalItemList$1.add(EvalItemList.java:75)
        at com.yahoo.pig.impl.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:133)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:313)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
Reply | Threaded
Open this post in threaded view
|

Re: Pig reduce :: NullPointerException

Alan Gates
It's hard to debug this at the moment, as the current code that handles
this has been entirely rewritten.  Pig is in the process of becoming an
incubator project in Apache (see
http://wiki.apache.org/incubator/PigProposal).  Once that is out you
will have access to pig code that fixes this bug.

Alan.

bhupesh bansal wrote:

> I am trying to use SubGroup and Flatten constructs, I get this Null pointer
> exception
>
> Queries:
>
> table1 = Load 'datafile' using PigStorage(' ') as (sid, eid, mid, pid, did,
> fid);
> table2  = Load 'datafile2' using PigStorage(' ') as (src_d, dest_d,
> score_d);
>
> table3 = COGROUP table1 by did INNER, table2 by src_d INNER;
> table4 = FOREACH table3 GENERATE FLATTEN(table1), FLATTEN(table2);
> store table4 'table4.txt'
>
> Error in File : /task_0002_r_000006_1/stderr/part-000000
>
> main: Executing driver on (100310770, {(100310770, 42611676, 101703187, 54,
> 61661798, 579874705)[0], (100310770, 492859788, 301826791, 16573, 61661798,
> 579874705)[0], (100310770, 91343671, 68965026, 26215, 790, 1705)[0],
> (100310770,
> 138450842, 112796472, 29238, 61661798, 579874705)[0]}, {(100310770,
> 138450842, 112796472, 29238, 61661798, 579874705)[1], (100310770, 492859788,
> 301826791, 16573, 61661798, 579874705)[1], (100310770, 91343671, 68965026,
> 26215, 790, 1
> 705)[1], (100310770, 42611676, 101703187, 54, 61661798, 579874705)[1]})
> java.lang.NullPointerException
>         at com.yahoo.pig.impl.eval.EvalItemList.eval(EvalItemList.java:292)
>         at
> com.yahoo.pig.impl.eval.EvalItemList.access$000(EvalItemList.java:22)
>         at com.yahoo.pig.impl.eval.EvalItemList$1.add(EvalItemList.java:75)
>         at
> com.yahoo.pig.impl.mapreduceExec.PigMapReduce.reduce(PigMapReduce.java:133)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:313)
>         at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
>