Issues while running map-reduce index tool.

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Issues while running map-reduce index tool.

Sujeet Singh
Hi Team,

I am trying to index a document from HDFS in version solr 4.9  and getting below error:

Command used by me :
QUEUE_NAME=default
MORPHLINE_CONF=/home/sshuser/solrruncls/morphline_retail_tr_2017.conf
OUTPUT_DIR=adl:///solr
ZK_HOST=zk0-hnrhba.thxycti2fi4ejjab25g3ibtkog.gx.internal.cloudapp.net:2181,zk2-hnrhba.thxycti2fi4ejjab25g3ibtkog.gx.internal.cloudapp.net:2181
SOLR_URI=hn0-hnrhba.thxycti2fi4ejjab25g3ibtkog.gx.internal.cloudapp.net:8983/solr
INPUT_DIR=adl:///solrinput
source /usr/hdp/current/solr/example/scripts/map-reduce/set-map-reduce-classpath.sh
hadoop jar /usr/hdp/current/solr/dist/solr-map-reduce-*.jar -D 'mapred.child.java.opts=-Xmx500m' -libjars "$HADOOP_LIBJAR" --morphline-file $MORPHLINE_CONF --zk-host $ZK_HOST --output-dir $OUTPUT_DIR --collection collection1 --go-live --verbose $INPUT_DIR



Error: java.lang.IllegalStateException: Failed to initialize record writer for org.apache.solr.hadoop.MapReduceIndexerTool/MorphlineMapper, attempt_1528778576449_0013_r_000000_5
        at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:127)
        at org.apache.solr.hadoop.SolrOutputFormat.getRecordWriter(SolrOutputFormat.java:162)
        at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:541)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: org.apache.solr.common.SolrException: Unable to create core: core1
        at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:911)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:568)
        at org.apache.solr.hadoop.SolrRecordWriter.createEmbeddedSolrServer(SolrRecordWriter.java:171)
        at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:120)
        ... 9 more
Caused by: org.apache.solr.common.SolrException: You must set the HdfsDirectoryFactory param solr.hdfs.home for relative dataDir paths to work
        at org.apache.solr.core.HdfsDirectoryFactory.getDataHome(HdfsDirectoryFactory.java:263)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:696)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:643)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:556)
        ... 11 more

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143.

18/06/12 11:53:36 INFO mapreduce.Job: Task Id : attempt_1528778576449_0013_r_000000_6, Status : FAILED
Error: java.lang.IllegalStateException: Failed to initialize record writer for org.apache.solr.hadoop.MapReduceIndexerTool/MorphlineMapper, attempt_1528778576449_0013_r_000000_6
        at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:127)
        at org.apache.solr.hadoop.SolrOutputFormat.getRecordWriter(SolrOutputFormat.java:162)
        at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:541)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: org.apache.solr.common.SolrException: Unable to create core: core1
        at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:911)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:568)
        at org.apache.solr.hadoop.SolrRecordWriter.createEmbeddedSolrServer(SolrRecordWriter.java:171)
        at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:120)
        ... 9 more
Caused by: org.apache.solr.common.SolrException: You must set the HdfsDirectoryFactory param solr.hdfs.home for relative dataDir paths to work
        at org.apache.solr.core.HdfsDirectoryFactory.getDataHome(HdfsDirectoryFactory.java:263)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:696)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:643)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:556)
        ... 11 more

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143.


Regards,
________________________________
Sujeet Singh | Sr. Software Analyst | cloudmoyo | E. [hidden email]<mailto:[hidden email]> | M. +91 9860586055

[CloudMoyo Logo]<http://www.cloudmoyo.com/?utm_source=signature&utm_medium=mail%20sign&utm_campaign=mail%20sign>
[https://icertisportalstorage.blob.core.windows.net/siteasset/icon-linkedin.png]<https://www.linkedin.com/company/cloudmoyo>[https://icertisportalstorage.blob.core.windows.net/siteasset/icon-fb.png]<https://www.facebook.com/gocloudmoyo>[https://icertisportalstorage.blob.core.windows.net/siteasset/icon-twitter.png]<https://twitter.com/GoCloudMoyo>
www.cloudmoyo.com





IMPORTANT NOTICE: This communication, including any attachment, contains information that may be confidential or privileged, and is intended solely for the entity or individual to whom it is addressed. If you are not the intended recipient, you should delete this message and are hereby notified that any disclosure, copying, or distribution of this message is strictly prohibited. Nothing in this email, including any attachment, is intended to be a legally binding signature.