Unable to index on Hadoop 3.2.0 with 1.16

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Unable to index on Hadoop 3.2.0 with 1.16

Markus Jelsma-2
Hello,

We're upgrading our stuff to 1.16 and got a peculiar problem when we started indexing:

2019-10-14 13:50:30,586 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalStateException: text width is less than 1, was <-41>
        at org.apache.commons.lang3.Validate.validState(Validate.java:829)
        at de.vandermeer.skb.interfaces.transformers.textformat.Text_To_FormattedText.transform(Text_To_FormattedText.java:215)
        at de.vandermeer.asciitable.AT_Renderer.renderAsCollection(AT_Renderer.java:250)
        at de.vandermeer.asciitable.AT_Renderer.render(AT_Renderer.java:128)
        at de.vandermeer.asciitable.AsciiTable.render(AsciiTable.java:191)
        at org.apache.nutch.indexer.IndexWriters.describe(IndexWriters.java:326)
        at org.apache.nutch.indexer.IndexerOutputFormat.getRecordWriter(IndexerOutputFormat.java:45)
        at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:542)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:615)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

The only IndexWriter we use is SolrIndexer, and locally everything is just fine.

Any thoughts?

Thanks,
Markus
Reply | Threaded
Open this post in threaded view
|

Re: Unable to index on Hadoop 3.2.0 with 1.16

Sebastian Nagel-2
Hi Markus,

I've tested in pseudo-distributed mode with Hadoop 3.2.1,
including indexing into Solr. It worked.

Could be a dependency version issue similar to that
causing NUTCH-2706. But that's only an assumption.

Since the IndexWriters.describe() is for help only,
I would just deactivate this method and open an issue to
investigate the reason. Need also to think when and where
to output the index writer options. Maybe better call
the describe() methods of the indexer plugins explicitly
via IndexingJob --help or similar.

Best,
Sebastian

On 14.10.19 17:08, Markus Jelsma wrote:

> Hello,
>
> We're upgrading our stuff to 1.16 and got a peculiar problem when we started indexing:
>
> 2019-10-14 13:50:30,586 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalStateException: text width is less than 1, was <-41>
> at org.apache.commons.lang3.Validate.validState(Validate.java:829)
> at de.vandermeer.skb.interfaces.transformers.textformat.Text_To_FormattedText.transform(Text_To_FormattedText.java:215)
> at de.vandermeer.asciitable.AT_Renderer.renderAsCollection(AT_Renderer.java:250)
> at de.vandermeer.asciitable.AT_Renderer.render(AT_Renderer.java:128)
> at de.vandermeer.asciitable.AsciiTable.render(AsciiTable.java:191)
> at org.apache.nutch.indexer.IndexWriters.describe(IndexWriters.java:326)
> at org.apache.nutch.indexer.IndexerOutputFormat.getRecordWriter(IndexerOutputFormat.java:45)
> at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:542)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:615)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
>
> The only IndexWriter we use is SolrIndexer, and locally everything is just fine.
>
> Any thoughts?
>
> Thanks,
> Markus
>

Reply | Threaded
Open this post in threaded view
|

Re: Unable to index on Hadoop 3.2.0 with 1.16

Sebastian Nagel-2
In reply to this post by Markus Jelsma-2
Hi Markus,

any updates on this? Just to make sure the issue gets resolved.

Thanks,
Sebastian

On 14.10.19 17:08, Markus Jelsma wrote:

> Hello,
>
> We're upgrading our stuff to 1.16 and got a peculiar problem when we started indexing:
>
> 2019-10-14 13:50:30,586 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalStateException: text width is less than 1, was <-41>
> at org.apache.commons.lang3.Validate.validState(Validate.java:829)
> at de.vandermeer.skb.interfaces.transformers.textformat.Text_To_FormattedText.transform(Text_To_FormattedText.java:215)
> at de.vandermeer.asciitable.AT_Renderer.renderAsCollection(AT_Renderer.java:250)
> at de.vandermeer.asciitable.AT_Renderer.render(AT_Renderer.java:128)
> at de.vandermeer.asciitable.AsciiTable.render(AsciiTable.java:191)
> at org.apache.nutch.indexer.IndexWriters.describe(IndexWriters.java:326)
> at org.apache.nutch.indexer.IndexerOutputFormat.getRecordWriter(IndexerOutputFormat.java:45)
> at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:542)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:615)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
>
> The only IndexWriter we use is SolrIndexer, and locally everything is just fine.
>
> Any thoughts?
>
> Thanks,
> Markus
>