TolerantUpdateProcessorFactory maxErrors=-1 issue

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

TolerantUpdateProcessorFactory maxErrors=-1 issue

Derek Poh
Hi

I am using CSV formatted indexupdates to index on tab delimited file.

I have define "TolerantUpdateProcessorFactory" with "maxErrors=-1" in
the solrconfig.xml to skip any document update error and proceed to
update the remaining documents without failing.
Howeverit does not seemto be workingas there is an document in the tab
delimited file withadditional number of fields and this caused the
indexing to abort instead.

This is how I start the indexing,
curl -o /apps/search/logs/indexing.log
"http://localhost:8983/solr/$collection/update?update.chain=$updateChainName&commit=true&separator=%09&encapsulator=^&fieldnames=$fieldnames$splitOptions"
--data-binary "@/apps/search/feed/$csvFilePath/$csvFileName" -H
'Content-type:application/csv'

This is how the TolerantUpdateProcessorFactory is defined in the
solrconfig.xml,
<updateRequestProcessorChain name="exhibitor-product-chain">
   <processor class="solr.CloneFieldUpdateProcessorFactory">
     <str name="source">P_SupplierId</str>
     <str name="source">P_TradeShowId</str>
     <str name="source">P_ProductId</str>
     <str name="dest">id</str>
   </processor>
   <processor class="solr.ConcatFieldUpdateProcessorFactory">
     <str name="fieldName">id</str>
     <str name="delimiter"></str>
   </processor>
   <processor class="solr.TolerantUpdateProcessorFactory">
      <int name="maxErrors">-1</int>
   </processor>
   <processor class="solr.processor.DocExpirationUpdateProcessorFactory">
     <null name="ttlFieldName"/>
     <null name="ttlParamName"/>
     <int name="autoDeletePeriodSeconds">43200</int>
     <str name="expirationFieldName">P_TradeShowOnlineEndDateUTC</str>
   </processor>
   <processor class="solr.LogUpdateProcessorFactory" />
   <processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>

Solr version is 6.6.2.

Derek

----------------------
CONFIDENTIALITY NOTICE

This e-mail (including any attachments) may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error, please inform the sender immediately and delete this e-mail (including any attachments) from your computer, and you must not use, disclose to anyone else or copy this e-mail (including any attachments), whether in whole or in part.

This e-mail and any reply to it may be monitored for security, legal, regulatory compliance and/or other appropriate reasons.
Reply | Threaded
Open this post in threaded view
|

Re: TolerantUpdateProcessorFactory maxErrors=-1 issue

Derek Poh
In addition, I tried withmaxErrors=3 and with only 1error document, the
indexing process still gets aborted.

Could it be the way I defined the TolerantUpdateProcessorFactory in
solrconfg.xml?

On 18/9/2018 3:13 PM, Derek Poh wrote:

> Hi
>
> I am using CSV formatted indexupdates to index on tab delimited file.
>
> I have define "TolerantUpdateProcessorFactory" with "maxErrors=-1" in
> the solrconfig.xml to skip any document update error and proceed to
> update the remaining documents without failing.
> Howeverit does not seemto be workingas there is an document in the tab
> delimited file withadditional number of fields and this caused the
> indexing to abort instead.
>
> This is how I start the indexing,
> curl -o /apps/search/logs/indexing.log
> "http://localhost:8983/solr/$collection/update?update.chain=$updateChainName&commit=true&separator=%09&encapsulator=^&fieldnames=$fieldnames$splitOptions"
> --data-binary "@/apps/search/feed/$csvFilePath/$csvFileName" -H
> 'Content-type:application/csv'
>
> This is how the TolerantUpdateProcessorFactory is defined in the
> solrconfig.xml,
> <updateRequestProcessorChain name="exhibitor-product-chain">
>   <processor class="solr.CloneFieldUpdateProcessorFactory">
>     <str name="source">P_SupplierId</str>
>     <str name="source">P_TradeShowId</str>
>     <str name="source">P_ProductId</str>
>     <str name="dest">id</str>
>   </processor>
>   <processor class="solr.ConcatFieldUpdateProcessorFactory">
>     <str name="fieldName">id</str>
>     <str name="delimiter"></str>
>   </processor>
>   <processor class="solr.TolerantUpdateProcessorFactory">
>      <int name="maxErrors">-1</int>
>   </processor>
>   <processor class="solr.processor.DocExpirationUpdateProcessorFactory">
>     <null name="ttlFieldName"/>
>     <null name="ttlParamName"/>
>     <int name="autoDeletePeriodSeconds">43200</int>
>     <str name="expirationFieldName">P_TradeShowOnlineEndDateUTC</str>
>   </processor>
>   <processor class="solr.LogUpdateProcessorFactory" />
>   <processor class="solr.RunUpdateProcessorFactory" />
> </updateRequestProcessorChain>
>
> Solr version is 6.6.2.
>
> Derek
>
> ----------------------
> CONFIDENTIALITY NOTICE
> This e-mail (including any attachments) may contain confidential
> and/or privileged information. If you are not the intended recipient
> or have received this e-mail in error, please inform the sender
> immediately and delete this e-mail (including any attachments) from
> your computer, and you must not use, disclose to anyone else or copy
> this e-mail (including any attachments), whether in whole or in part.
> This e-mail and any reply to it may be monitored for security, legal,
> regulatory compliance and/or other appropriate reasons.


----------------------
CONFIDENTIALITY NOTICE

This e-mail (including any attachments) may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error, please inform the sender immediately and delete this e-mail (including any attachments) from your computer, and you must not use, disclose to anyone else or copy this e-mail (including any attachments), whether in whole or in part.

This e-mail and any reply to it may be monitored for security, legal, regulatory compliance and/or other appropriate reasons.
Reply | Threaded
Open this post in threaded view
|

Re: TolerantUpdateProcessorFactory maxErrors=-1 issue

Derek Poh
Does any one have any idea whatcould be the causeof this?

On 19/9/2018 11:40 AM, Derek Poh wrote:

> In addition, I tried withmaxErrors=3 and with only 1error document,
> the indexing process still gets aborted.
>
> Could it be the way I defined the TolerantUpdateProcessorFactory in
> solrconfg.xml?
>
> On 18/9/2018 3:13 PM, Derek Poh wrote:
>> Hi
>>
>> I am using CSV formatted indexupdates to index on tab delimited file.
>>
>> I have define "TolerantUpdateProcessorFactory" with "maxErrors=-1" in
>> the solrconfig.xml to skip any document update error and proceed to
>> update the remaining documents without failing.
>> Howeverit does not seemto be workingas there is an document in the
>> tab delimited file withadditional number of fields and this caused
>> the indexing to abort instead.
>>
>> This is how I start the indexing,
>> curl -o /apps/search/logs/indexing.log
>> "http://localhost:8983/solr/$collection/update?update.chain=$updateChainName&commit=true&separator=%09&encapsulator=^&fieldnames=$fieldnames$splitOptions"
>> --data-binary "@/apps/search/feed/$csvFilePath/$csvFileName" -H
>> 'Content-type:application/csv'
>>
>> This is how the TolerantUpdateProcessorFactory is defined in the
>> solrconfig.xml,
>> <updateRequestProcessorChain name="exhibitor-product-chain">
>>   <processor class="solr.CloneFieldUpdateProcessorFactory">
>>     <str name="source">P_SupplierId</str>
>>     <str name="source">P_TradeShowId</str>
>>     <str name="source">P_ProductId</str>
>>     <str name="dest">id</str>
>>   </processor>
>>   <processor class="solr.ConcatFieldUpdateProcessorFactory">
>>     <str name="fieldName">id</str>
>>     <str name="delimiter"></str>
>>   </processor>
>>   <processor class="solr.TolerantUpdateProcessorFactory">
>>      <int name="maxErrors">-1</int>
>>   </processor>
>>   <processor class="solr.processor.DocExpirationUpdateProcessorFactory">
>>     <null name="ttlFieldName"/>
>>     <null name="ttlParamName"/>
>>     <int name="autoDeletePeriodSeconds">43200</int>
>>     <str name="expirationFieldName">P_TradeShowOnlineEndDateUTC</str>
>>   </processor>
>>   <processor class="solr.LogUpdateProcessorFactory" />
>>   <processor class="solr.RunUpdateProcessorFactory" />
>> </updateRequestProcessorChain>
>>
>> Solr version is 6.6.2.
>>
>> Derek
>>
>> ----------------------
>> CONFIDENTIALITY NOTICE
>> This e-mail (including any attachments) may contain confidential
>> and/or privileged information. If you are not the intended recipient
>> or have received this e-mail in error, please inform the sender
>> immediately and delete this e-mail (including any attachments) from
>> your computer, and you must not use, disclose to anyone else or copy
>> this e-mail (including any attachments), whether in whole or in part.
>> This e-mail and any reply to it may be monitored for security, legal,
>> regulatory compliance and/or other appropriate reasons.
>
>
> ----------------------
> CONFIDENTIALITY NOTICE
> This e-mail (including any attachments) may contain confidential
> and/or privileged information. If you are not the intended recipient
> or have received this e-mail in error, please inform the sender
> immediately and delete this e-mail (including any attachments) from
> your computer, and you must not use, disclose to anyone else or copy
> this e-mail (including any attachments), whether in whole or in part.
> This e-mail and any reply to it may be monitored for security, legal,
> regulatory compliance and/or other appropriate reasons.


----------------------
CONFIDENTIALITY NOTICE

This e-mail (including any attachments) may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error, please inform the sender immediately and delete this e-mail (including any attachments) from your computer, and you must not use, disclose to anyone else or copy this e-mail (including any attachments), whether in whole or in part.

This e-mail and any reply to it may be monitored for security, legal, regulatory compliance and/or other appropriate reasons.
Reply | Threaded
Open this post in threaded view
|

Re: TolerantUpdateProcessorFactory maxErrors=-1 issue

Tomás Fernández Löbbe
Hi Derek,
I suspect you need to move the TolerantUpdateProcessorFactory to the
beginning of the chain

On Thu, Sep 20, 2018 at 6:17 PM Derek Poh <[hidden email]> wrote:

> Does any one have any idea whatcould be the causeof this?
>
> On 19/9/2018 11:40 AM, Derek Poh wrote:
> > In addition, I tried withmaxErrors=3 and with only 1error document,
> > the indexing process still gets aborted.
> >
> > Could it be the way I defined the TolerantUpdateProcessorFactory in
> > solrconfg.xml?
> >
> > On 18/9/2018 3:13 PM, Derek Poh wrote:
> >> Hi
> >>
> >> I am using CSV formatted indexupdates to index on tab delimited file.
> >>
> >> I have define "TolerantUpdateProcessorFactory" with "maxErrors=-1" in
> >> the solrconfig.xml to skip any document update error and proceed to
> >> update the remaining documents without failing.
> >> Howeverit does not seemto be workingas there is an document in the
> >> tab delimited file withadditional number of fields and this caused
> >> the indexing to abort instead.
> >>
> >> This is how I start the indexing,
> >> curl -o /apps/search/logs/indexing.log
> >> "
> http://localhost:8983/solr/$collection/update?update.chain=$updateChainName&commit=true&separator=%09&encapsulator=^&fieldnames=$fieldnames$splitOptions"
>
> >> --data-binary "@/apps/search/feed/$csvFilePath/$csvFileName" -H
> >> 'Content-type:application/csv'
> >>
> >> This is how the TolerantUpdateProcessorFactory is defined in the
> >> solrconfig.xml,
> >> <updateRequestProcessorChain name="exhibitor-product-chain">
> >>   <processor class="solr.CloneFieldUpdateProcessorFactory">
> >>     <str name="source">P_SupplierId</str>
> >>     <str name="source">P_TradeShowId</str>
> >>     <str name="source">P_ProductId</str>
> >>     <str name="dest">id</str>
> >>   </processor>
> >>   <processor class="solr.ConcatFieldUpdateProcessorFactory">
> >>     <str name="fieldName">id</str>
> >>     <str name="delimiter"></str>
> >>   </processor>
> >>   <processor class="solr.TolerantUpdateProcessorFactory">
> >>      <int name="maxErrors">-1</int>
> >>   </processor>
> >>   <processor class="solr.processor.DocExpirationUpdateProcessorFactory">
> >>     <null name="ttlFieldName"/>
> >>     <null name="ttlParamName"/>
> >>     <int name="autoDeletePeriodSeconds">43200</int>
> >>     <str name="expirationFieldName">P_TradeShowOnlineEndDateUTC</str>
> >>   </processor>
> >>   <processor class="solr.LogUpdateProcessorFactory" />
> >>   <processor class="solr.RunUpdateProcessorFactory" />
> >> </updateRequestProcessorChain>
> >>
> >> Solr version is 6.6.2.
> >>
> >> Derek
> >>
> >> ----------------------
> >> CONFIDENTIALITY NOTICE
> >> This e-mail (including any attachments) may contain confidential
> >> and/or privileged information. If you are not the intended recipient
> >> or have received this e-mail in error, please inform the sender
> >> immediately and delete this e-mail (including any attachments) from
> >> your computer, and you must not use, disclose to anyone else or copy
> >> this e-mail (including any attachments), whether in whole or in part.
> >> This e-mail and any reply to it may be monitored for security, legal,
> >> regulatory compliance and/or other appropriate reasons.
> >
> >
> > ----------------------
> > CONFIDENTIALITY NOTICE
> > This e-mail (including any attachments) may contain confidential
> > and/or privileged information. If you are not the intended recipient
> > or have received this e-mail in error, please inform the sender
> > immediately and delete this e-mail (including any attachments) from
> > your computer, and you must not use, disclose to anyone else or copy
> > this e-mail (including any attachments), whether in whole or in part.
> > This e-mail and any reply to it may be monitored for security, legal,
> > regulatory compliance and/or other appropriate reasons.
>
>
> ----------------------
> CONFIDENTIALITY NOTICE
>
> This e-mail (including any attachments) may contain confidential and/or
> privileged information. If you are not the intended recipient or have
> received this e-mail in error, please inform the sender immediately and
> delete this e-mail (including any attachments) from your computer, and you
> must not use, disclose to anyone else or copy this e-mail (including any
> attachments), whether in whole or in part.
>
> This e-mail and any reply to it may be monitored for security, legal,
> regulatory compliance and/or other appropriate reasons.
Reply | Threaded
Open this post in threaded view
|

Re: TolerantUpdateProcessorFactory maxErrors=-1 issue

Derek Poh
Hi Tomas

I moved TolerantUpdateProcessorFactoryto the beginning of the chain,
reload the collection.
The indexing process still abort.

On 22/9/2018 4:28 AM, Tomás Fernández Löbbe wrote:

> Hi Derek,
> I suspect you need to move the TolerantUpdateProcessorFactory to the
> beginning of the chain
>
> On Thu, Sep 20, 2018 at 6:17 PM Derek Poh <[hidden email]> wrote:
>
>> Does any one have any idea whatcould be the causeof this?
>>
>> On 19/9/2018 11:40 AM, Derek Poh wrote:
>>> In addition, I tried withmaxErrors=3 and with only 1error document,
>>> the indexing process still gets aborted.
>>>
>>> Could it be the way I defined the TolerantUpdateProcessorFactory in
>>> solrconfg.xml?
>>>
>>> On 18/9/2018 3:13 PM, Derek Poh wrote:
>>>> Hi
>>>>
>>>> I am using CSV formatted indexupdates to index on tab delimited file.
>>>>
>>>> I have define "TolerantUpdateProcessorFactory" with "maxErrors=-1" in
>>>> the solrconfig.xml to skip any document update error and proceed to
>>>> update the remaining documents without failing.
>>>> Howeverit does not seemto be workingas there is an document in the
>>>> tab delimited file withadditional number of fields and this caused
>>>> the indexing to abort instead.
>>>>
>>>> This is how I start the indexing,
>>>> curl -o /apps/search/logs/indexing.log
>>>> "
>> http://localhost:8983/solr/$collection/update?update.chain=$updateChainName&commit=true&separator=%09&encapsulator=^&fieldnames=$fieldnames$splitOptions"
>>
>>>> --data-binary "@/apps/search/feed/$csvFilePath/$csvFileName" -H
>>>> 'Content-type:application/csv'
>>>>
>>>> This is how the TolerantUpdateProcessorFactory is defined in the
>>>> solrconfig.xml,
>>>> <updateRequestProcessorChain name="exhibitor-product-chain">
>>>>    <processor class="solr.CloneFieldUpdateProcessorFactory">
>>>>      <str name="source">P_SupplierId</str>
>>>>      <str name="source">P_TradeShowId</str>
>>>>      <str name="source">P_ProductId</str>
>>>>      <str name="dest">id</str>
>>>>    </processor>
>>>>    <processor class="solr.ConcatFieldUpdateProcessorFactory">
>>>>      <str name="fieldName">id</str>
>>>>      <str name="delimiter"></str>
>>>>    </processor>
>>>>    <processor class="solr.TolerantUpdateProcessorFactory">
>>>>       <int name="maxErrors">-1</int>
>>>>    </processor>
>>>>    <processor class="solr.processor.DocExpirationUpdateProcessorFactory">
>>>>      <null name="ttlFieldName"/>
>>>>      <null name="ttlParamName"/>
>>>>      <int name="autoDeletePeriodSeconds">43200</int>
>>>>      <str name="expirationFieldName">P_TradeShowOnlineEndDateUTC</str>
>>>>    </processor>
>>>>    <processor class="solr.LogUpdateProcessorFactory" />
>>>>    <processor class="solr.RunUpdateProcessorFactory" />
>>>> </updateRequestProcessorChain>
>>>>
>>>> Solr version is 6.6.2.
>>>>
>>>> Derek
>>>>
>>>> ----------------------
>>>> CONFIDENTIALITY NOTICE
>>>> This e-mail (including any attachments) may contain confidential
>>>> and/or privileged information. If you are not the intended recipient
>>>> or have received this e-mail in error, please inform the sender
>>>> immediately and delete this e-mail (including any attachments) from
>>>> your computer, and you must not use, disclose to anyone else or copy
>>>> this e-mail (including any attachments), whether in whole or in part.
>>>> This e-mail and any reply to it may be monitored for security, legal,
>>>> regulatory compliance and/or other appropriate reasons.
>>>
>>> ----------------------
>>> CONFIDENTIALITY NOTICE
>>> This e-mail (including any attachments) may contain confidential
>>> and/or privileged information. If you are not the intended recipient
>>> or have received this e-mail in error, please inform the sender
>>> immediately and delete this e-mail (including any attachments) from
>>> your computer, and you must not use, disclose to anyone else or copy
>>> this e-mail (including any attachments), whether in whole or in part.
>>> This e-mail and any reply to it may be monitored for security, legal,
>>> regulatory compliance and/or other appropriate reasons.
>>
>> ----------------------
>> CONFIDENTIALITY NOTICE
>>
>> This e-mail (including any attachments) may contain confidential and/or
>> privileged information. If you are not the intended recipient or have
>> received this e-mail in error, please inform the sender immediately and
>> delete this e-mail (including any attachments) from your computer, and you
>> must not use, disclose to anyone else or copy this e-mail (including any
>> attachments), whether in whole or in part.
>>
>> This e-mail and any reply to it may be monitored for security, legal,
>> regulatory compliance and/or other appropriate reasons.


----------------------
CONFIDENTIALITY NOTICE

This e-mail (including any attachments) may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error, please inform the sender immediately and delete this e-mail (including any attachments) from your computer, and you must not use, disclose to anyone else or copy this e-mail (including any attachments), whether in whole or in part.

This e-mail and any reply to it may be monitored for security, legal, regulatory compliance and/or other appropriate reasons.