ConcurrentModificationException in SolrInputDocument writeMap

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

ConcurrentModificationException in SolrInputDocument writeMap

Tim Swetland
I'm currently running into a ConcurrentModificationException ingesting data
as we attempt to upgrade from Solr 8.1 to 8.2. It's not every document, but
it definitely appears regularly in our logs. We didn't run into this
problem in 8.1, so I'm not sure what might have changed. I feel like this
is probably a bug, but if there's a workaround or if there's an idea of
something I might be doing wrong, please let me know.

Stack trace:
o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
SolrCmdDistributor$Req: cmd=add{_version=<version>,id=<id>}; node=StdNode:
https://<server>/solr/coll_shard1_replica_n2/ to
https://<server>/solr/coll_shard1_replica_n2/
=> java.util.ConcurrentModificationException
    at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
java.util.ConcurrentModificationException: null
  at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
  at
org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)
  at
org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:658)
  at
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
  at
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
  at
org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:813)

  at
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:411)

  at
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
  at
org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:750)

  at
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:395)

  at
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
  at
org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:248)

  at
org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)

  at
org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
  at
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:167)
  at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:102)
  at
org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
  at
org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:338)

  at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:231)

  at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176)

  at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
  at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil
.java:209)
  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

  at java.lang.Thread.run(Thread.java:748)
Reply | Threaded
Open this post in threaded view
|

Re: ConcurrentModificationException in SolrInputDocument writeMap

Tim Swetland
Nevermind my comment on not having this problem in 8.1. We do have it there
as well, I just didn't look far enough back in our logs on my initial
search. Would still appreciate whatever thoughts anyone might have on the
exception.

On Wed, Nov 6, 2019 at 10:17 AM Tim Swetland <[hidden email]> wrote:

> I'm currently running into a ConcurrentModificationException ingesting
> data as we attempt to upgrade from Solr 8.1 to 8.2. It's not every
> document, but it definitely appears regularly in our logs. We didn't run
> into this problem in 8.1, so I'm not sure what might have changed. I feel
> like this is probably a bug, but if there's a workaround or if there's an
> idea of something I might be doing wrong, please let me know.
>
> Stack trace:
> o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> SolrCmdDistributor$Req: cmd=add{_version=<version>,id=<id>}; node=StdNode:
> https://<server>/solr/coll_shard1_replica_n2/ to https://<server>/solr/coll_shard1_replica_n2/
> => java.util.ConcurrentModificationException
>     at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> java.util.ConcurrentModificationException: null
>   at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
>   at
> org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:658)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:813)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:411)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:750)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:395)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:248)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
>   at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:167)
>   at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:102)
>   at
> org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
>   at
> org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:338)
>
>   at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:231)
>
>   at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176)
>
>   at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
>   at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil
> .java:209)
>   at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>   at java.lang.Thread.run(Thread.java:748)
>
>
Reply | Threaded
Open this post in threaded view
|

Re: ConcurrentModificationException in SolrInputDocument writeMap

Mikhail Khludnev-2
Hello, Tim.
Please confirm my understanding. Does exception happens in standalone Java
ingesting app?
If, it's so, Does it reuse either SolrInputDocument instances of
fields/values collections between update calls?

On Wed, Nov 6, 2019 at 8:00 AM Tim Swetland <[hidden email]> wrote:

> Nevermind my comment on not having this problem in 8.1. We do have it there
> as well, I just didn't look far enough back in our logs on my initial
> search. Would still appreciate whatever thoughts anyone might have on the
> exception.
>
> On Wed, Nov 6, 2019 at 10:17 AM Tim Swetland <[hidden email]> wrote:
>
> > I'm currently running into a ConcurrentModificationException ingesting
> > data as we attempt to upgrade from Solr 8.1 to 8.2. It's not every
> > document, but it definitely appears regularly in our logs. We didn't run
> > into this problem in 8.1, so I'm not sure what might have changed. I feel
> > like this is probably a bug, but if there's a workaround or if there's an
> > idea of something I might be doing wrong, please let me know.
> >
> > Stack trace:
> > o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> > SolrCmdDistributor$Req: cmd=add{_version=<version>,id=<id>};
> node=StdNode:
> > https://<server>/solr/coll_shard1_replica_n2/ to https://
> <server>/solr/coll_shard1_replica_n2/
> > => java.util.ConcurrentModificationException
> >     at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> > java.util.ConcurrentModificationException: null
> >   at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> >   at
> >
> org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:658)
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
> >   at
> > org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:813)
> >
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:411)
> >
> >   at
> > org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:750)
> >
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:395)
> >
> >   at
> > org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:248)
> >
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
> >
> >   at
> > org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> >   at
> > org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:167)
> >   at
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:102)
> >   at
> >
> org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
> >   at
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:338)
> >
> >   at
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:231)
> >
> >   at
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176)
> >
> >   at
> >
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> >   at
> >
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil
> > .java:209)
> >   at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >   at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >
> >   at java.lang.Thread.run(Thread.java:748)
> >
> >
>


--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: ConcurrentModificationException in SolrInputDocument writeMap

Edward Ribeiro
You probably hit
https://issues.apache.org/jira/projects/SOLR/issues/SOLR-8028


Regards,
Edward


Em qua, 6 de nov de 2019 13:23, Mikhail Khludnev <[hidden email]> escreveu:

> Hello, Tim.
> Please confirm my understanding. Does exception happens in standalone Java
> ingesting app?
> If, it's so, Does it reuse either SolrInputDocument instances of
> fields/values collections between update calls?
>
> On Wed, Nov 6, 2019 at 8:00 AM Tim Swetland <[hidden email]> wrote:
>
> > Nevermind my comment on not having this problem in 8.1. We do have it
> there
> > as well, I just didn't look far enough back in our logs on my initial
> > search. Would still appreciate whatever thoughts anyone might have on the
> > exception.
> >
> > On Wed, Nov 6, 2019 at 10:17 AM Tim Swetland <[hidden email]>
> wrote:
> >
> > > I'm currently running into a ConcurrentModificationException ingesting
> > > data as we attempt to upgrade from Solr 8.1 to 8.2. It's not every
> > > document, but it definitely appears regularly in our logs. We didn't
> run
> > > into this problem in 8.1, so I'm not sure what might have changed. I
> feel
> > > like this is probably a bug, but if there's a workaround or if there's
> an
> > > idea of something I might be doing wrong, please let me know.
> > >
> > > Stack trace:
> > > o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> > > SolrCmdDistributor$Req: cmd=add{_version=<version>,id=<id>};
> > node=StdNode:
> > > https://<server>/solr/coll_shard1_replica_n2/ to https://
> > <server>/solr/coll_shard1_replica_n2/
> > > => java.util.ConcurrentModificationException
> > >     at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> > > java.util.ConcurrentModificationException: null
> > >   at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> > >   at
> > >
> >
> org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:658)
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
> > >   at
> > >
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:813)
> > >
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:411)
> > >
> > >   at
> > >
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:750)
> > >
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:395)
> > >
> > >   at
> > >
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:248)
> > >
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
> > >
> > >   at
> > >
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> > >   at
> > > org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:167)
> > >   at
> > >
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:102)
> > >   at
> > >
> >
> org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
> > >   at
> > >
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:338)
> > >
> > >   at
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:231)
> > >
> > >   at
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176)
> > >
> > >   at
> > >
> >
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> > >   at
> > >
> >
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil
> > > .java:209)
> > >   at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >   at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >
> > >   at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
Reply | Threaded
Open this post in threaded view
|

Re: ConcurrentModificationException in SolrInputDocument writeMap

Shawn Heisey-2
In reply to this post by Tim Swetland
On 11/6/2019 8:17 AM, Tim Swetland wrote:

> I'm currently running into a ConcurrentModificationException ingesting data
> as we attempt to upgrade from Solr 8.1 to 8.2. It's not every document, but
> it definitely appears regularly in our logs. We didn't run into this
> problem in 8.1, so I'm not sure what might have changed. I feel like this
> is probably a bug, but if there's a workaround or if there's an idea of
> something I might be doing wrong, please let me know.
>
> Stack trace:
> o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> SolrCmdDistributor$Req: cmd=add{_version=<version>,id=<id>}; node=StdNode:
> https://<server>/solr/coll_shard1_replica_n2/ to
> https://<server>/solr/coll_shard1_replica_n2/
> => java.util.ConcurrentModificationException
>      at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> java.util.ConcurrentModificationException: null
>    at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
>    at
> org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)

This error, as mentioned in the SOLR-8028 issue linked by Edward
Ribeiro, sounds like you are re-using objects when you are building
SolrInputDocument instances for your indexing.

Looking at the actual code involved in the stacktrace, I think what's
happening is that at the same time SolrJ is converting a
SolrInputDocument object to the javabin format so it can be sent to
Solr, something else has modified that SolrInputDocument object.

You should never re-use SolrJ objects that you construct for indexing.
A brand new SolrInputDocument instance should be created every time you
need one, and any objects that go into its creation should also be new
objects.  This is especially important when the code is multi-threaded,
but there are certain code constructs that can cause this to happen even
in code that does not create multiple threads.  Also, code that uses
CloudSolrClient can very easily become multi-threaded even if the user's
code isn't.

In Solr 8.2, clients and the server were updated to use HTTP/2.  I did
not closely follow the work for this, but it would have required some
pretty major changes to SolrJ, changes that very well could have altered
the precise timing of operations.  Your additional note says you are
also seeing this problem with 8.1, which does not surprise me.  The
precise timing of the 8.1 code might have been such that the problem was
far less noticeable.

If you can share your code that uses SolrJ, we *might* be able to help
you narrow down what's happening and get it fixed.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: ConcurrentModificationException in SolrInputDocument writeMap

Tim Swetland
 <https://issues.apache.org/jira/browse/SOLR-13471>
Finally had a chance to get back to this and figured out what our problem
was. Thanks to Mikhail for pointing me in the right direction. For some
reason one of our custom UpdateRequestProcessors was calling
super.processAdd as the first line in our overridden processAdd method. It
looked something like this:

@Override
public void processAdd(AddUpdateCommand cmd) throws IOException {
  super.processAdd(cmd);

  SolrInputDocument doc = cmd.getSolrInputDocument();
  // Do stuff and set field in doc

  super.next.processAdd(cmd);
}

Once I got rid of the first line, everything worked fine. I suspect the
fact that we were going through a decent portion of the update request
processor chain twice was causing some unknown issues. The big issue
causing the ConcurrentModificationException, however, appeared to happen
when trying to distribute the update to a replica. It looks like it would
try and write out to the replica as we'd take the second trip through our
processor chain and thus two things would try and modify the
SolrInputDocument at the same time causing the exception. This would
additionally cause a ClassCastException (java.lang.String cannot be cast to
java.util.Map) on the replica as in issue SOLR-13471
<https://issues.apache.org/jira/browse/SOLR-13471>.

Anyway, thanks for the insight everyone,
Tim

On Fri, Nov 8, 2019 at 12:26 AM Shawn Heisey <[hidden email]> wrote:

> On 11/6/2019 8:17 AM, Tim Swetland wrote:
> > I'm currently running into a ConcurrentModificationException ingesting
> data
> > as we attempt to upgrade from Solr 8.1 to 8.2. It's not every document,
> but
> > it definitely appears regularly in our logs. We didn't run into this
> > problem in 8.1, so I'm not sure what might have changed. I feel like this
> > is probably a bug, but if there's a workaround or if there's an idea of
> > something I might be doing wrong, please let me know.
> >
> > Stack trace:
> > o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> > SolrCmdDistributor$Req: cmd=add{_version=<version>,id=<id>};
> node=StdNode:
> > https://<server>/solr/coll_shard1_replica_n2/ to
> > https://<server>/solr/coll_shard1_replica_n2/
> > => java.util.ConcurrentModificationException
> >      at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> > java.util.ConcurrentModificationException: null
> >    at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> >    at
> >
> org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)
>
> This error, as mentioned in the SOLR-8028 issue linked by Edward
> Ribeiro, sounds like you are re-using objects when you are building
> SolrInputDocument instances for your indexing.
>
> Looking at the actual code involved in the stacktrace, I think what's
> happening is that at the same time SolrJ is converting a
> SolrInputDocument object to the javabin format so it can be sent to
> Solr, something else has modified that SolrInputDocument object.
>
> You should never re-use SolrJ objects that you construct for indexing.
> A brand new SolrInputDocument instance should be created every time you
> need one, and any objects that go into its creation should also be new
> objects.  This is especially important when the code is multi-threaded,
> but there are certain code constructs that can cause this to happen even
> in code that does not create multiple threads.  Also, code that uses
> CloudSolrClient can very easily become multi-threaded even if the user's
> code isn't.
>
> In Solr 8.2, clients and the server were updated to use HTTP/2.  I did
> not closely follow the work for this, but it would have required some
> pretty major changes to SolrJ, changes that very well could have altered
> the precise timing of operations.  Your additional note says you are
> also seeing this problem with 8.1, which does not surprise me.  The
> precise timing of the 8.1 code might have been such that the problem was
> far less noticeable.
>
> If you can share your code that uses SolrJ, we *might* be able to help
> you narrow down what's happening and get it fixed.
>
> Thanks,
> Shawn
>