deleteById commitWithin question

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

deleteById commitWithin question

Jamie Johnson
I am running off of a snapshot taken 5/3/2012 of solr 4.0 and am
noticing some issues around deleteById when a commitWithin parameter
is included using SolrJ, specifically commit isn't executed.  If I
later just call commit on the solr instance I see the item is deleted
though.  Is anyone aware if this should work in that snapshot?
Reply | Threaded
Open this post in threaded view
|

Re: deleteById commitWithin question

Yonik Seeley-2-2
On Thu, Jul 5, 2012 at 4:29 PM, Jamie Johnson <[hidden email]> wrote:
> I am running off of a snapshot taken 5/3/2012 of solr 4.0 and am
> noticing some issues around deleteById when a commitWithin parameter
> is included using SolrJ, specifically commit isn't executed.  If I
> later just call commit on the solr instance I see the item is deleted
> though.  Is anyone aware if this should work in that snapshot?


I thought I remembered something like this... but looking at the
commit log for DUH2, I don't see it.

/opt/code/lusolr4$ svn log
./solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java
| less

------------------------------------------------------------------------
r1357332 | yonik | 2012-07-04 12:23:09 -0400 (Wed, 04 Jul 2012) | 1 line

log DBQ reordering events
------------------------------------------------------------------------
r1356858 | markrmiller | 2012-07-03 14:18:48 -0400 (Tue, 03 Jul 2012) | 1 line

SOLR-3587: After reloading a SolrCore, the original Analyzer is still
used rather than a new one
------------------------------------------------------------------------
r1356845 | yonik | 2012-07-03 13:47:56 -0400 (Tue, 03 Jul 2012) | 1 line

SOLR-3559: DBQ reorder support
------------------------------------------------------------------------
r1355088 | sarowe | 2012-06-28 13:51:38 -0400 (Thu, 28 Jun 2012) | 1 line

LUCENE-4172: clean up redundant throws clauses (merge from trunk)
------------------------------------------------------------------------
r1348984 | hossman | 2012-06-11 15:46:14 -0400 (Mon, 11 Jun 2012) | 1 line

LUCENE-3949: fix license headers to not be javadoc style comments
------------------------------------------------------------------------
r1343813 | rmuir | 2012-05-29 12:16:38 -0400 (Tue, 29 May 2012) | 1 line

create stable branch for 4.x releases
------------------------------------------------------------------------
r1328890 | yonik | 2012-04-22 11:01:55 -0400 (Sun, 22 Apr 2012) | 1 line

SOLR-3392: fix search leak when openSearcher=false
------------------------------------------------------------------------
r1328883 | yonik | 2012-04-22 09:58:00 -0400 (Sun, 22 Apr 2012) | 1 line

SOLR-3391: Make explicit commits cancel pending autocommits.


I'll try out trunk quick and see if it currently works.

-Yonik
http://lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

Re: deleteById commitWithin question

Yonik Seeley-2-2
In reply to this post by Jamie Johnson
On Thu, Jul 5, 2012 at 4:29 PM, Jamie Johnson <[hidden email]> wrote:
> I am running off of a snapshot taken 5/3/2012 of solr 4.0 and am
> noticing some issues around deleteById when a commitWithin parameter
> is included using SolrJ

Oh wait... I just realized you were talking about SolrJ specifically -
so the issue may be there.

-Yonik
http://lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

Re: deleteById commitWithin question

Jamie Johnson
Thanks for the fast response.  Yeah, it could be in either place
either SolrJ or on the Server side.

On Thu, Jul 5, 2012 at 4:47 PM, Yonik Seeley <[hidden email]> wrote:

> On Thu, Jul 5, 2012 at 4:29 PM, Jamie Johnson <[hidden email]> wrote:
>> I am running off of a snapshot taken 5/3/2012 of solr 4.0 and am
>> noticing some issues around deleteById when a commitWithin parameter
>> is included using SolrJ
>
> Oh wait... I just realized you were talking about SolrJ specifically -
> so the issue may be there.
>
> -Yonik
> http://lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

Re: deleteById commitWithin question

Jamie Johnson
Ok, so some more context, hopefully this is useful.

I didn't think this was a SolrCloud issue but it appears to be.  I
have a simple 2 shard set up, I add 1 document which goes to shard 1.
I then issue a delete to shard 2.  The delete gets there and I see the
commit eventually being run in the log, but it doesn't appear to be
being distributed to shard 1.  If I manually call a server.commit()
the item is removed.

On Thu, Jul 5, 2012 at 4:56 PM, Jamie Johnson <[hidden email]> wrote:

> Thanks for the fast response.  Yeah, it could be in either place
> either SolrJ or on the Server side.
>
> On Thu, Jul 5, 2012 at 4:47 PM, Yonik Seeley <[hidden email]> wrote:
>> On Thu, Jul 5, 2012 at 4:29 PM, Jamie Johnson <[hidden email]> wrote:
>>> I am running off of a snapshot taken 5/3/2012 of solr 4.0 and am
>>> noticing some issues around deleteById when a commitWithin parameter
>>> is included using SolrJ
>>
>> Oh wait... I just realized you were talking about SolrJ specifically -
>> so the issue may be there.
>>
>> -Yonik
>> http://lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

Re: deleteById commitWithin question

Yonik Seeley-2-2
On Thu, Jul 5, 2012 at 5:12 PM, Jamie Johnson <[hidden email]> wrote:
> Ok, so some more context, hopefully this is useful.
>
> I didn't think this was a SolrCloud issue but it appears to be.  I
> have a simple 2 shard set up, I add 1 document which goes to shard 1.
> I then issue a delete to shard 2.  The delete gets there and I see the
> commit eventually being run in the log, but it doesn't appear to be
> being distributed to shard 1.  If I manually call a server.commit()
> the item is removed.

OK, I just verified this for trunk / 4x.  It appears to work fine now,
but I'm not exactly sure why it may not with the snapshot you have
(much has changed  since).

I followed the solr cloud wiki except that the path now needs to have
collection1 in it, and I set the number of shards to 1 (with 2
replicas).   I indexed the example docs, then queried both shards to
verify the doc existed on both.  Then I deleted the doc through the
non-leader via:
curl "http://localhost:7574/solr/update?commitWithin=60000" -H
'Content-type:application/json' -d '{"delete": { "id":"SOLR1000"}}'

This was to test both the forwarding to the leader and the fowarding
from the leader to the replica.
After 60 seconds, the document was gone on both replicas.

aside: If you use deleteByQuery, be aware that full distributed
indexing support was just finished yesterday by SOLR-3559

-Yonik
http://lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

Re: deleteById commitWithin question

Jamie Johnson
Thanks Yonik, glad that it is not an issue on Trunk.  I'll see if our
group is interested in updating now to take advantage of this.  In the
mean time we just issue a commit after we do deletes and that seems to
be a sufficient workaround.

On Thu, Jul 5, 2012 at 6:05 PM, Yonik Seeley <[hidden email]> wrote:

> On Thu, Jul 5, 2012 at 5:12 PM, Jamie Johnson <[hidden email]> wrote:
>> Ok, so some more context, hopefully this is useful.
>>
>> I didn't think this was a SolrCloud issue but it appears to be.  I
>> have a simple 2 shard set up, I add 1 document which goes to shard 1.
>> I then issue a delete to shard 2.  The delete gets there and I see the
>> commit eventually being run in the log, but it doesn't appear to be
>> being distributed to shard 1.  If I manually call a server.commit()
>> the item is removed.
>
> OK, I just verified this for trunk / 4x.  It appears to work fine now,
> but I'm not exactly sure why it may not with the snapshot you have
> (much has changed  since).
>
> I followed the solr cloud wiki except that the path now needs to have
> collection1 in it, and I set the number of shards to 1 (with 2
> replicas).   I indexed the example docs, then queried both shards to
> verify the doc existed on both.  Then I deleted the doc through the
> non-leader via:
> curl "http://localhost:7574/solr/update?commitWithin=60000" -H
> 'Content-type:application/json' -d '{"delete": { "id":"SOLR1000"}}'
>
> This was to test both the forwarding to the leader and the fowarding
> from the leader to the replica.
> After 60 seconds, the document was gone on both replicas.
>
> aside: If you use deleteByQuery, be aware that full distributed
> indexing support was just finished yesterday by SOLR-3559
>
> -Yonik
> http://lucidimagination.com