Problem with CoreAdmin API CREATE command

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with CoreAdmin API CREATE command

Yago Riveiro
Hi,

I’m having some issues with the command CREATE of the CoreAdmin API in solr 4.10.4.

When I try to load a previous unloaded core with CREATE command, the result of this operation is 2 replicas in down state. One with the original coreNodeName set in clusterstate.json and other with an new one.

I’m with the new solr.xml format and the core.properties of cores looks like (I set this configuration when  upgraded from 4.6.1 in legacy solr.xml to 4.10.4:

name=bucket-15_shardX_replicaY
shard=shardX
collection=bucket-15

The API command looks like:

solr/admin/cores?action=CREATE&name=bucket-15_shardX_replicaY&collection=bucket-15&shard=shardX&wt=json

What I’m doing wrong?
Best regards /Yago
Reply | Threaded
Open this post in threaded view
|

Re: Problem with CoreAdmin API CREATE command

Erick Erickson
The not-very-helpful answer is that you're using the core admin API in
a SolrCloud setup. Please do not do this as (you're well aware of this by now!)
it's far too easy to get "interesting" results.

Instead, use the Collections API, specifically the ADDREPLICA and DELETEREPLICA
commands. Under the covers, they actually create a core via the core admin API,
but they insure that all the core create parameters are correct. For ADDREPLICA,
you can also easily insure that a node lands on a particular machine.

Best,
Erick

On Tue, Sep 15, 2015 at 9:46 AM, Yago Riveiro <[hidden email]> wrote:

> Hi,
>
> I’m having some issues with the command CREATE of the CoreAdmin API in solr
> 4.10.4.
>
> When I try to load a previous unloaded core with CREATE command, the result
> of this operation is 2 replicas in down state. One with the original
> coreNodeName set in clusterstate.json and other with an new one.
>
> I’m with the new solr.xml format and the core.properties of cores looks like
> (I set this configuration when  upgraded from 4.6.1 in legacy solr.xml to
> 4.10.4:
>
> name=bucket-15_shardX_replicaY
> shard=shardX
> collection=bucket-15
>
> The API command looks like:
>
> solr/admin/cores?action=CREATE&name=bucket-15_shardX_replicaY&collection=bucket-15&shard=shardX&wt=json
>
> What I’m doing wrong?
>
>
>
> -----
> Best regards
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Problem-with-CoreAdmin-API-CREATE-command-tp4229248.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Problem with CoreAdmin API CREATE command

Yago Riveiro
I have a very old index with more than 12T (re-index data is not an option ...) that I want upgrade to 5.3, I’m using lucene-core-4.10.4.jar (I’m in 4.10.4 right now) to upgrade old segments of data. With solr running I can run the command because solr has the lock of the core.





I only want to unload it to perform an index upgrade command to load again without stop the node with the core.





The issue here is that the DELETEREPLICA deletes the data … the UNLOAD command doesn’t save the original core.properties and creates a new one core.properties.unloaded that not store the shard param and creates a new random coreNodeName that doesn’t correspond to the name of the clusterstate, the result … the core is loaded twice or even worse, in some situation is attached to a wrong shard.








—/Yago Riveiro

On Wed, Sep 16, 2015 at 4:38 PM, Erick Erickson <[hidden email]>
wrote:

> The not-very-helpful answer is that you're using the core admin API in
> a SolrCloud setup. Please do not do this as (you're well aware of this by now!)
> it's far too easy to get "interesting" results.
> Instead, use the Collections API, specifically the ADDREPLICA and DELETEREPLICA
> commands. Under the covers, they actually create a core via the core admin API,
> but they insure that all the core create parameters are correct. For ADDREPLICA,
> you can also easily insure that a node lands on a particular machine.
> Best,
> Erick
> On Tue, Sep 15, 2015 at 9:46 AM, Yago Riveiro <[hidden email]> wrote:
>> Hi,
>>
>> I’m having some issues with the command CREATE of the CoreAdmin API in solr
>> 4.10.4.
>>
>> When I try to load a previous unloaded core with CREATE command, the result
>> of this operation is 2 replicas in down state. One with the original
>> coreNodeName set in clusterstate.json and other with an new one.
>>
>> I’m with the new solr.xml format and the core.properties of cores looks like
>> (I set this configuration when  upgraded from 4.6.1 in legacy solr.xml to
>> 4.10.4:
>>
>> name=bucket-15_shardX_replicaY
>> shard=shardX
>> collection=bucket-15
>>
>> The API command looks like:
>>
>> solr/admin/cores?action=CREATE&name=bucket-15_shardX_replicaY&collection=bucket-15&shard=shardX&wt=json
>>
>> What I’m doing wrong?
>>
>>
>>
>> -----
>> Best regards
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/Problem-with-CoreAdmin-API-CREATE-command-tp4229248.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
Best regards /Yago
Reply | Threaded
Open this post in threaded view
|

Re: Problem with CoreAdmin API CREATE command

Shai Erera
Solr 5.3 can read Solr 4.10.4 indexes as-is. Why are you trying to upgrade
the indexes in the first place?

Shai

On Thu, Sep 17, 2015 at 3:05 PM, Yago Riveiro <[hidden email]>
wrote:

> I have a very old index with more than 12T (re-index data is not an option
> ...) that I want upgrade to 5.3, I’m using lucene-core-4.10.4.jar (I’m in
> 4.10.4 right now) to upgrade old segments of data. With solr running I can
> run the command because solr has the lock of the core.
>
>
>
>
>
> I only want to unload it to perform an index upgrade command to load again
> without stop the node with the core.
>
>
>
>
>
> The issue here is that the DELETEREPLICA deletes the data … the UNLOAD
> command doesn’t save the original core.properties and creates a new one
> core.properties.unloaded that not store the shard param and creates a new
> random coreNodeName that doesn’t correspond to the name of
> the clusterstate, the result … the core is loaded twice or even worse, in
> some situation is attached to a wrong shard.
>
>
>
>
>
>
>
>
> —/Yago Riveiro
>
> On Wed, Sep 16, 2015 at 4:38 PM, Erick Erickson <[hidden email]>
> wrote:
>
> > The not-very-helpful answer is that you're using the core admin API in
> > a SolrCloud setup. Please do not do this as (you're well aware of this
> by now!)
> > it's far too easy to get "interesting" results.
> > Instead, use the Collections API, specifically the ADDREPLICA and
> DELETEREPLICA
> > commands. Under the covers, they actually create a core via the core
> admin API,
> > but they insure that all the core create parameters are correct. For
> ADDREPLICA,
> > you can also easily insure that a node lands on a particular machine.
> > Best,
> > Erick
> > On Tue, Sep 15, 2015 at 9:46 AM, Yago Riveiro <[hidden email]>
> wrote:
> >> Hi,
> >>
> >> I’m having some issues with the command CREATE of the CoreAdmin API in
> solr
> >> 4.10.4.
> >>
> >> When I try to load a previous unloaded core with CREATE command, the
> result
> >> of this operation is 2 replicas in down state. One with the original
> >> coreNodeName set in clusterstate.json and other with an new one.
> >>
> >> I’m with the new solr.xml format and the core.properties of cores looks
> like
> >> (I set this configuration when  upgraded from 4.6.1 in legacy solr.xml
> to
> >> 4.10.4:
> >>
> >> name=bucket-15_shardX_replicaY
> >> shard=shardX
> >> collection=bucket-15
> >>
> >> The API command looks like:
> >>
> >>
> solr/admin/cores?action=CREATE&name=bucket-15_shardX_replicaY&collection=bucket-15&shard=shardX&wt=json
> >>
> >> What I’m doing wrong?
> >>
> >>
> >>
> >> -----
> >> Best regards
> >> --
> >> View this message in context:
> http://lucene.472066.n3.nabble.com/Problem-with-CoreAdmin-API-CREATE-command-tp4229248.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Problem with CoreAdmin API CREATE command

Yago Riveiro
90% of my data was indexed in 4.6.1 or lower.




My goal is upgrade all data to 4.10.4 and then upgrade to 5.3




In previous test that I did with 5.3 in dev cluster, I see some strange behaviour with data indexed with 4.6.1 that in 4.10.4 didn’t reproduce.




Some queries to data indexed in 4.6.1 that returns data with source 4.6.1, in 5.3 some times returned zero docs (in a random fashion way, I can’t find the pattern that produced the failure of the query). The same data upgraded to 4.10.4 with source code of 5.3 worked as expected with any issue.


—/Yago Riveiro

On Thu, Sep 17, 2015 at 1:08 PM, Shai Erera <[hidden email]> wrote:

> Solr 5.3 can read Solr 4.10.4 indexes as-is. Why are you trying to upgrade
> the indexes in the first place?
> Shai
> On Thu, Sep 17, 2015 at 3:05 PM, Yago Riveiro <[hidden email]>
> wrote:
>> I have a very old index with more than 12T (re-index data is not an option
>> ...) that I want upgrade to 5.3, I’m using lucene-core-4.10.4.jar (I’m in
>> 4.10.4 right now) to upgrade old segments of data. With solr running I can
>> run the command because solr has the lock of the core.
>>
>>
>>
>>
>>
>> I only want to unload it to perform an index upgrade command to load again
>> without stop the node with the core.
>>
>>
>>
>>
>>
>> The issue here is that the DELETEREPLICA deletes the data … the UNLOAD
>> command doesn’t save the original core.properties and creates a new one
>> core.properties.unloaded that not store the shard param and creates a new
>> random coreNodeName that doesn’t correspond to the name of
>> the clusterstate, the result … the core is loaded twice or even worse, in
>> some situation is attached to a wrong shard.
>>
>>
>>
>>
>>
>>
>>
>>
>> —/Yago Riveiro
>>
>> On Wed, Sep 16, 2015 at 4:38 PM, Erick Erickson <[hidden email]>
>> wrote:
>>
>> > The not-very-helpful answer is that you're using the core admin API in
>> > a SolrCloud setup. Please do not do this as (you're well aware of this
>> by now!)
>> > it's far too easy to get "interesting" results.
>> > Instead, use the Collections API, specifically the ADDREPLICA and
>> DELETEREPLICA
>> > commands. Under the covers, they actually create a core via the core
>> admin API,
>> > but they insure that all the core create parameters are correct. For
>> ADDREPLICA,
>> > you can also easily insure that a node lands on a particular machine.
>> > Best,
>> > Erick
>> > On Tue, Sep 15, 2015 at 9:46 AM, Yago Riveiro <[hidden email]>
>> wrote:
>> >> Hi,
>> >>
>> >> I’m having some issues with the command CREATE of the CoreAdmin API in
>> solr
>> >> 4.10.4.
>> >>
>> >> When I try to load a previous unloaded core with CREATE command, the
>> result
>> >> of this operation is 2 replicas in down state. One with the original
>> >> coreNodeName set in clusterstate.json and other with an new one.
>> >>
>> >> I’m with the new solr.xml format and the core.properties of cores looks
>> like
>> >> (I set this configuration when  upgraded from 4.6.1 in legacy solr.xml
>> to
>> >> 4.10.4:
>> >>
>> >> name=bucket-15_shardX_replicaY
>> >> shard=shardX
>> >> collection=bucket-15
>> >>
>> >> The API command looks like:
>> >>
>> >>
>> solr/admin/cores?action=CREATE&name=bucket-15_shardX_replicaY&collection=bucket-15&shard=shardX&wt=json
>> >>
>> >> What I’m doing wrong?
>> >>
>> >>
>> >>
>> >> -----
>> >> Best regards
>> >> --
>> >> View this message in context:
>> http://lucene.472066.n3.nabble.com/Problem-with-CoreAdmin-API-CREATE-command-tp4229248.html
>> >> Sent from the Solr - User mailing list archive at Nabble.com.
Best regards /Yago
Reply | Threaded
Open this post in threaded view
|

Re: Problem with CoreAdmin API CREATE command

Shai Erera
That's definitely strange as Solr 5.x should support all Solr 4.x indexes.

Anyway, you can somewhat force an upgrade by running a forceMerge command
after you've upgraded the libraries to 5.3.0. This will rewrite the index
into one segment whose version will be 5.3. It is usually not recommended
to run this as it's a heavy operation, but if you want all index segments
to be upgraded, this is an option.

But again, I don't see any reason to do that. If you hit an error that you
can reproduce, it might be worth it to file a bug report.

Shai

On Thu, Sep 17, 2015 at 4:01 PM, Yago Riveiro <[hidden email]>
wrote:

> 90% of my data was indexed in 4.6.1 or lower.
>
>
>
>
> My goal is upgrade all data to 4.10.4 and then upgrade to 5.3
>
>
>
>
> In previous test that I did with 5.3 in dev cluster, I see some strange
> behaviour with data indexed with 4.6.1 that in 4.10.4 didn’t reproduce.
>
>
>
>
> Some queries to data indexed in 4.6.1 that returns data with source 4.6.1,
> in 5.3 some times returned zero docs (in a random fashion way, I can’t find
> the pattern that produced the failure of the query). The same data upgraded
> to 4.10.4 with source code of 5.3 worked as expected with any issue.
>
>
> —/Yago Riveiro
>
> On Thu, Sep 17, 2015 at 1:08 PM, Shai Erera <[hidden email]> wrote:
>
> > Solr 5.3 can read Solr 4.10.4 indexes as-is. Why are you trying to
> upgrade
> > the indexes in the first place?
> > Shai
> > On Thu, Sep 17, 2015 at 3:05 PM, Yago Riveiro <[hidden email]>
> > wrote:
> >> I have a very old index with more than 12T (re-index data is not an
> option
> >> ...) that I want upgrade to 5.3, I’m using lucene-core-4.10.4.jar (I’m
> in
> >> 4.10.4 right now) to upgrade old segments of data. With solr running I
> can
> >> run the command because solr has the lock of the core.
> >>
> >>
> >>
> >>
> >>
> >> I only want to unload it to perform an index upgrade command to load
> again
> >> without stop the node with the core.
> >>
> >>
> >>
> >>
> >>
> >> The issue here is that the DELETEREPLICA deletes the data … the UNLOAD
> >> command doesn’t save the original core.properties and creates a new one
> >> core.properties.unloaded that not store the shard param and creates a
> new
> >> random coreNodeName that doesn’t correspond to the name of
> >> the clusterstate, the result … the core is loaded twice or even worse,
> in
> >> some situation is attached to a wrong shard.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> —/Yago Riveiro
> >>
> >> On Wed, Sep 16, 2015 at 4:38 PM, Erick Erickson <
> [hidden email]>
> >> wrote:
> >>
> >> > The not-very-helpful answer is that you're using the core admin API in
> >> > a SolrCloud setup. Please do not do this as (you're well aware of this
> >> by now!)
> >> > it's far too easy to get "interesting" results.
> >> > Instead, use the Collections API, specifically the ADDREPLICA and
> >> DELETEREPLICA
> >> > commands. Under the covers, they actually create a core via the core
> >> admin API,
> >> > but they insure that all the core create parameters are correct. For
> >> ADDREPLICA,
> >> > you can also easily insure that a node lands on a particular machine.
> >> > Best,
> >> > Erick
> >> > On Tue, Sep 15, 2015 at 9:46 AM, Yago Riveiro <[hidden email]
> >
> >> wrote:
> >> >> Hi,
> >> >>
> >> >> I’m having some issues with the command CREATE of the CoreAdmin API
> in
> >> solr
> >> >> 4.10.4.
> >> >>
> >> >> When I try to load a previous unloaded core with CREATE command, the
> >> result
> >> >> of this operation is 2 replicas in down state. One with the original
> >> >> coreNodeName set in clusterstate.json and other with an new one.
> >> >>
> >> >> I’m with the new solr.xml format and the core.properties of cores
> looks
> >> like
> >> >> (I set this configuration when  upgraded from 4.6.1 in legacy
> solr.xml
> >> to
> >> >> 4.10.4:
> >> >>
> >> >> name=bucket-15_shardX_replicaY
> >> >> shard=shardX
> >> >> collection=bucket-15
> >> >>
> >> >> The API command looks like:
> >> >>
> >> >>
> >>
> solr/admin/cores?action=CREATE&name=bucket-15_shardX_replicaY&collection=bucket-15&shard=shardX&wt=json
> >> >>
> >> >> What I’m doing wrong?
> >> >>
> >> >>
> >> >>
> >> >> -----
> >> >> Best regards
> >> >> --
> >> >> View this message in context:
> >>
> http://lucene.472066.n3.nabble.com/Problem-with-CoreAdmin-API-CREATE-command-tp4229248.html
> >> >> Sent from the Solr - User mailing list archive at Nabble.com.