query other solr collection from within a solr plugin

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

query other solr collection from within a solr plugin

Nicolas Franck
I'm writing a solr plugin in java that has to query another solr collection to gather
information. What is the best way to do this?

For now I'm just using a SolrClient ( CloudSolrClient ), but has several disadvantages:

* you have to extract from core metadata where your server resides, and setup your SolrClient accordingly.
* you are just knocking at the same door
* search has to go over http for the same core.

Is there a better way? Are there any examples?

Thanks in advance

Nicolas Franck
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: query other solr collection from within a solr plugin

Malcolm Upayavira Holmes
Go look in the source for the Join query parser. It does this.

Upayavira

On Thu, 26 Jul 2018, at 1:04 PM, Nicolas Franck wrote:

> I'm writing a solr plugin in java that has to query another solr
> collection to gather
> information. What is the best way to do this?
>
> For now I'm just using a SolrClient ( CloudSolrClient ), but has several
> disadvantages:
>
> * you have to extract from core metadata where your server resides, and
> setup your SolrClient accordingly.
> * you are just knocking at the same door
> * search has to go over http for the same core.
>
> Is there a better way? Are there any examples?
>
> Thanks in advance
>
> Nicolas Franck
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: query other solr collection from within a solr plugin

Mikhail Khludnev-2
In reply to this post by Nicolas Franck
[subquery] calls remote cloud collections if collection parameter (which is somewhat not well known, documented) is supplied 


On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <[hidden email]> wrote:
I'm writing a solr plugin in java that has to query another solr collection to gather
information. What is the best way to do this?

For now I'm just using a SolrClient ( CloudSolrClient ), but has several disadvantages:

* you have to extract from core metadata where your server resides, and setup your SolrClient accordingly.
* you are just knocking at the same door
* search has to go over http for the same core.

Is there a better way? Are there any examples?

Thanks in advance

Nicolas Franck
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: query other solr collection from within a solr plugin

Nicolas Franck
In reply to this post by Malcolm Upayavira Holmes
thanks, I'll have a look!

> On 26 Jul 2018, at 14:15, Upayavira <[hidden email]> wrote:
>
> Go look in the source for the Join query parser. It does this.
>
> Upayavira
>
> On Thu, 26 Jul 2018, at 1:04 PM, Nicolas Franck wrote:
>> I'm writing a solr plugin in java that has to query another solr
>> collection to gather
>> information. What is the best way to do this?
>>
>> For now I'm just using a SolrClient ( CloudSolrClient ), but has several
>> disadvantages:
>>
>> * you have to extract from core metadata where your server resides, and
>> setup your SolrClient accordingly.
>> * you are just knocking at the same door
>> * search has to go over http for the same core.
>>
>> Is there a better way? Are there any examples?
>>
>> Thanks in advance
>>
>> Nicolas Franck
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: query other solr collection from within a solr plugin

Nicolas Franck
In reply to this post by Mikhail Khludnev-2
thanks, I'll have a look!

On 26 Jul 2018, at 14:28, Mikhail Khludnev <[hidden email]> wrote:

[subquery] calls remote cloud collections if collection parameter (which is somewhat not well known, documented) is supplied 


On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <[hidden email]> wrote:
I'm writing a solr plugin in java that has to query another solr collection to gather
information. What is the best way to do this?

For now I'm just using a SolrClient ( CloudSolrClient ), but has several disadvantages:

* you have to extract from core metadata where your server resides, and setup your SolrClient accordingly.
* you are just knocking at the same door
* search has to go over http for the same core.

Is there a better way? Are there any examples?

Thanks in advance

Nicolas Franck
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
Sincerely yours
Mikhail Khludnev

Reply | Threaded
Open this post in threaded view
|

Re: query other solr collection from within a solr plugin

Nicolas Franck
In reply to this post by Mikhail Khludnev-2
From what I've seen now, it seems that you can only directly connect to a specific core on your own node,
right? Should have expected that: it is local ;-)

Then I'll stick to the old solution that worked after all.

Thanks for all the advice

On 26 Jul 2018, at 14:28, Mikhail Khludnev <[hidden email]> wrote:

[subquery] calls remote cloud collections if collection parameter (which is somewhat not well known, documented) is supplied 


On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <[hidden email]> wrote:
I'm writing a solr plugin in java that has to query another solr collection to gather
information. What is the best way to do this?

For now I'm just using a SolrClient ( CloudSolrClient ), but has several disadvantages:

* you have to extract from core metadata where your server resides, and setup your SolrClient accordingly.
* you are just knocking at the same door
* search has to go over http for the same core.

Is there a better way? Are there any examples?

Thanks in advance

Nicolas Franck
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
Sincerely yours
Mikhail Khludnev

Reply | Threaded
Open this post in threaded view
|

Re: query other solr collection from within a solr plugin

Mikhail Khludnev-2

On Fri, Jul 27, 2018 at 11:48 AM Nicolas Franck <[hidden email]> wrote:
From what I've seen now, it seems that you can only directly connect to a specific core on your own node,
right? Should have expected that: it is local ;-)

Then I'll stick to the old solution that worked after all.

Thanks for all the advice

On 26 Jul 2018, at 14:28, Mikhail Khludnev <[hidden email]> wrote:

[subquery] calls remote cloud collections if collection parameter (which is somewhat not well known, documented) is supplied 


On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <[hidden email]> wrote:
I'm writing a solr plugin in java that has to query another solr collection to gather
information. What is the best way to do this?

For now I'm just using a SolrClient ( CloudSolrClient ), but has several disadvantages:

* you have to extract from core metadata where your server resides, and setup your SolrClient accordingly.
* you are just knocking at the same door
* search has to go over http for the same core.

Is there a better way? Are there any examples?

Thanks in advance

Nicolas Franck
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
Sincerely yours
Mikhail Khludnev



--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: query other solr collection from within a solr plugin

Nicolas Franck
@Mikhail Khludnev: thanks for your response

You mean something like this (source collection is "collection1", and I want to query "collection2"):

  SolrClient solrClient = new EmbeddedSolrServer(req.getCore());
  ModifiableSolrParams newParams = new ModifiableSolrParams();
  newParams.add("collection","collection2");
  SolrDocumentList docs = solrClient.getById(ids, newParams);

which is basically the same as:


but in both cases, I get this error:

   org.apache.solr.common.SolrException: Can't find shard 'collection1_shard1' at org.apache.solr.handler.component.RealTimeGetComponent.sliceToShards(RealTimeGetComponent.java:897) at 


apparently it forgets its own cloud information? What am I missing here?

BTW: is it necessary to  recreate a client like this on every request, giving the nature of a solrcloud where the shards can change?

On 27 Jul 2018, at 16:04, Mikhail Khludnev <[hidden email]> wrote:

Sure, it's up to you. But if for matter of fact, it EmbeddedSolrServer request has "collection" param, it pulls shards from zk and executed distributed request. 

On Fri, Jul 27, 2018 at 11:48 AM Nicolas Franck <[hidden email]> wrote:
From what I've seen now, it seems that you can only directly connect to a specific core on your own node,
right? Should have expected that: it is local ;-)

Then I'll stick to the old solution that worked after all.

Thanks for all the advice

On 26 Jul 2018, at 14:28, Mikhail Khludnev <[hidden email]> wrote:

[subquery] calls remote cloud collections if collection parameter (which is somewhat not well known, documented) is supplied 


On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <[hidden email]> wrote:
I'm writing a solr plugin in java that has to query another solr collection to gather
information. What is the best way to do this?

For now I'm just using a SolrClient ( CloudSolrClient ), but has several disadvantages:

* you have to extract from core metadata where your server resides, and setup your SolrClient accordingly.
* you are just knocking at the same door
* search has to go over http for the same core.

Is there a better way? Are there any examples?

Thanks in advance

Nicolas Franck
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
Sincerely yours
Mikhail Khludnev



--
Sincerely yours
Mikhail Khludnev

Reply | Threaded
Open this post in threaded view
|

Re: query other solr collection from within a solr plugin

Mikhail Khludnev-2
Well. I certainly haven't invoked RTG by this way. You might need to debug and understand what's going on. I briefly looked through code, maybe supplying shards param is an option  (see RealTimeGetComponent.createSubRequests(ResponseBuilder)), but it implies "gathering cloud info".

On Wed, Aug 1, 2018 at 9:59 AM Nicolas Franck <[hidden email]> wrote:
@Mikhail Khludnev: thanks for your response

You mean something like this (source collection is "collection1", and I want to query "collection2"):

  SolrClient solrClient = new EmbeddedSolrServer(req.getCore());
  ModifiableSolrParams newParams = new ModifiableSolrParams();
  newParams.add("collection","collection2");
  SolrDocumentList docs = solrClient.getById(ids, newParams);

which is basically the same as:


but in both cases, I get this error:

   org.apache.solr.common.SolrException: Can't find shard 'collection1_shard1' at org.apache.solr.handler.component.RealTimeGetComponent.sliceToShards(RealTimeGetComponent.java:897) at 


apparently it forgets its own cloud information? What am I missing here?

BTW: is it necessary to  recreate a client like this on every request, giving the nature of a solrcloud where the shards can change?

On 27 Jul 2018, at 16:04, Mikhail Khludnev <[hidden email]> wrote:

Sure, it's up to you. But if for matter of fact, it EmbeddedSolrServer request has "collection" param, it pulls shards from zk and executed distributed request. 

On Fri, Jul 27, 2018 at 11:48 AM Nicolas Franck <[hidden email]> wrote:
From what I've seen now, it seems that you can only directly connect to a specific core on your own node,
right? Should have expected that: it is local ;-)

Then I'll stick to the old solution that worked after all.

Thanks for all the advice

On 26 Jul 2018, at 14:28, Mikhail Khludnev <[hidden email]> wrote:

[subquery] calls remote cloud collections if collection parameter (which is somewhat not well known, documented) is supplied 


On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <[hidden email]> wrote:
I'm writing a solr plugin in java that has to query another solr collection to gather
information. What is the best way to do this?

For now I'm just using a SolrClient ( CloudSolrClient ), but has several disadvantages:

* you have to extract from core metadata where your server resides, and setup your SolrClient accordingly.
* you are just knocking at the same door
* search has to go over http for the same core.

Is there a better way? Are there any examples?

Thanks in advance

Nicolas Franck
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
Sincerely yours
Mikhail Khludnev



--
Sincerely yours
Mikhail Khludnev



--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: query other solr collection from within a solr plugin

Shawn Heisey-2
In reply to this post by Nicolas Franck
On 8/1/2018 12:59 AM, Nicolas Franck wrote:

> @Mikhail Khludnev: thanks for your response
>
> You mean something like this (source collection is "collection1", and
> I want to query "collection2"):
>
>   SolrClient solrClient = new EmbeddedSolrServer(req.getCore());
>   ModifiableSolrParams newParams = new ModifiableSolrParams();
>   newParams.add("collection","collection2");
>   SolrDocumentList docs = solrClient.getById(ids, newParams);
>
> which is basically the same as:
>
> http://localhost:8983/collection1_shard1_replica_n1/select?collection=collection2&&qt=/get&ids=myid
>

An instance of EmbeddedSolrServer has no http access. Also, it can't do
SolrCloud -- because SolrCloud requires http. EmbeddedSolrServer is a
complete Solr server running in standalone (not cloud) mode, without
http access.

If you're doing this code within a Solr plugin, then you can't start an
EmbeddedSolrServer on one of the cores from the Solr install.  Any cores
you try to use will already be open, so the embedded server will not be
able to open them.  Since you're already running inside a Solr server,
there's no reason to start *another* Solr server.

I don't have any other ideas for you.  I've written a couple of update
processors for Solr, but nothing that does queries.

> but in both cases, I get this error:
>
> org.apache.solr.common.SolrException: Can't find shard
> 'collection1_shard1' at
> org.apache.solr.handler.component.RealTimeGetComponent.sliceToShards(RealTimeGetComponent.java:897)
> at
>
>
> apparently it forgets its own cloud information? What am I missing here?

As already mentioned, EmbeddedSolrServer can't do SolrCloud.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: query other solr collection from within a solr plugin

Nicolas Franck
Alright, as I expected.

Thanks for all your help!

> On 1 Aug 2018, at 16:22, Shawn Heisey <[hidden email]> wrote:
>
> On 8/1/2018 12:59 AM, Nicolas Franck wrote:
>> @Mikhail Khludnev: thanks for your response
>>
>> You mean something like this (source collection is "collection1", and I want to query "collection2"):
>>
>>   SolrClient solrClient = new EmbeddedSolrServer(req.getCore());
>>   ModifiableSolrParams newParams = new ModifiableSolrParams();
>>   newParams.add("collection","collection2");
>>   SolrDocumentList docs = solrClient.getById(ids, newParams);
>>
>> which is basically the same as:
>>
>> http://localhost:8983/collection1_shard1_replica_n1/select?collection=collection2&&qt=/get&ids=myid
>>
>
> An instance of EmbeddedSolrServer has no http access. Also, it can't do SolrCloud -- because SolrCloud requires http. EmbeddedSolrServer is a complete Solr server running in standalone (not cloud) mode, without http access.
>
> If you're doing this code within a Solr plugin, then you can't start an EmbeddedSolrServer on one of the cores from the Solr install.  Any cores you try to use will already be open, so the embedded server will not be able to open them.  Since you're already running inside a Solr server, there's no reason to start *another* Solr server.
>
> I don't have any other ideas for you.  I've written a couple of update processors for Solr, but nothing that does queries.
>
>> but in both cases, I get this error:
>>
>> org.apache.solr.common.SolrException: Can't find shard 'collection1_shard1' at org.apache.solr.handler.component.RealTimeGetComponent.sliceToShards(RealTimeGetComponent.java:897) at
>>
>>
>> apparently it forgets its own cloud information? What am I missing here?
>
> As already mentioned, EmbeddedSolrServer can't do SolrCloud.
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]