SolrClient from inside processAdd function

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

SolrClient from inside processAdd function

gnandre
Hi,

Is there a way to create SolrClient from inside processAdd function for
custom update processor for the same Solr on which it is executing?
Reply | Threaded
Open this post in threaded view
|

Re: SolrClient from inside processAdd function

Erick Erickson
This really sounds like an XY problem. What do you need the SolrClient _for_? I suspect there’s an easier way to do this…..

Best,
Erick

> On Sep 3, 2019, at 6:17 PM, Arnold Bronley <[hidden email]> wrote:
>
> Hi,
>
> Is there a way to create SolrClient from inside processAdd function for
> custom update processor for the same Solr on which it is executing?

Reply | Threaded
Open this post in threaded view
|

Re: SolrClient from inside processAdd function

gnandre
I need to search some other collection inside processAdd function and
append that information to the indexing request.

On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson <[hidden email]>
wrote:

> This really sounds like an XY problem. What do you need the SolrClient
> _for_? I suspect there’s an easier way to do this…..
>
> Best,
> Erick
>
> > On Sep 3, 2019, at 6:17 PM, Arnold Bronley <[hidden email]>
> wrote:
> >
> > Hi,
> >
> > Is there a way to create SolrClient from inside processAdd function for
> > custom update processor for the same Solr on which it is executing?
>
>
Reply | Threaded
Open this post in threaded view
|

Re: SolrClient from inside processAdd function

Simon Rosenthal-2
Similarly, I had considered a URP which would call the Solr Tagger to add
new metadata fields  for indexing to incoming documents (and recall
discussing this with David Smiley), but eventually decided against this
approach on the grounds of complexity.

-Simon

On Wed, Sep 4, 2019 at 2:10 PM Arnold Bronley <[hidden email]>
wrote:

> I need to search some other collection inside processAdd function and
> append that information to the indexing request.
>
> On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson <[hidden email]>
> wrote:
>
> > This really sounds like an XY problem. What do you need the SolrClient
> > _for_? I suspect there’s an easier way to do this…..
> >
> > Best,
> > Erick
> >
> > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley <[hidden email]>
> > wrote:
> > >
> > > Hi,
> > >
> > > Is there a way to create SolrClient from inside processAdd function for
> > > custom update processor for the same Solr on which it is executing?
> >
> >
>


--
I am transferring my email  from Yahoo to [hidden email]. I will
continue to receive Yahoo email but will reply from this account. Please
update your address lists accordingly.
Reply | Threaded
Open this post in threaded view
|

Re: SolrClient from inside processAdd function

gnandre
Hi Simon,

I am interested in knowing what did you end up doing in your use-case then.
Can you please share it at least at high level?

On Wed, Sep 4, 2019 at 2:26 PM Simon Rosenthal <[hidden email]>
wrote:

> Similarly, I had considered a URP which would call the Solr Tagger to add
> new metadata fields  for indexing to incoming documents (and recall
> discussing this with David Smiley), but eventually decided against this
> approach on the grounds of complexity.
>
> -Simon
>
> On Wed, Sep 4, 2019 at 2:10 PM Arnold Bronley <[hidden email]>
> wrote:
>
> > I need to search some other collection inside processAdd function and
> > append that information to the indexing request.
> >
> > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson <[hidden email]>
> > wrote:
> >
> > > This really sounds like an XY problem. What do you need the SolrClient
> > > _for_? I suspect there’s an easier way to do this…..
> > >
> > > Best,
> > > Erick
> > >
> > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley <[hidden email]>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > Is there a way to create SolrClient from inside processAdd function
> for
> > > > custom update processor for the same Solr on which it is executing?
> > >
> > >
> >
>
>
> --
> I am transferring my email  from Yahoo to [hidden email]. I
> will
> continue to receive Yahoo email but will reply from this account. Please
> update your address lists accordingly.
>
Reply | Threaded
Open this post in threaded view
|

RE: SolrClient from inside processAdd function

Markus Jelsma-2
In reply to this post by gnandre
Hello Arnold,

Yes, we do this too for several cases.

You can create the SolrClient in the Factory's inform() method, and pass is to the URP when it is created. You must implement SolrCoreAware and close the client when the core closes as well. Use a CloseHook for this.

If you do not close the client, it will cause trouble if you run unit tests, and most certainly when you regularly reload cores.

Regards,
Markus

 
 
-----Original message-----

> From:Arnold Bronley <[hidden email]>
> Sent: Wednesday 4th September 2019 20:10
> To: [hidden email]
> Subject: Re: SolrClient from inside processAdd function
>
> I need to search some other collection inside processAdd function and
> append that information to the indexing request.
>
> On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson <[hidden email]>
> wrote:
>
> > This really sounds like an XY problem. What do you need the SolrClient
> > _for_? I suspect there’s an easier way to do this…..
> >
> > Best,
> > Erick
> >
> > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley <[hidden email]>
> > wrote:
> > >
> > > Hi,
> > >
> > > Is there a way to create SolrClient from inside processAdd function for
> > > custom update processor for the same Solr on which it is executing?
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: SolrClient from inside processAdd function

gnandre
Hi Markus,

Is there any way to get the information about the current Solr endpoint
from within the custom URP?

On Wed, Sep 4, 2019 at 3:10 PM Markus Jelsma <[hidden email]>
wrote:

> Hello Arnold,
>
> Yes, we do this too for several cases.
>
> You can create the SolrClient in the Factory's inform() method, and pass
> is to the URP when it is created. You must implement SolrCoreAware and
> close the client when the core closes as well. Use a CloseHook for this.
>
> If you do not close the client, it will cause trouble if you run unit
> tests, and most certainly when you regularly reload cores.
>
> Regards,
> Markus
>
>
>
> -----Original message-----
> > From:Arnold Bronley <[hidden email]>
> > Sent: Wednesday 4th September 2019 20:10
> > To: [hidden email]
> > Subject: Re: SolrClient from inside processAdd function
> >
> > I need to search some other collection inside processAdd function and
> > append that information to the indexing request.
> >
> > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson <[hidden email]>
> > wrote:
> >
> > > This really sounds like an XY problem. What do you need the SolrClient
> > > _for_? I suspect there’s an easier way to do this…..
> > >
> > > Best,
> > > Erick
> > >
> > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley <[hidden email]>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > Is there a way to create SolrClient from inside processAdd function
> for
> > > > custom update processor for the same Solr on which it is executing?
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

RE: SolrClient from inside processAdd function

Markus Jelsma-2
In reply to this post by gnandre
Hello Arnold,

In the Factory's inform() method you receive a SolrCore reference. Using this you can get the CloudDescriptor and the ZkController references. These provide access to what you need to open a connection for SolrClient.

Our plugins usually work in cloud and non-cloud environments, so we initialize different things for each situation. Like this abstracted in some CloudUtils thing:

    cloudDescriptor = core.getCoreDescriptor().getCloudDescriptor();
    zk = core.getCoreContainer().getZkController(); // this is the ZkController ref
    coreName = core.getCoreDescriptor().getName();
   
    // Are we in cloud mode?
    if (zk != null) {
      collectionName = core.getCoreDescriptor().getCollectionName();
      shardId = cloudDescriptor.getShardId();
    } else {
      collectionName = null;
      shardId = null;
    }

Depending on cloudMode we create new SolrClient instances based on these classes.

Check the apidocs and you'll quickly see what you need.

We use these api's to get what we need. But you can also find these things if you check the Java system properties, which is easier. We use the api's to read the data because if api's change, we get a compile error. If the system properties change, we don't. So the system properties is easier, but the api's are safer. Although a unit tests should guard against that as well.

Regards,
Markus

ps, on this list there is normally no need to create a new thread for an existing one, even if you are eagerly waiting for a reply. It might take some patience though.
 
-----Original message-----

> From:Arnold Bronley <[hidden email]>
> Sent: Thursday 5th September 2019 18:44
> To: [hidden email]
> Subject: Re: SolrClient from inside processAdd function
>
> Hi Markus,
>
> Is there any way to get the information about the current Solr endpoint
> from within the custom URP?
>
> On Wed, Sep 4, 2019 at 3:10 PM Markus Jelsma <[hidden email]>
> wrote:
>
> > Hello Arnold,
> >
> > Yes, we do this too for several cases.
> >
> > You can create the SolrClient in the Factory's inform() method, and pass
> > is to the URP when it is created. You must implement SolrCoreAware and
> > close the client when the core closes as well. Use a CloseHook for this.
> >
> > If you do not close the client, it will cause trouble if you run unit
> > tests, and most certainly when you regularly reload cores.
> >
> > Regards,
> > Markus
> >
> >
> >
> > -----Original message-----
> > > From:Arnold Bronley <[hidden email]>
> > > Sent: Wednesday 4th September 2019 20:10
> > > To: [hidden email]
> > > Subject: Re: SolrClient from inside processAdd function
> > >
> > > I need to search some other collection inside processAdd function and
> > > append that information to the indexing request.
> > >
> > > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson <[hidden email]>
> > > wrote:
> > >
> > > > This really sounds like an XY problem. What do you need the SolrClient
> > > > _for_? I suspect there’s an easier way to do this…..
> > > >
> > > > Best,
> > > > Erick
> > > >
> > > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley <[hidden email]>
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > Is there a way to create SolrClient from inside processAdd function
> > for
> > > > > custom update processor for the same Solr on which it is executing?
> > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: SolrClient from inside processAdd function

gnandre
Hi Markus,

"Depending on cloudMode we create new SolrClient instances based on these
classes.   "

But I still do not see SolrClient creation anywhere in your code snippet.
Am I missing something? I tried the solution with system properties and it
works but I would like to avoid that.

On Thu, Sep 5, 2019 at 6:20 PM Markus Jelsma <[hidden email]>
wrote:

> Hello Arnold,
>
> In the Factory's inform() method you receive a SolrCore reference. Using
> this you can get the CloudDescriptor and the ZkController references. These
> provide access to what you need to open a connection for SolrClient.
>
> Our plugins usually work in cloud and non-cloud environments, so we
> initialize different things for each situation. Like this abstracted in
> some CloudUtils thing:
>
>     cloudDescriptor = core.getCoreDescriptor().getCloudDescriptor();
>     zk = core.getCoreContainer().getZkController(); // this is the
> ZkController ref
>     coreName = core.getCoreDescriptor().getName();
>
>     // Are we in cloud mode?
>     if (zk != null) {
>       collectionName = core.getCoreDescriptor().getCollectionName();
>       shardId = cloudDescriptor.getShardId();
>     } else {
>       collectionName = null;
>       shardId = null;
>     }
>
> Depending on cloudMode we create new SolrClient instances based on these
> classes.
>
> Check the apidocs and you'll quickly see what you need.
>
> We use these api's to get what we need. But you can also find these things
> if you check the Java system properties, which is easier. We use the api's
> to read the data because if api's change, we get a compile error. If the
> system properties change, we don't. So the system properties is easier, but
> the api's are safer. Although a unit tests should guard against that as
> well.
>
> Regards,
> Markus
>
> ps, on this list there is normally no need to create a new thread for an
> existing one, even if you are eagerly waiting for a reply. It might take
> some patience though.
>
> -----Original message-----
> > From:Arnold Bronley <[hidden email]>
> > Sent: Thursday 5th September 2019 18:44
> > To: [hidden email]
> > Subject: Re: SolrClient from inside processAdd function
> >
> > Hi Markus,
> >
> > Is there any way to get the information about the current Solr endpoint
> > from within the custom URP?
> >
> > On Wed, Sep 4, 2019 at 3:10 PM Markus Jelsma <[hidden email]
> >
> > wrote:
> >
> > > Hello Arnold,
> > >
> > > Yes, we do this too for several cases.
> > >
> > > You can create the SolrClient in the Factory's inform() method, and
> pass
> > > is to the URP when it is created. You must implement SolrCoreAware and
> > > close the client when the core closes as well. Use a CloseHook for
> this.
> > >
> > > If you do not close the client, it will cause trouble if you run unit
> > > tests, and most certainly when you regularly reload cores.
> > >
> > > Regards,
> > > Markus
> > >
> > >
> > >
> > > -----Original message-----
> > > > From:Arnold Bronley <[hidden email]>
> > > > Sent: Wednesday 4th September 2019 20:10
> > > > To: [hidden email]
> > > > Subject: Re: SolrClient from inside processAdd function
> > > >
> > > > I need to search some other collection inside processAdd function and
> > > > append that information to the indexing request.
> > > >
> > > > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > This really sounds like an XY problem. What do you need the
> SolrClient
> > > > > _for_? I suspect there’s an easier way to do this…..
> > > > >
> > > > > Best,
> > > > > Erick
> > > > >
> > > > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley <
> [hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Is there a way to create SolrClient from inside processAdd
> function
> > > for
> > > > > > custom update processor for the same Solr on which it is
> executing?
> > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: SolrClient from inside processAdd function

gnandre
Hi,

Thanks for all this information. I am doing this now like following:

@Override
public void inform(SolrCore core) {
    HttpSolrClient.Builder builder = new HttpSolrClient.Builder();
    String baseURL =
core.getCoreContainer().getZkController().getBaseUrl() + "/" +
dataInfo.dataSource;
    builder.withBaseSolrUrl(baseURL);
    solrClient =  builder.build();

    core.addCloseHook(new CloseHook() {
        @Override
        public void preClose(SolrCore core) {
            //no cleanup needed before closing the core
        }

        @Override
        public void postClose(SolrCore core) {
            try {
                solrClient.close();
            } catch (IOException e) {
                logger.error("Not able to close solr client  after closing " +
                        "Solr core '{}'", core.getName(), e);
            }
        }
    });
}


On Fri, Sep 6, 2019 at 5:40 PM Arnold Bronley <[hidden email]>
wrote:

> Hi Markus,
>
> "Depending on cloudMode we create new SolrClient instances based on these
> classes.   "
>
> But I still do not see SolrClient creation anywhere in your code snippet.
> Am I missing something? I tried the solution with system properties and it
> works but I would like to avoid that.
>
> On Thu, Sep 5, 2019 at 6:20 PM Markus Jelsma <[hidden email]>
> wrote:
>
>> Hello Arnold,
>>
>> In the Factory's inform() method you receive a SolrCore reference. Using
>> this you can get the CloudDescriptor and the ZkController references. These
>> provide access to what you need to open a connection for SolrClient.
>>
>> Our plugins usually work in cloud and non-cloud environments, so we
>> initialize different things for each situation. Like this abstracted in
>> some CloudUtils thing:
>>
>>     cloudDescriptor = core.getCoreDescriptor().getCloudDescriptor();
>>     zk = core.getCoreContainer().getZkController(); // this is the
>> ZkController ref
>>     coreName = core.getCoreDescriptor().getName();
>>
>>     // Are we in cloud mode?
>>     if (zk != null) {
>>       collectionName = core.getCoreDescriptor().getCollectionName();
>>       shardId = cloudDescriptor.getShardId();
>>     } else {
>>       collectionName = null;
>>       shardId = null;
>>     }
>>
>> Depending on cloudMode we create new SolrClient instances based on these
>> classes.
>>
>> Check the apidocs and you'll quickly see what you need.
>>
>> We use these api's to get what we need. But you can also find these
>> things if you check the Java system properties, which is easier. We use the
>> api's to read the data because if api's change, we get a compile error. If
>> the system properties change, we don't. So the system properties is easier,
>> but the api's are safer. Although a unit tests should guard against that as
>> well.
>>
>> Regards,
>> Markus
>>
>> ps, on this list there is normally no need to create a new thread for an
>> existing one, even if you are eagerly waiting for a reply. It might take
>> some patience though.
>>
>> -----Original message-----
>> > From:Arnold Bronley <[hidden email]>
>> > Sent: Thursday 5th September 2019 18:44
>> > To: [hidden email]
>> > Subject: Re: SolrClient from inside processAdd function
>> >
>> > Hi Markus,
>> >
>> > Is there any way to get the information about the current Solr endpoint
>> > from within the custom URP?
>> >
>> > On Wed, Sep 4, 2019 at 3:10 PM Markus Jelsma <
>> [hidden email]>
>> > wrote:
>> >
>> > > Hello Arnold,
>> > >
>> > > Yes, we do this too for several cases.
>> > >
>> > > You can create the SolrClient in the Factory's inform() method, and
>> pass
>> > > is to the URP when it is created. You must implement SolrCoreAware and
>> > > close the client when the core closes as well. Use a CloseHook for
>> this.
>> > >
>> > > If you do not close the client, it will cause trouble if you run unit
>> > > tests, and most certainly when you regularly reload cores.
>> > >
>> > > Regards,
>> > > Markus
>> > >
>> > >
>> > >
>> > > -----Original message-----
>> > > > From:Arnold Bronley <[hidden email]>
>> > > > Sent: Wednesday 4th September 2019 20:10
>> > > > To: [hidden email]
>> > > > Subject: Re: SolrClient from inside processAdd function
>> > > >
>> > > > I need to search some other collection inside processAdd function
>> and
>> > > > append that information to the indexing request.
>> > > >
>> > > > On Tue, Sep 3, 2019 at 7:55 PM Erick Erickson <
>> [hidden email]>
>> > > > wrote:
>> > > >
>> > > > > This really sounds like an XY problem. What do you need the
>> SolrClient
>> > > > > _for_? I suspect there’s an easier way to do this…..
>> > > > >
>> > > > > Best,
>> > > > > Erick
>> > > > >
>> > > > > > On Sep 3, 2019, at 6:17 PM, Arnold Bronley <
>> [hidden email]>
>> > > > > wrote:
>> > > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Is there a way to create SolrClient from inside processAdd
>> function
>> > > for
>> > > > > > custom update processor for the same Solr on which it is
>> executing?
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>