How to synchronize the imports (DIH) delta imports

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

How to synchronize the imports (DIH) delta imports

Srinivas Kashyap-2
Hello,

We have our architecture of index server, where delta-imports run periodically based on modify_ts of the records.

We have another adhoc import handler on each core where import is called based on the key of solr core. This adhoc import is also called periodically.

We have scenario where multiple records are picked up for the adhoc import and the index server starts indexing them sequentially. At the subsequent time, if another adhoc import command is called, the records are not being indexed(skipped) as the solr core is busy re-indexing the earlier records.

Is there a way we can poll the import status of index server in SolrJ, so that we can refrain sending another adhoc import command while the index is still runnning?

Thanks and Regards,
Srinivas Kashyap
Senior Software Engineer
"GURUDAS HERITAGE"
100 Feet Ring Road,
Kadirenahalli,
Banashankari 2nd Stage,
Bangalore-560070
P:  973-986-6105
Bamboo Rose
The only B2B marketplace powered by proven trade engines.
www.BambooRose.com<http://www.bamboorose.com/>

Make Retail. Fun. Connected. Easier. Smarter. Together. Better.

________________________________

DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are confidential. If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
Reply | Threaded
Open this post in threaded view
|

Re: How to synchronize the imports (DIH) delta imports

Mikhail Khludnev-2
Hello, Srinivas.

You can literally poll import status.

On Wed, Jun 21, 2017 at 7:41 AM, Srinivas Kashyap <[hidden email]>
wrote:

> Hello,
>
> We have our architecture of index server, where delta-imports run
> periodically based on modify_ts of the records.
>
> We have another adhoc import handler on each core where import is called
> based on the key of solr core. This adhoc import is also called
> periodically.
>
> We have scenario where multiple records are picked up for the adhoc import
> and the index server starts indexing them sequentially. At the subsequent
> time, if another adhoc import command is called, the records are not being
> indexed(skipped) as the solr core is busy re-indexing the earlier records.
>
> Is there a way we can poll the import status of index server in SolrJ, so
> that we can refrain sending another adhoc import command while the index is
> still runnning?
>
> Thanks and Regards,
> Srinivas Kashyap
> Senior Software Engineer
> "GURUDAS HERITAGE"
> 100 Feet Ring Road,
> Kadirenahalli,
> Banashankari 2nd Stage,
> Bangalore-560070
> P:  973-986-6105
> Bamboo Rose
> The only B2B marketplace powered by proven trade engines.
> www.BambooRose.com<http://www.bamboorose.com/>
>
> Make Retail. Fun. Connected. Easier. Smarter. Together. Better.
>
> ________________________________
>
> DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are
> confidential. If you are not the intended recipient, please notify the
> sender immediately by replying to the e-mail, and then delete it without
> making copies or using it in any way. No representation is made that this
> email or any attachments are free of viruses. Virus scanning is recommended
> and is the responsibility of the recipient.
>



--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

RE: How to synchronize the imports (DIH) delta imports

Srinivas Kashyap-2
Thanks Mikhail,

Can you please explain the same? How can it be done in SolrJ

Thanks and Regards,
Srinivas Kashyap
Senior Software Engineer
“GURUDAS HERITAGE”
100 Feet Ring Road,
Kadirenahalli,
Banashankari 2nd Stage,
Bangalore-560070
P:  973-986-6105
Bamboo Rose
The only B2B marketplace powered by proven trade engines.
www.BambooRose.com

Make Retail. Fun. Connected. Easier. Smarter. Together. Better.

-----Original Message-----
From: Mikhail Khludnev [mailto:[hidden email]]
Sent: 21 June 2017 11:57 AM
To: solr-user <[hidden email]>
Subject: Re: How to synchronize the imports (DIH) delta imports

Hello, Srinivas.

You can literally poll import status.

On Wed, Jun 21, 2017 at 7:41 AM, Srinivas Kashyap <[hidden email]>
wrote:

> Hello,
>
> We have our architecture of index server, where delta-imports run
> periodically based on modify_ts of the records.
>
> We have another adhoc import handler on each core where import is
> called based on the key of solr core. This adhoc import is also called
> periodically.
>
> We have scenario where multiple records are picked up for the adhoc
> import and the index server starts indexing them sequentially. At the
> subsequent time, if another adhoc import command is called, the
> records are not being
> indexed(skipped) as the solr core is busy re-indexing the earlier records.
>
> Is there a way we can poll the import status of index server in SolrJ,
> so that we can refrain sending another adhoc import command while the
> index is still runnning?
>
> Thanks and Regards,
> Srinivas Kashyap
> Senior Software Engineer
> "GURUDAS HERITAGE"
> 100 Feet Ring Road,
> Kadirenahalli,
> Banashankari 2nd Stage,
> Bangalore-560070
> P:  973-986-6105
> Bamboo Rose
> The only B2B marketplace powered by proven trade engines.
> www.BambooRose.com<http://www.bamboorose.com/>
>
> Make Retail. Fun. Connected. Easier. Smarter. Together. Better.
>
> ________________________________
>
> DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are
> confidential. If you are not the intended recipient, please notify the
> sender immediately by replying to the e-mail, and then delete it
> without making copies or using it in any way. No representation is
> made that this email or any attachments are free of viruses. Virus
> scanning is recommended and is the responsibility of the recipient.
>



--
Sincerely yours
Mikhail Khludnev
________________________________

DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are confidential. If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
________________________________

DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are confidential. If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
Reply | Threaded
Open this post in threaded view
|

Re: How to synchronize the imports (DIH) delta imports

Mikhail Khludnev-2
Ok. It should be something like the scratch below (I'm sorry for full
package names).
The snippet below requests /dataimport status on db core and yields "idle"
string

org.apache.solr.client.solrj.impl.HttpSolrClient solrj=

  new org.apache.solr.client.solrj.impl.HttpSolrClient.Builder()

  .withBaseSolrUrl("http://localhost:8983/solr/").build();

org.apache.solr.common.util.NamedList<Object> statRsp= solrj.request(new
org.apache.solr.client.solrj.request.GenericSolrRequest(org.apache.solr.client.solrj.SolrRequest.METHOD.GET,

        "/dataimport",new
org.apache.solr.common.params.ModifiableSolrParams(){{

            add("command","STATUS");

        }}),"db");

statRsp.get("status");



On Wed, Jun 21, 2017 at 10:10 AM, Srinivas Kashyap <[hidden email]>
wrote:

> Thanks Mikhail,
>
> Can you please explain the same? How can it be done in SolrJ
>
> Thanks and Regards,
> Srinivas Kashyap
> Senior Software Engineer
> “GURUDAS HERITAGE”
> 100 Feet Ring Road,
> Kadirenahalli,
> Banashankari 2nd Stage,
> Bangalore-560070
> P:  973-986-6105
> Bamboo Rose
> The only B2B marketplace powered by proven trade engines.
> www.BambooRose.com
>
> Make Retail. Fun. Connected. Easier. Smarter. Together. Better.
>
> -----Original Message-----
> From: Mikhail Khludnev [mailto:[hidden email]]
> Sent: 21 June 2017 11:57 AM
> To: solr-user <[hidden email]>
> Subject: Re: How to synchronize the imports (DIH) delta imports
>
> Hello, Srinivas.
>
> You can literally poll import status.
>
> On Wed, Jun 21, 2017 at 7:41 AM, Srinivas Kashyap <[hidden email]
> >
> wrote:
>
> > Hello,
> >
> > We have our architecture of index server, where delta-imports run
> > periodically based on modify_ts of the records.
> >
> > We have another adhoc import handler on each core where import is
> > called based on the key of solr core. This adhoc import is also called
> > periodically.
> >
> > We have scenario where multiple records are picked up for the adhoc
> > import and the index server starts indexing them sequentially. At the
> > subsequent time, if another adhoc import command is called, the
> > records are not being
> > indexed(skipped) as the solr core is busy re-indexing the earlier
> records.
> >
> > Is there a way we can poll the import status of index server in SolrJ,
> > so that we can refrain sending another adhoc import command while the
> > index is still runnning?
> >
> > Thanks and Regards,
> > Srinivas Kashyap
> > Senior Software Engineer
> > "GURUDAS HERITAGE"
> > 100 Feet Ring Road,
> > Kadirenahalli,
> > Banashankari 2nd Stage,
> > Bangalore-560070
> > P:  973-986-6105
> > Bamboo Rose
> > The only B2B marketplace powered by proven trade engines.
> > www.BambooRose.com<http://www.bamboorose.com/>
> >
> > Make Retail. Fun. Connected. Easier. Smarter. Together. Better.
> >
> > ________________________________
> >
> > DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are
> > confidential. If you are not the intended recipient, please notify the
> > sender immediately by replying to the e-mail, and then delete it
> > without making copies or using it in any way. No representation is
> > made that this email or any attachments are free of viruses. Virus
> > scanning is recommended and is the responsibility of the recipient.
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> ________________________________
>
> DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are
> confidential. If you are not the intended recipient, please notify the
> sender immediately by replying to the e-mail, and then delete it without
> making copies or using it in any way. No representation is made that this
> email or any attachments are free of viruses. Virus scanning is recommended
> and is the responsibility of the recipient.
> ________________________________
>
> DISCLAIMER: E-mails and attachments from Bamboo Rose, LLC are
> confidential. If you are not the intended recipient, please notify the
> sender immediately by replying to the e-mail, and then delete it without
> making copies or using it in any way. No representation is made that this
> email or any attachments are free of viruses. Virus scanning is recommended
> and is the responsibility of the recipient.
>



--
Sincerely yours
Mikhail Khludnev