SolrPlugin update existing documents in newSearcher()

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

SolrPlugin update existing documents in newSearcher()

Maria Muslea
Hi,

I have a plugin that extends the AbstractSolrEventListener. I override the
newSearcher() method and the plan is to add some extra functionality,
namely updating existing documents by setting new values for existing
fields as well as adding new fields to the documents.

I can see that the plugin is invoked and I can get the list of documents,
but I cannot update existing fields or add new fields. I have tried various
approaches, but I cannot get it to work.

If you have any suggestions I would really appreciate it. The code that I
am currently trying is below.

Thank you,
Maria

     for (DocIterator iter = docs.iterator(); iter.hasNext();) {

        int doci = iter.nextDoc();

        Document document = newSearcher.doc(doci);



        SolrInputDocument solrInputDocument1 = new SolrInputDocument();

        AddUpdateCommand addUpdateCommand1 = new AddUpdateCommand(req);

        addUpdateCommand1.clear();

        solrInputDocument1.setField("id", document.get("id"));

        solrInputDocument1.addField("newfield", "newvalue");

        solrInputDocument1.setField("existingfield", "value");

        addUpdateCommand1.solrDoc = solrInputDocument1;

        getCore().getUpdateHandler().addDoc(addUpdateCommand1);


        SolrQueryResponse re = new SolrQueryResponse();

        SolrQueryRequest rq = new LocalSolrQueryRequest(getCore(), new
 ModifiableSolrParams());

        CommitUpdateCommand commit = new CommitUpdateCommand(rq,false);

         getCore().getUpdateHandler().commit(commit);


     }
Reply | Threaded
Open this post in threaded view
|

Re: SolrPlugin update existing documents in newSearcher()

Jan Høydahl / Cominvent
Hi

I don't see your requirement clearly. Sounds like what you really need is an UpdateRequestProcessor where you CAN intercept docs being added and modify them as you wish. https://lucene.apache.org/solr/guide/7_7/update-request-processors.html

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 1. mai 2019 kl. 22:31 skrev Maria Muslea <[hidden email]>:
>
> Hi,
>
> I have a plugin that extends the AbstractSolrEventListener. I override the
> newSearcher() method and the plan is to add some extra functionality,
> namely updating existing documents by setting new values for existing
> fields as well as adding new fields to the documents.
>
> I can see that the plugin is invoked and I can get the list of documents,
> but I cannot update existing fields or add new fields. I have tried various
> approaches, but I cannot get it to work.
>
> If you have any suggestions I would really appreciate it. The code that I
> am currently trying is below.
>
> Thank you,
> Maria
>
>     for (DocIterator iter = docs.iterator(); iter.hasNext();) {
>
>        int doci = iter.nextDoc();
>
>        Document document = newSearcher.doc(doci);
>
>
>
>        SolrInputDocument solrInputDocument1 = new SolrInputDocument();
>
>        AddUpdateCommand addUpdateCommand1 = new AddUpdateCommand(req);
>
>        addUpdateCommand1.clear();
>
>        solrInputDocument1.setField("id", document.get("id"));
>
>        solrInputDocument1.addField("newfield", "newvalue");
>
>        solrInputDocument1.setField("existingfield", "value");
>
>        addUpdateCommand1.solrDoc = solrInputDocument1;
>
>        getCore().getUpdateHandler().addDoc(addUpdateCommand1);
>
>
>        SolrQueryResponse re = new SolrQueryResponse();
>
>        SolrQueryRequest rq = new LocalSolrQueryRequest(getCore(), new
> ModifiableSolrParams());
>
>        CommitUpdateCommand commit = new CommitUpdateCommand(rq,false);
>
>         getCore().getUpdateHandler().commit(commit);
>
>
>     }

Reply | Threaded
Open this post in threaded view
|

Re: SolrPlugin update existing documents in newSearcher()

Maria Muslea
Yes, I will want to also do that, but initially I need to modify the docs
that are already in SOLR, and I thought of doing that at startup.
I am able to get the documents that I would like to modify, but the
operations for modifying the documents don't seem to be doing anything.

Do you see anything wrong with the way I am trying to modify the documents?

Thank you for your help,
Maria

On Thu, May 2, 2019 at 3:34 AM Jan Høydahl <[hidden email]> wrote:

> Hi
>
> I don't see your requirement clearly. Sounds like what you really need is
> an UpdateRequestProcessor where you CAN intercept docs being added and
> modify them as you wish.
> https://lucene.apache.org/solr/guide/7_7/update-request-processors.html
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 1. mai 2019 kl. 22:31 skrev Maria Muslea <[hidden email]>:
> >
> > Hi,
> >
> > I have a plugin that extends the AbstractSolrEventListener. I override
> the
> > newSearcher() method and the plan is to add some extra functionality,
> > namely updating existing documents by setting new values for existing
> > fields as well as adding new fields to the documents.
> >
> > I can see that the plugin is invoked and I can get the list of documents,
> > but I cannot update existing fields or add new fields. I have tried
> various
> > approaches, but I cannot get it to work.
> >
> > If you have any suggestions I would really appreciate it. The code that I
> > am currently trying is below.
> >
> > Thank you,
> > Maria
> >
> >     for (DocIterator iter = docs.iterator(); iter.hasNext();) {
> >
> >        int doci = iter.nextDoc();
> >
> >        Document document = newSearcher.doc(doci);
> >
> >
> >
> >        SolrInputDocument solrInputDocument1 = new SolrInputDocument();
> >
> >        AddUpdateCommand addUpdateCommand1 = new AddUpdateCommand(req);
> >
> >        addUpdateCommand1.clear();
> >
> >        solrInputDocument1.setField("id", document.get("id"));
> >
> >        solrInputDocument1.addField("newfield", "newvalue");
> >
> >        solrInputDocument1.setField("existingfield", "value");
> >
> >        addUpdateCommand1.solrDoc = solrInputDocument1;
> >
> >        getCore().getUpdateHandler().addDoc(addUpdateCommand1);
> >
> >
> >        SolrQueryResponse re = new SolrQueryResponse();
> >
> >        SolrQueryRequest rq = new LocalSolrQueryRequest(getCore(), new
> > ModifiableSolrParams());
> >
> >        CommitUpdateCommand commit = new CommitUpdateCommand(rq,false);
> >
> >         getCore().getUpdateHandler().commit(commit);
> >
> >
> >     }
>
>
Reply | Threaded
Open this post in threaded view
|

Re: SolrPlugin update existing documents in newSearcher()

Jan Høydahl / Cominvent
I think you cannot do that. The callback is sent AFTER a searcher is opened on the segment, so the index is already there.
Normally you re-index from source if you need changes in schema or processing.
If that is not possible, you must first check if ALL your fields are stored or docValues, if not you cannot hope to re-index from the content in the index in a lossless way. If you have everything stored, I'd create a new collection and write a script that reads all docs using cursorMark and indexes them into the new collection.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 2. mai 2019 kl. 19:39 skrev Maria Muslea <[hidden email]>:
>
> Yes, I will want to also do that, but initially I need to modify the docs
> that are already in SOLR, and I thought of doing that at startup.
> I am able to get the documents that I would like to modify, but the
> operations for modifying the documents don't seem to be doing anything.
>
> Do you see anything wrong with the way I am trying to modify the documents?
>
> Thank you for your help,
> Maria
>
> On Thu, May 2, 2019 at 3:34 AM Jan Høydahl <[hidden email]> wrote:
>
>> Hi
>>
>> I don't see your requirement clearly. Sounds like what you really need is
>> an UpdateRequestProcessor where you CAN intercept docs being added and
>> modify them as you wish.
>> https://lucene.apache.org/solr/guide/7_7/update-request-processors.html
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>>> 1. mai 2019 kl. 22:31 skrev Maria Muslea <[hidden email]>:
>>>
>>> Hi,
>>>
>>> I have a plugin that extends the AbstractSolrEventListener. I override
>> the
>>> newSearcher() method and the plan is to add some extra functionality,
>>> namely updating existing documents by setting new values for existing
>>> fields as well as adding new fields to the documents.
>>>
>>> I can see that the plugin is invoked and I can get the list of documents,
>>> but I cannot update existing fields or add new fields. I have tried
>> various
>>> approaches, but I cannot get it to work.
>>>
>>> If you have any suggestions I would really appreciate it. The code that I
>>> am currently trying is below.
>>>
>>> Thank you,
>>> Maria
>>>
>>>    for (DocIterator iter = docs.iterator(); iter.hasNext();) {
>>>
>>>       int doci = iter.nextDoc();
>>>
>>>       Document document = newSearcher.doc(doci);
>>>
>>>
>>>
>>>       SolrInputDocument solrInputDocument1 = new SolrInputDocument();
>>>
>>>       AddUpdateCommand addUpdateCommand1 = new AddUpdateCommand(req);
>>>
>>>       addUpdateCommand1.clear();
>>>
>>>       solrInputDocument1.setField("id", document.get("id"));
>>>
>>>       solrInputDocument1.addField("newfield", "newvalue");
>>>
>>>       solrInputDocument1.setField("existingfield", "value");
>>>
>>>       addUpdateCommand1.solrDoc = solrInputDocument1;
>>>
>>>       getCore().getUpdateHandler().addDoc(addUpdateCommand1);
>>>
>>>
>>>       SolrQueryResponse re = new SolrQueryResponse();
>>>
>>>       SolrQueryRequest rq = new LocalSolrQueryRequest(getCore(), new
>>> ModifiableSolrParams());
>>>
>>>       CommitUpdateCommand commit = new CommitUpdateCommand(rq,false);
>>>
>>>        getCore().getUpdateHandler().commit(commit);
>>>
>>>
>>>    }
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: SolrPlugin update existing documents in newSearcher()

Maria Muslea
Thank you so much, this is very helpful.

Maria

On Thu, May 2, 2019 at 2:39 PM Jan Høydahl <[hidden email]> wrote:

> I think you cannot do that. The callback is sent AFTER a searcher is
> opened on the segment, so the index is already there.
> Normally you re-index from source if you need changes in schema or
> processing.
> If that is not possible, you must first check if ALL your fields are
> stored or docValues, if not you cannot hope to re-index from the content in
> the index in a lossless way. If you have everything stored, I'd create a
> new collection and write a script that reads all docs using cursorMark and
> indexes them into the new collection.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 2. mai 2019 kl. 19:39 skrev Maria Muslea <[hidden email]>:
> >
> > Yes, I will want to also do that, but initially I need to modify the docs
> > that are already in SOLR, and I thought of doing that at startup.
> > I am able to get the documents that I would like to modify, but the
> > operations for modifying the documents don't seem to be doing anything.
> >
> > Do you see anything wrong with the way I am trying to modify the
> documents?
> >
> > Thank you for your help,
> > Maria
> >
> > On Thu, May 2, 2019 at 3:34 AM Jan Høydahl <[hidden email]>
> wrote:
> >
> >> Hi
> >>
> >> I don't see your requirement clearly. Sounds like what you really need
> is
> >> an UpdateRequestProcessor where you CAN intercept docs being added and
> >> modify them as you wish.
> >> https://lucene.apache.org/solr/guide/7_7/update-request-processors.html
> >>
> >> --
> >> Jan Høydahl, search solution architect
> >> Cominvent AS - www.cominvent.com
> >>
> >>> 1. mai 2019 kl. 22:31 skrev Maria Muslea <[hidden email]>:
> >>>
> >>> Hi,
> >>>
> >>> I have a plugin that extends the AbstractSolrEventListener. I override
> >> the
> >>> newSearcher() method and the plan is to add some extra functionality,
> >>> namely updating existing documents by setting new values for existing
> >>> fields as well as adding new fields to the documents.
> >>>
> >>> I can see that the plugin is invoked and I can get the list of
> documents,
> >>> but I cannot update existing fields or add new fields. I have tried
> >> various
> >>> approaches, but I cannot get it to work.
> >>>
> >>> If you have any suggestions I would really appreciate it. The code
> that I
> >>> am currently trying is below.
> >>>
> >>> Thank you,
> >>> Maria
> >>>
> >>>    for (DocIterator iter = docs.iterator(); iter.hasNext();) {
> >>>
> >>>       int doci = iter.nextDoc();
> >>>
> >>>       Document document = newSearcher.doc(doci);
> >>>
> >>>
> >>>
> >>>       SolrInputDocument solrInputDocument1 = new SolrInputDocument();
> >>>
> >>>       AddUpdateCommand addUpdateCommand1 = new AddUpdateCommand(req);
> >>>
> >>>       addUpdateCommand1.clear();
> >>>
> >>>       solrInputDocument1.setField("id", document.get("id"));
> >>>
> >>>       solrInputDocument1.addField("newfield", "newvalue");
> >>>
> >>>       solrInputDocument1.setField("existingfield", "value");
> >>>
> >>>       addUpdateCommand1.solrDoc = solrInputDocument1;
> >>>
> >>>       getCore().getUpdateHandler().addDoc(addUpdateCommand1);
> >>>
> >>>
> >>>       SolrQueryResponse re = new SolrQueryResponse();
> >>>
> >>>       SolrQueryRequest rq = new LocalSolrQueryRequest(getCore(), new
> >>> ModifiableSolrParams());
> >>>
> >>>       CommitUpdateCommand commit = new CommitUpdateCommand(rq,false);
> >>>
> >>>        getCore().getUpdateHandler().commit(commit);
> >>>
> >>>
> >>>    }
> >>
> >>
>
>