updating documents via csv

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

updating documents via csv

rhys J
Is there a way to update documents already stored in the solr cores via csv?

The reason I am asking is because I am running into a problem with updating
via script with single quotes embedded into the field itself.

Example:

curl http://localhost:8983/solr/dbtr/update?commit=true -d '[{ "id":
"356767", "name1": {"set": "NORTH AMERICAN INT'L"},"name2": {"set": " "}}]'

I have tried the following as well:

curl http://localhost:8983/solr/dbtr/update?commit=true -d '[{ "id":
"356767", "name1": {"set": "NORTH AMERICAN INT\'L"},"name2": {"set": " "}}]'

curl http://localhost:8983/solr/dbtr/update?commit=true -d '[{ "id":
"356767", "name1": {"set": "NORTH AMERICAN INT\\'L"},"name2": {"set": "
"}}]'

curl http://localhost:8983/solr/dbtr/update?commit=true -d '[{ \\"id\\":
\\"356767\\", \\"name1\\": {\\"set\\": \\"NORTH AMERICAN INT\\'L\\"},}]'

All of these break on the single quote embedded in field name1.

Does anyone have any ideas as to what I can do to get around this?

I will also eventually need to get around having an & inside a field too,
but that hasn't come up yet.

Thanks,

Rhys
Reply | Threaded
Open this post in threaded view
|

Starting Solr automatically

Anuj Bhargava
Often solr stops working. We have to then go to the root directory and give
the command *'service solr start*'

Is there a way to automatically start solr when it stops.

Regards,
Anuj

>
Reply | Threaded
Open this post in threaded view
|

Re: updating documents via csv

Paras Lehana
In reply to this post by rhys J
Hi Rhys,

I use CDATA for XMLs:

<field name1="title" update="set">   <![CDATA[NORTH AMERICAN INT'L]]>
 </field>

There should be a similar solution for JSON though I couldn't find the
specific one on the internet. If you are okay to use XMLs for indexing, you
can use this.

On Tue, 17 Dec 2019 at 01:40, rhys J <[hidden email]> wrote:

> Is there a way to update documents already stored in the solr cores via
> csv?
>
> The reason I am asking is because I am running into a problem with updating
> via script with single quotes embedded into the field itself.
>
> Example:
>
> curl http://localhost:8983/solr/dbtr/update?commit=true -d '[{ "id":
> "356767", "name1": {"set": "NORTH AMERICAN INT'L"},"name2": {"set": " "}}]'
>
> I have tried the following as well:
>
> curl http://localhost:8983/solr/dbtr/update?commit=true -d '[{ "id":
> "356767", "name1": {"set": "NORTH AMERICAN INT\'L"},"name2": {"set": "
> "}}]'
>
> curl http://localhost:8983/solr/dbtr/update?commit=true -d '[{ "id":
> "356767", "name1": {"set": "NORTH AMERICAN INT\\'L"},"name2": {"set": "
> "}}]'
>
> curl http://localhost:8983/solr/dbtr/update?commit=true -d '[{ \\"id\\":
> \\"356767\\", \\"name1\\": {\\"set\\": \\"NORTH AMERICAN INT\\'L\\"},}]'
>
> All of these break on the single quote embedded in field name1.
>
> Does anyone have any ideas as to what I can do to get around this?
>
> I will also eventually need to get around having an & inside a field too,
> but that hasn't come up yet.
>
> Thanks,
>
> Rhys
>


--
--
Regards,

*Paras Lehana* [65871]
Development Engineer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

--
*
*

 <https://www.facebook.com/IndiaMART/videos/578196442936091/>
Reply | Threaded
Open this post in threaded view
|

Re: Starting Solr automatically

Paras Lehana
In reply to this post by Anuj Bhargava
Hi Anuj,

Firstly, you should be checking into the logs for the reason of Solr
getting stopped. We had started Solr since a year ago and it's still up. I
guess OOM in your case.

Secondly, there are many ways to restart solr. For example, if it's
registered as a service, make a cron to restart solr whenever it's not
running.

But as I have said before, please look for the reason of solr getting
stopped.

On Tue, 17 Dec 2019 at 10:18, Anuj Bhargava <[hidden email]> wrote:

> Often solr stops working. We have to then go to the root directory and give
> the command *'service solr start*'
>
> Is there a way to automatically start solr when it stops.
>
> Regards,
> Anuj
>
> >
>


--
--
Regards,

*Paras Lehana* [65871]
Development Engineer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

--
*
*

 <https://www.facebook.com/IndiaMART/videos/578196442936091/>
Reply | Threaded
Open this post in threaded view
|

Re: updating documents via csv

rhys J
In reply to this post by Paras Lehana
On Mon, Dec 16, 2019 at 11:58 PM Paras Lehana <[hidden email]>
wrote:

> Hi Rhys,
>
> I use CDATA for XMLs:
>
> <field name1="title" update="set">   <![CDATA[NORTH AMERICAN INT'L]]>
>  </field>
>
> There should be a similar solution for JSON though I couldn't find the
> specific one on the internet. If you are okay to use XMLs for indexing, you
> can use this.
>
>
We are set on using json, but I figured out how to handle the single quote.

If i use curl " and then single quotes inside, I can escape the single
quote in the field with no problem.

Thanks for the help!

Rhys
Reply | Threaded
Open this post in threaded view
|

Re: updating documents via csv

Paras Lehana
Oh lol. How could I miss that! This is actually true for any bash command.
Glad that it worked.

On Wed, 18 Dec, 2019, 00:29 rhys J, <[hidden email]> wrote:

> On Mon, Dec 16, 2019 at 11:58 PM Paras Lehana <[hidden email]>
> wrote:
>
> > Hi Rhys,
> >
> > I use CDATA for XMLs:
> >
> > <field name1="title" update="set">   <![CDATA[NORTH AMERICAN INT'L]]>
> >  </field>
> >
> > There should be a similar solution for JSON though I couldn't find the
> > specific one on the internet. If you are okay to use XMLs for indexing,
> you
> > can use this.
> >
> >
> We are set on using json, but I figured out how to handle the single quote.
>
> If i use curl " and then single quotes inside, I can escape the single
> quote in the field with no problem.
>
> Thanks for the help!
>
> Rhys
>

--
*
*

 <https://www.facebook.com/IndiaMART/videos/578196442936091/>
Reply | Threaded
Open this post in threaded view
|

CVE-2017-7525 fix for Solr 7.7.x

Mehai, Lotfi
In reply to this post by rhys J
Hello;

We are using Solr 7.7.0. The CVE-2017-7525 have been fixed for Solr 8.x.
https://issues.apache.org/jira/browse/SOLR-13110

When the fix will be available for Solr 7.7.x

Lotfi
Reply | Threaded
Open this post in threaded view
|

Re: CVE-2017-7525 fix for Solr 7.7.x

Kevin Risden-3
There are no specific plans for any 7.x branch releases that I'm aware of.
Specifically for SOLR-13110, that required upgrading Hadoop 2.x to 3.x for
specifically jackson-mapper-asl and there are no plans to backport that to
7.x even if there was a future 7.x release.

Kevin Risden


On Wed, Dec 18, 2019 at 8:44 AM Mehai, Lotfi <[hidden email]>
wrote:

> Hello;
>
> We are using Solr 7.7.0. The CVE-2017-7525 have been fixed for Solr 8.x.
> https://issues.apache.org/jira/browse/SOLR-13110
>
> When the fix will be available for Solr 7.7.x
>
> Lotfi
>
Reply | Threaded
Open this post in threaded view
|

Re: Starting Solr automatically

Shawn Heisey-2
In reply to this post by Anuj Bhargava
On 12/16/2019 9:48 PM, Anuj Bhargava wrote:
> Often solr stops working. We have to then go to the root directory and give
> the command *'service solr start*'
>
> Is there a way to automatically start solr when it stops.

If Solr is stopping, then something went wrong.  Something that will
probably continue to go wrong if you simply restart Solr.  An OOM (out
of memory) like Paras mentioned in his reply is the most likely cause.

You haven't mentioned which version of Solr you've got.  Most recent
versions, when started on a non-Windows operating system, will
self-terminate if Java throws an OutOfMemoryError (OOME) exception.  If
this happens, a separate logfile with "oom" in its filename should be
created.  The reason that this happens is that operation of any Java
program is completely unpredictable after an OOME.

If OOME is happening, then you have exactly two ways to deal with it.
They are outlined here:

https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems#SolrPerformanceProblems-JavaHeap

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: CVE-2017-7525 fix for Solr 7.7.x

Colvin Cowie
In reply to this post by Kevin Risden-3
Hi,

We've got users on Solr 6 (and use Jackson ourselves), so I had a look at
this CVE and related Jackson exploits, to see whether they are actually
exploitable in Solr.

   - What parts of Solr actually use Jackson (I thought noggit was used for
   the JSON de/serialization)?
   - Do any of the object mappers used enable default typing? (which is
   necessary to exploit CVE-2017-7525
   https://adamcaudill.com/2017/10/04/exploiting-jackson-rce-cve-2017-7525/
   )
   - Is polymorphism used with Jackson without restricting subtypes (e.g.
   @JsonTypeInfo with JsonTypeInfo.Id.CLASS, which allows other exploits like
   CVE-2017-15095
   https://medium.com/@cowtowncoder/on-jackson-cves-dont-panic-here-is-what-you-need-to-know-54cd0d6e8062
   )

Aside from test classes, the only users of Jackson appear to be

   - org.apache.solr.analytics.AnalyticsRequestParser
   - org.apache.solr.prometheus.scraper.SolrScraper

From what I can see in the source on master and the 7_7 branch default
typing isn't ever enabled, and @JsonTypeInfo is restricted to named
subtypes.

In the 6_6 branch source it seems Jackson is only used in a handful of
tests.
Prior to Solr 6.3 (https://issues.apache.org/jira/browse/SOLR-9589)
org.apache.solr.client.solrj.response.DelegationTokenResponse.JsonMapResponseParser
constructed an ObjectMapper without configuration.

So, as far as I can see, the polymorphic deserialization Remote Code
Execution vulnerabilities on (older versions of) Jackson shouldn't actually
be exploitable in Solr 7.7... but I could be wrong, and new vulnerabilities
may still be discovered.

Colvin


On Wed, 18 Dec 2019 at 18:16, Kevin Risden <[hidden email]> wrote:

> There are no specific plans for any 7.x branch releases that I'm aware of.
> Specifically for SOLR-13110, that required upgrading Hadoop 2.x to 3.x for
> specifically jackson-mapper-asl and there are no plans to backport that to
> 7.x even if there was a future 7.x release.
>
> Kevin Risden
>
>
> On Wed, Dec 18, 2019 at 8:44 AM Mehai, Lotfi <[hidden email]>
> wrote:
>
> > Hello;
> >
> > We are using Solr 7.7.0. The CVE-2017-7525 have been fixed for Solr 8.x.
> > https://issues.apache.org/jira/browse/SOLR-13110
> >
> > When the fix will be available for Solr 7.7.x
> >
> > Lotfi
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: CVE-2017-7525 fix for Solr 7.7.x

Colvin Cowie
Sorry, in Solr 8 and master there are some additional users of Jackson. But
they still don't appear to use default typing or unrestricted subtypes.


On Thu, 19 Dec 2019 at 16:50, Colvin Cowie <[hidden email]>
wrote:

> Hi,
>
> We've got users on Solr 6 (and use Jackson ourselves), so I had a look at
> this CVE and related Jackson exploits, to see whether they are actually
> exploitable in Solr.
>
>    - What parts of Solr actually use Jackson (I thought noggit was used
>    for the JSON de/serialization)?
>    - Do any of the object mappers used enable default typing? (which is
>    necessary to exploit CVE-2017-7525
>    https://adamcaudill.com/2017/10/04/exploiting-jackson-rce-cve-2017-7525/
>    )
>    - Is polymorphism used with Jackson without restricting subtypes (e.g.
>    @JsonTypeInfo with JsonTypeInfo.Id.CLASS, which allows other exploits like
>    CVE-2017-15095
>    https://medium.com/@cowtowncoder/on-jackson-cves-dont-panic-here-is-what-you-need-to-know-54cd0d6e8062
>    )
>
> Aside from test classes, the only users of Jackson appear to be
>
>    - org.apache.solr.analytics.AnalyticsRequestParser
>    - org.apache.solr.prometheus.scraper.SolrScraper
>
> From what I can see in the source on master and the 7_7 branch default
> typing isn't ever enabled, and @JsonTypeInfo is restricted to named
> subtypes.
>
> In the 6_6 branch source it seems Jackson is only used in a handful of
> tests.
> Prior to Solr 6.3 (https://issues.apache.org/jira/browse/SOLR-9589)
> org.apache.solr.client.solrj.response.DelegationTokenResponse.JsonMapResponseParser
> constructed an ObjectMapper without configuration.
>
> So, as far as I can see, the polymorphic deserialization Remote Code
> Execution vulnerabilities on (older versions of) Jackson shouldn't actually
> be exploitable in Solr 7.7... but I could be wrong, and new vulnerabilities
> may still be discovered.
>
> Colvin
>
>
> On Wed, 18 Dec 2019 at 18:16, Kevin Risden <[hidden email]> wrote:
>
>> There are no specific plans for any 7.x branch releases that I'm aware of.
>> Specifically for SOLR-13110, that required upgrading Hadoop 2.x to 3.x for
>> specifically jackson-mapper-asl and there are no plans to backport that to
>> 7.x even if there was a future 7.x release.
>>
>> Kevin Risden
>>
>>
>> On Wed, Dec 18, 2019 at 8:44 AM Mehai, Lotfi <[hidden email]>
>> wrote:
>>
>> > Hello;
>> >
>> > We are using Solr 7.7.0. The CVE-2017-7525 have been fixed for Solr 8.x.
>> > https://issues.apache.org/jira/browse/SOLR-13110
>> >
>> > When the fix will be available for Solr 7.7.x
>> >
>> > Lotfi
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: CVE-2017-7525 fix for Solr 7.7.x

Mehai, Lotfi
Kevin & Colvin
Thanks for this details response.

Lotfi



On Thu, Dec 19, 2019 at 11:59 AM Colvin Cowie <[hidden email]>
wrote:

> Sorry, in Solr 8 and master there are some additional users of Jackson. But
> they still don't appear to use default typing or unrestricted subtypes.
>
>
> On Thu, 19 Dec 2019 at 16:50, Colvin Cowie <[hidden email]>
> wrote:
>
> > Hi,
> >
> > We've got users on Solr 6 (and use Jackson ourselves), so I had a look at
> > this CVE and related Jackson exploits, to see whether they are actually
> > exploitable in Solr.
> >
> >    - What parts of Solr actually use Jackson (I thought noggit was used
> >    for the JSON de/serialization)?
> >    - Do any of the object mappers used enable default typing? (which is
> >    necessary to exploit CVE-2017-7525
> >
> https://adamcaudill.com/2017/10/04/exploiting-jackson-rce-cve-2017-7525/
> >    )
> >    - Is polymorphism used with Jackson without restricting subtypes (e.g.
> >    @JsonTypeInfo with JsonTypeInfo.Id.CLASS, which allows other exploits
> like
> >    CVE-2017-15095
> >
> https://medium.com/@cowtowncoder/on-jackson-cves-dont-panic-here-is-what-you-need-to-know-54cd0d6e8062
> >    )
> >
> > Aside from test classes, the only users of Jackson appear to be
> >
> >    - org.apache.solr.analytics.AnalyticsRequestParser
> >    - org.apache.solr.prometheus.scraper.SolrScraper
> >
> > From what I can see in the source on master and the 7_7 branch default
> > typing isn't ever enabled, and @JsonTypeInfo is restricted to named
> > subtypes.
> >
> > In the 6_6 branch source it seems Jackson is only used in a handful of
> > tests.
> > Prior to Solr 6.3 (https://issues.apache.org/jira/browse/SOLR-9589)
> >
> org.apache.solr.client.solrj.response.DelegationTokenResponse.JsonMapResponseParser
> > constructed an ObjectMapper without configuration.
> >
> > So, as far as I can see, the polymorphic deserialization Remote Code
> > Execution vulnerabilities on (older versions of) Jackson shouldn't
> actually
> > be exploitable in Solr 7.7... but I could be wrong, and new
> vulnerabilities
> > may still be discovered.
> >
> > Colvin
> >
> >
> > On Wed, 18 Dec 2019 at 18:16, Kevin Risden <[hidden email]> wrote:
> >
> >> There are no specific plans for any 7.x branch releases that I'm aware
> of.
> >> Specifically for SOLR-13110, that required upgrading Hadoop 2.x to 3.x
> for
> >> specifically jackson-mapper-asl and there are no plans to backport that
> to
> >> 7.x even if there was a future 7.x release.
> >>
> >> Kevin Risden
> >>
> >>
> >> On Wed, Dec 18, 2019 at 8:44 AM Mehai, Lotfi <[hidden email]>
> >> wrote:
> >>
> >> > Hello;
> >> >
> >> > We are using Solr 7.7.0. The CVE-2017-7525 have been fixed for Solr
> 8.x.
> >> > https://issues.apache.org/jira/browse/SOLR-13110
> >> >
> >> > When the fix will be available for Solr 7.7.x
> >> >
> >> > Lotfi
> >> >
> >>
> >
>