Integrating Stanford NLP or any other NLP for Natural Language Query

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Integrating Stanford NLP or any other NLP for Natural Language Query

Puneet Pawaia
Hi

I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x if
required.
I am working on a POC for natural language query using Solr. Should I use
the Stanford libraries or are there any other libraries having integration
with Solr already available.
Any direction in how to do this would be most appreciated. How should I
process the query to give relevant results.

Regards
Puneet
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Joel Bernstein
You may want to take a look at NLP4J. There is no integration yet with
Solr, but it seems like it would be a good fit.

Joel Bernstein
http://joelsolr.blogspot.com/

On Thu, Jul 7, 2016 at 1:09 PM, Puneet Pawaia <[hidden email]>
wrote:

> Hi
>
> I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x if
> required.
> I am working on a POC for natural language query using Solr. Should I use
> the Stanford libraries or are there any other libraries having integration
> with Solr already available.
> Any direction in how to do this would be most appreciated. How should I
> process the query to give relevant results.
>
> Regards
> Puneet
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Jay Urbain
In reply to this post by Puneet Pawaia
I use Stanford NLP and cTakes (based on OpenNLP) while indexing with a
SOLRJ application.

Best,
Jay

On Thu, Jul 7, 2016 at 12:09 PM, Puneet Pawaia <[hidden email]>
wrote:

> Hi
>
> I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x if
> required.
> I am working on a POC for natural language query using Solr. Should I use
> the Stanford libraries or are there any other libraries having integration
> with Solr already available.
> Any direction in how to do this would be most appreciated. How should I
> process the query to give relevant results.
>
> Regards
> Puneet
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Yangrui Guo
In reply to this post by Puneet Pawaia
What is your NLP search like? I have a NLP solution for Solr and just open
sourced it. Not sure if it fits your need

Yangrui

On Thursday, July 7, 2016, Puneet Pawaia <[hidden email]> wrote:

> Hi
>
> I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x if
> required.
> I am working on a POC for natural language query using Solr. Should I use
> the Stanford libraries or are there any other libraries having integration
> with Solr already available.
> Any direction in how to do this would be most appreciated. How should I
> process the query to give relevant results.
>
> Regards
> Puneet
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

John Blythe
can you share a link, i'd be interested in checking it out.

thanks-

--
*John Blythe*
Product Manager & Lead Developer

251.605.3071 | [hidden email]
www.curvolabs.com

58 Adams Ave
Evansville, IN 47713

On Thu, Jul 7, 2016 at 4:32 PM, Yangrui Guo <[hidden email]> wrote:

> What is your NLP search like? I have a NLP solution for Solr and just open
> sourced it. Not sure if it fits your need
>
> Yangrui
>
> On Thursday, July 7, 2016, Puneet Pawaia <[hidden email]> wrote:
>
> > Hi
> >
> > I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x if
> > required.
> > I am working on a POC for natural language query using Solr. Should I use
> > the Stanford libraries or are there any other libraries having
> integration
> > with Solr already available.
> > Any direction in how to do this would be most appreciated. How should I
> > process the query to give relevant results.
> >
> > Regards
> > Puneet
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Yangrui Guo
https://github.com/guoyangrui/squery

It's not well documented yet but the idea was simple. Users should first
format their database tables into triples by creating view, then Solr and
Stanford NLP handles the data retrieval part. I hope someone could continue
contribute to its developement.

Yangrui

On Thursday, July 7, 2016, John Blythe <[hidden email]> wrote:

> can you share a link, i'd be interested in checking it out.
>
> thanks-
>
> --
> *John Blythe*
> Product Manager & Lead Developer
>
> 251.605.3071 | [hidden email] <javascript:;>
> www.curvolabs.com
>
> 58 Adams Ave
> Evansville, IN 47713
>
> On Thu, Jul 7, 2016 at 4:32 PM, Yangrui Guo <[hidden email]
> <javascript:;>> wrote:
>
> > What is your NLP search like? I have a NLP solution for Solr and just
> open
> > sourced it. Not sure if it fits your need
> >
> > Yangrui
> >
> > On Thursday, July 7, 2016, Puneet Pawaia <[hidden email]
> <javascript:;>> wrote:
> >
> > > Hi
> > >
> > > I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x if
> > > required.
> > > I am working on a POC for natural language query using Solr. Should I
> use
> > > the Stanford libraries or are there any other libraries having
> > integration
> > > with Solr already available.
> > > Any direction in how to do this would be most appreciated. How should I
> > > process the query to give relevant results.
> > >
> > > Regards
> > > Puneet
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Puneet Pawaia
In reply to this post by Yangrui Guo
Hi  Yangrui
I would like users to be able to write queries in natural language rather
than keyword based search.
A link to your solution would be worth looking at.
Regards
Puneet

On 8 Jul 2016 03:02, "Yangrui Guo" <[hidden email]> wrote:

What is your NLP search like? I have a NLP solution for Solr and just open
sourced it. Not sure if it fits your need

Yangrui

On Thursday, July 7, 2016, Puneet Pawaia <[hidden email]> wrote:

> Hi
>
> I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x if
> required.
> I am working on a POC for natural language query using Solr. Should I use
> the Stanford libraries or are there any other libraries having integration
> with Solr already available.
> Any direction in how to do this would be most appreciated. How should I
> process the query to give relevant results.
>
> Regards
> Puneet
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Puneet Pawaia
In reply to this post by Jay Urbain
Hi Jay
Any place I can learn more on this method of integration?
Thanks
Puneet

On 8 Jul 2016 02:58, "Jay Urbain" <[hidden email]> wrote:

> I use Stanford NLP and cTakes (based on OpenNLP) while indexing with a
> SOLRJ application.
>
> Best,
> Jay
>
> On Thu, Jul 7, 2016 at 12:09 PM, Puneet Pawaia <[hidden email]>
> wrote:
>
> > Hi
> >
> > I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x if
> > required.
> > I am working on a POC for natural language query using Solr. Should I use
> > the Stanford libraries or are there any other libraries having
> integration
> > with Solr already available.
> > Any direction in how to do this would be most appreciated. How should I
> > process the query to give relevant results.
> >
> > Regards
> > Puneet
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Puneet Pawaia
In reply to this post by Yangrui Guo
Thanks for the link.
I'll take a look at it later in the day once I am at office.
Puneet

On 8 Jul 2016 08:19, "Yangrui Guo" <[hidden email]> wrote:

https://github.com/guoyangrui/squery

It's not well documented yet but the idea was simple. Users should first
format their database tables into triples by creating view, then Solr and
Stanford NLP handles the data retrieval part. I hope someone could continue
contribute to its developement.

Yangrui

On Thursday, July 7, 2016, John Blythe <[hidden email]> wrote:

> can you share a link, i'd be interested in checking it out.
>
> thanks-
>
> --
> *John Blythe*
> Product Manager & Lead Developer
>
> 251.605.3071 | [hidden email] <javascript:;>
> www.curvolabs.com
>
> 58 Adams Ave
> Evansville, IN 47713
>
> On Thu, Jul 7, 2016 at 4:32 PM, Yangrui Guo <[hidden email]
> <javascript:;>> wrote:
>
> > What is your NLP search like? I have a NLP solution for Solr and just
> open
> > sourced it. Not sure if it fits your need
> >
> > Yangrui
> >
> > On Thursday, July 7, 2016, Puneet Pawaia <[hidden email]
> <javascript:;>> wrote:
> >
> > > Hi
> > >
> > > I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x if
> > > required.
> > > I am working on a POC for natural language query using Solr. Should I
> use
> > > the Stanford libraries or are there any other libraries having
> > integration
> > > with Solr already available.
> > > Any direction in how to do this would be most appreciated. How should
I
> > > process the query to give relevant results.
> > >
> > > Regards
> > > Puneet
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Yangrui Guo
In reply to this post by Puneet Pawaia
My solution lets users retrieve data entities using queries like "find me a
job that only requires a high school degree" and "I want a car from
American with alloy wheels". It can also be expanded to perform other
database queries, like date time or price range searches. I use Stanford
NLP to identify the main entity and its related attributes in a user
sentence.

Yangrui

On Thursday, July 7, 2016, Puneet Pawaia <[hidden email]> wrote:

> Hi  Yangrui
> I would like users to be able to write queries in natural language rather
> than keyword based search.
> A link to your solution would be worth looking at.
> Regards
> Puneet
>
> On 8 Jul 2016 03:02, "Yangrui Guo" <[hidden email] <javascript:;>>
> wrote:
>
> What is your NLP search like? I have a NLP solution for Solr and just open
> sourced it. Not sure if it fits your need
>
> Yangrui
>
> On Thursday, July 7, 2016, Puneet Pawaia <[hidden email]
> <javascript:;>> wrote:
>
> > Hi
> >
> > I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x if
> > required.
> > I am working on a POC for natural language query using Solr. Should I use
> > the Stanford libraries or are there any other libraries having
> integration
> > with Solr already available.
> > Any direction in how to do this would be most appreciated. How should I
> > process the query to give relevant results.
> >
> > Regards
> > Puneet
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Jay Urbain
In reply to this post by Puneet Pawaia
I've added multivalued fields within my SOLR schema for indexing entities
extracted using NLP methods applied to the text I'm indexing, along with
fields for other discrete data extracted from relational databases.

A Java application reads data out of multiple relational databases, uses
NLP on the text and indexes each document (de-normalized) using SOLRJ.

I initially tried doing this with content handlers, but found it much
easier to just write a Java application.

SOLRJ Java API reference:
https://cwiki.apache.org/confluence/display/solr/Using+SolrJ

Stanford NLP:
http://stanfordnlp.github.io/CoreNLP/

Best,
Jay


On Thu, Jul 7, 2016 at 9:52 PM, Puneet Pawaia <[hidden email]>
wrote:

> Hi Jay
> Any place I can learn more on this method of integration?
> Thanks
> Puneet
>
> On 8 Jul 2016 02:58, "Jay Urbain" <[hidden email]> wrote:
>
> > I use Stanford NLP and cTakes (based on OpenNLP) while indexing with a
> > SOLRJ application.
> >
> > Best,
> > Jay
> >
> > On Thu, Jul 7, 2016 at 12:09 PM, Puneet Pawaia <[hidden email]>
> > wrote:
> >
> > > Hi
> > >
> > > I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x if
> > > required.
> > > I am working on a POC for natural language query using Solr. Should I
> use
> > > the Stanford libraries or are there any other libraries having
> > integration
> > > with Solr already available.
> > > Any direction in how to do this would be most appreciated. How should I
> > > process the query to give relevant results.
> > >
> > > Regards
> > > Puneet
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Alessandro Benedetti
Hi Puneet,
your requirement :
"I would like users to be able to write queries in natural language rather
than keyword based search."

Is really really vague :(
Can you try to help us with some specific example, starting of course from
the simplest use cases you have initially in mind ?

Moving from keyword based search to natural language is a really complex
task.
Proceeding step by step can help you.

Do you want for example to set up a Q&A basic system ?
In that case you should take care of query rewriting.
You need basically to identify your base requirement and then build a
specific parser for that.
You can use triple stores and knowledge bases to enrich both your query and
your index, but let's start from the basis, what is your simplest
requirement ?

On Fri, Jul 8, 2016 at 1:56 PM, Jay Urbain <[hidden email]> wrote:

> I've added multivalued fields within my SOLR schema for indexing entities
> extracted using NLP methods applied to the text I'm indexing, along with
> fields for other discrete data extracted from relational databases.
>
> A Java application reads data out of multiple relational databases, uses
> NLP on the text and indexes each document (de-normalized) using SOLRJ.
>
> I initially tried doing this with content handlers, but found it much
> easier to just write a Java application.
>
> SOLRJ Java API reference:
> https://cwiki.apache.org/confluence/display/solr/Using+SolrJ
>
> Stanford NLP:
> http://stanfordnlp.github.io/CoreNLP/
>
> Best,
> Jay
>
>
> On Thu, Jul 7, 2016 at 9:52 PM, Puneet Pawaia <[hidden email]>
> wrote:
>
> > Hi Jay
> > Any place I can learn more on this method of integration?
> > Thanks
> > Puneet
> >
> > On 8 Jul 2016 02:58, "Jay Urbain" <[hidden email]> wrote:
> >
> > > I use Stanford NLP and cTakes (based on OpenNLP) while indexing with a
> > > SOLRJ application.
> > >
> > > Best,
> > > Jay
> > >
> > > On Thu, Jul 7, 2016 at 12:09 PM, Puneet Pawaia <
> [hidden email]>
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x
> if
> > > > required.
> > > > I am working on a POC for natural language query using Solr. Should I
> > use
> > > > the Stanford libraries or are there any other libraries having
> > > integration
> > > > with Solr already available.
> > > > Any direction in how to do this would be most appreciated. How
> should I
> > > > process the query to give relevant results.
> > > >
> > > > Regards
> > > > Puneet
> > > >
> > >
> >
>



--
--------------------------

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Puneet Pawaia
Hi Alessandro

I am looking at being able to answer questions like "Can a non-compete
clause in an employment agreement be enforced after the expiry of the
agreement?"

On Sat, Jul 9, 2016 at 4:34 AM, Alessandro Benedetti <[hidden email]>
wrote:

> Hi Puneet,
> your requirement :
> "I would like users to be able to write queries in natural language rather
> than keyword based search."
>
> Is really really vague :(
> Can you try to help us with some specific example, starting of course from
> the simplest use cases you have initially in mind ?
>
> Moving from keyword based search to natural language is a really complex
> task.
> Proceeding step by step can help you.
>
> Do you want for example to set up a Q&A basic system ?
> In that case you should take care of query rewriting.
> You need basically to identify your base requirement and then build a
> specific parser for that.
> You can use triple stores and knowledge bases to enrich both your query and
> your index, but let's start from the basis, what is your simplest
> requirement ?
>
> On Fri, Jul 8, 2016 at 1:56 PM, Jay Urbain <[hidden email]> wrote:
>
> > I've added multivalued fields within my SOLR schema for indexing entities
> > extracted using NLP methods applied to the text I'm indexing, along with
> > fields for other discrete data extracted from relational databases.
> >
> > A Java application reads data out of multiple relational databases, uses
> > NLP on the text and indexes each document (de-normalized) using SOLRJ.
> >
> > I initially tried doing this with content handlers, but found it much
> > easier to just write a Java application.
> >
> > SOLRJ Java API reference:
> > https://cwiki.apache.org/confluence/display/solr/Using+SolrJ
> >
> > Stanford NLP:
> > http://stanfordnlp.github.io/CoreNLP/
> >
> > Best,
> > Jay
> >
> >
> > On Thu, Jul 7, 2016 at 9:52 PM, Puneet Pawaia <[hidden email]>
> > wrote:
> >
> > > Hi Jay
> > > Any place I can learn more on this method of integration?
> > > Thanks
> > > Puneet
> > >
> > > On 8 Jul 2016 02:58, "Jay Urbain" <[hidden email]> wrote:
> > >
> > > > I use Stanford NLP and cTakes (based on OpenNLP) while indexing with
> a
> > > > SOLRJ application.
> > > >
> > > > Best,
> > > > Jay
> > > >
> > > > On Thu, Jul 7, 2016 at 12:09 PM, Puneet Pawaia <
> > [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > I am currently using Solr 5.5.x to test but can upgrade to Solr 6.x
> > if
> > > > > required.
> > > > > I am working on a POC for natural language query using Solr.
> Should I
> > > use
> > > > > the Stanford libraries or are there any other libraries having
> > > > integration
> > > > > with Solr already available.
> > > > > Any direction in how to do this would be most appreciated. How
> > should I
> > > > > process the query to give relevant results.
> > > > >
> > > > > Regards
> > > > > Puneet
> > > > >
> > > >
> > >
> >
>
>
>
> --
> --------------------------
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Puneet Pawaia
Hi Alessandro

I am looking at being able to answer questions like "Can a non-compete
clause in an employment agreement be enforced after the expiry of the
agreement?"
We are doing some testing with IBM Watson and with a sample test data, we
are able to get relevant replies to the above question. Since IBM Watson
uses Solr at its backend, I was wondering if we can get the same working at
the Solr level without having to use Watson.

Regards
Puneet

On Sat, Jul 9, 2016 at 11:34 AM, Puneet Pawaia <[hidden email]>
wrote:

> Hi Alessandro
>
> I am looking at being able to answer questions like "Can a non-compete
> clause in an employment agreement be enforced after the expiry of the
> agreement?"
>
> On Sat, Jul 9, 2016 at 4:34 AM, Alessandro Benedetti <
> [hidden email]> wrote:
>
>> Hi Puneet,
>> your requirement :
>> "I would like users to be able to write queries in natural language rather
>> than keyword based search."
>>
>> Is really really vague :(
>> Can you try to help us with some specific example, starting of course from
>> the simplest use cases you have initially in mind ?
>>
>> Moving from keyword based search to natural language is a really complex
>> task.
>> Proceeding step by step can help you.
>>
>> Do you want for example to set up a Q&A basic system ?
>> In that case you should take care of query rewriting.
>> You need basically to identify your base requirement and then build a
>> specific parser for that.
>> You can use triple stores and knowledge bases to enrich both your query
>> and
>> your index, but let's start from the basis, what is your simplest
>> requirement ?
>>
>> On Fri, Jul 8, 2016 at 1:56 PM, Jay Urbain <[hidden email]> wrote:
>>
>> > I've added multivalued fields within my SOLR schema for indexing
>> entities
>> > extracted using NLP methods applied to the text I'm indexing, along with
>> > fields for other discrete data extracted from relational databases.
>> >
>> > A Java application reads data out of multiple relational databases, uses
>> > NLP on the text and indexes each document (de-normalized) using SOLRJ.
>> >
>> > I initially tried doing this with content handlers, but found it much
>> > easier to just write a Java application.
>> >
>> > SOLRJ Java API reference:
>> > https://cwiki.apache.org/confluence/display/solr/Using+SolrJ
>> >
>> > Stanford NLP:
>> > http://stanfordnlp.github.io/CoreNLP/
>> >
>> > Best,
>> > Jay
>> >
>> >
>> > On Thu, Jul 7, 2016 at 9:52 PM, Puneet Pawaia <[hidden email]>
>> > wrote:
>> >
>> > > Hi Jay
>> > > Any place I can learn more on this method of integration?
>> > > Thanks
>> > > Puneet
>> > >
>> > > On 8 Jul 2016 02:58, "Jay Urbain" <[hidden email]> wrote:
>> > >
>> > > > I use Stanford NLP and cTakes (based on OpenNLP) while indexing
>> with a
>> > > > SOLRJ application.
>> > > >
>> > > > Best,
>> > > > Jay
>> > > >
>> > > > On Thu, Jul 7, 2016 at 12:09 PM, Puneet Pawaia <
>> > [hidden email]>
>> > > > wrote:
>> > > >
>> > > > > Hi
>> > > > >
>> > > > > I am currently using Solr 5.5.x to test but can upgrade to Solr
>> 6.x
>> > if
>> > > > > required.
>> > > > > I am working on a POC for natural language query using Solr.
>> Should I
>> > > use
>> > > > > the Stanford libraries or are there any other libraries having
>> > > > integration
>> > > > > with Solr already available.
>> > > > > Any direction in how to do this would be most appreciated. How
>> > should I
>> > > > > process the query to give relevant results.
>> > > > >
>> > > > > Regards
>> > > > > Puneet
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> --------------------------
>>
>> Benedetti Alessandro
>> Visiting card : http://about.me/alessandro_benedetti
>>
>> "Tyger, tyger burning bright
>> In the forests of the night,
>> What immortal hand or eye
>> Could frame thy fearful symmetry?"
>>
>> William Blake - Songs of Experience -1794 England
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Puneet Pawaia
Hi Yangrui,

I have been looking at your code for squery.
Unfortunately, I am not very conversant with SolrJ.  I seem to be missing
how and what data is added to the Solr index.
Also, I see some references to IBM Watson in your code. Are you using IBM
Watson? If yes, then why use the Stanford NLP if you can use the Watson NLP?

Regards
Puneet


On Sat, Jul 9, 2016 at 11:37 AM, Puneet Pawaia <[hidden email]>
wrote:

> Hi Alessandro
>
> I am looking at being able to answer questions like "Can a non-compete
> clause in an employment agreement be enforced after the expiry of the
> agreement?"
> We are doing some testing with IBM Watson and with a sample test data, we
> are able to get relevant replies to the above question. Since IBM Watson
> uses Solr at its backend, I was wondering if we can get the same working at
> the Solr level without having to use Watson.
>
> Regards
> Puneet
>
> On Sat, Jul 9, 2016 at 11:34 AM, Puneet Pawaia <[hidden email]>
> wrote:
>
>> Hi Alessandro
>>
>> I am looking at being able to answer questions like "Can a non-compete
>> clause in an employment agreement be enforced after the expiry of the
>> agreement?"
>>
>> On Sat, Jul 9, 2016 at 4:34 AM, Alessandro Benedetti <
>> [hidden email]> wrote:
>>
>>> Hi Puneet,
>>> your requirement :
>>> "I would like users to be able to write queries in natural language
>>> rather
>>> than keyword based search."
>>>
>>> Is really really vague :(
>>> Can you try to help us with some specific example, starting of course
>>> from
>>> the simplest use cases you have initially in mind ?
>>>
>>> Moving from keyword based search to natural language is a really complex
>>> task.
>>> Proceeding step by step can help you.
>>>
>>> Do you want for example to set up a Q&A basic system ?
>>> In that case you should take care of query rewriting.
>>> You need basically to identify your base requirement and then build a
>>> specific parser for that.
>>> You can use triple stores and knowledge bases to enrich both your query
>>> and
>>> your index, but let's start from the basis, what is your simplest
>>> requirement ?
>>>
>>> On Fri, Jul 8, 2016 at 1:56 PM, Jay Urbain <[hidden email]> wrote:
>>>
>>> > I've added multivalued fields within my SOLR schema for indexing
>>> entities
>>> > extracted using NLP methods applied to the text I'm indexing, along
>>> with
>>> > fields for other discrete data extracted from relational databases.
>>> >
>>> > A Java application reads data out of multiple relational databases,
>>> uses
>>> > NLP on the text and indexes each document (de-normalized) using SOLRJ.
>>> >
>>> > I initially tried doing this with content handlers, but found it much
>>> > easier to just write a Java application.
>>> >
>>> > SOLRJ Java API reference:
>>> > https://cwiki.apache.org/confluence/display/solr/Using+SolrJ
>>> >
>>> > Stanford NLP:
>>> > http://stanfordnlp.github.io/CoreNLP/
>>> >
>>> > Best,
>>> > Jay
>>> >
>>> >
>>> > On Thu, Jul 7, 2016 at 9:52 PM, Puneet Pawaia <[hidden email]
>>> >
>>> > wrote:
>>> >
>>> > > Hi Jay
>>> > > Any place I can learn more on this method of integration?
>>> > > Thanks
>>> > > Puneet
>>> > >
>>> > > On 8 Jul 2016 02:58, "Jay Urbain" <[hidden email]> wrote:
>>> > >
>>> > > > I use Stanford NLP and cTakes (based on OpenNLP) while indexing
>>> with a
>>> > > > SOLRJ application.
>>> > > >
>>> > > > Best,
>>> > > > Jay
>>> > > >
>>> > > > On Thu, Jul 7, 2016 at 12:09 PM, Puneet Pawaia <
>>> > [hidden email]>
>>> > > > wrote:
>>> > > >
>>> > > > > Hi
>>> > > > >
>>> > > > > I am currently using Solr 5.5.x to test but can upgrade to Solr
>>> 6.x
>>> > if
>>> > > > > required.
>>> > > > > I am working on a POC for natural language query using Solr.
>>> Should I
>>> > > use
>>> > > > > the Stanford libraries or are there any other libraries having
>>> > > > integration
>>> > > > > with Solr already available.
>>> > > > > Any direction in how to do this would be most appreciated. How
>>> > should I
>>> > > > > process the query to give relevant results.
>>> > > > >
>>> > > > > Regards
>>> > > > > Puneet
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> --------------------------
>>>
>>> Benedetti Alessandro
>>> Visiting card : http://about.me/alessandro_benedetti
>>>
>>> "Tyger, tyger burning bright
>>> In the forests of the night,
>>> What immortal hand or eye
>>> Could frame thy fearful symmetry?"
>>>
>>> William Blake - Songs of Experience -1794 England
>>>
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Yangrui Guo
Hi Puneet,

I only use Watson's text to speech as user interface, because a lot of
people think NLP is the same as voice recognition. If you don't need voice
recognition you could remove Watson from it. Stanford has better dependency
parsing and can be used offline. However it seems you are using Watson's
retrieve and rank API, which is based on Solr, am I correct?

Yangrui

On Saturday, July 9, 2016, Puneet Pawaia <[hidden email]> wrote:

> Hi Yangrui,
>
> I have been looking at your code for squery.
> Unfortunately, I am not very conversant with SolrJ.  I seem to be missing
> how and what data is added to the Solr index.
> Also, I see some references to IBM Watson in your code. Are you using IBM
> Watson? If yes, then why use the Stanford NLP if you can use the Watson
> NLP?
>
> Regards
> Puneet
>
>
> On Sat, Jul 9, 2016 at 11:37 AM, Puneet Pawaia <[hidden email]
> <javascript:;>>
> wrote:
>
> > Hi Alessandro
> >
> > I am looking at being able to answer questions like "Can a non-compete
> > clause in an employment agreement be enforced after the expiry of the
> > agreement?"
> > We are doing some testing with IBM Watson and with a sample test data, we
> > are able to get relevant replies to the above question. Since IBM Watson
> > uses Solr at its backend, I was wondering if we can get the same working
> at
> > the Solr level without having to use Watson.
> >
> > Regards
> > Puneet
> >
> > On Sat, Jul 9, 2016 at 11:34 AM, Puneet Pawaia <[hidden email]
> <javascript:;>>
> > wrote:
> >
> >> Hi Alessandro
> >>
> >> I am looking at being able to answer questions like "Can a non-compete
> >> clause in an employment agreement be enforced after the expiry of the
> >> agreement?"
> >>
> >> On Sat, Jul 9, 2016 at 4:34 AM, Alessandro Benedetti <
> >> [hidden email] <javascript:;>> wrote:
> >>
> >>> Hi Puneet,
> >>> your requirement :
> >>> "I would like users to be able to write queries in natural language
> >>> rather
> >>> than keyword based search."
> >>>
> >>> Is really really vague :(
> >>> Can you try to help us with some specific example, starting of course
> >>> from
> >>> the simplest use cases you have initially in mind ?
> >>>
> >>> Moving from keyword based search to natural language is a really
> complex
> >>> task.
> >>> Proceeding step by step can help you.
> >>>
> >>> Do you want for example to set up a Q&A basic system ?
> >>> In that case you should take care of query rewriting.
> >>> You need basically to identify your base requirement and then build a
> >>> specific parser for that.
> >>> You can use triple stores and knowledge bases to enrich both your query
> >>> and
> >>> your index, but let's start from the basis, what is your simplest
> >>> requirement ?
> >>>
> >>> On Fri, Jul 8, 2016 at 1:56 PM, Jay Urbain <[hidden email]
> <javascript:;>> wrote:
> >>>
> >>> > I've added multivalued fields within my SOLR schema for indexing
> >>> entities
> >>> > extracted using NLP methods applied to the text I'm indexing, along
> >>> with
> >>> > fields for other discrete data extracted from relational databases.
> >>> >
> >>> > A Java application reads data out of multiple relational databases,
> >>> uses
> >>> > NLP on the text and indexes each document (de-normalized) using
> SOLRJ.
> >>> >
> >>> > I initially tried doing this with content handlers, but found it much
> >>> > easier to just write a Java application.
> >>> >
> >>> > SOLRJ Java API reference:
> >>> > https://cwiki.apache.org/confluence/display/solr/Using+SolrJ
> >>> >
> >>> > Stanford NLP:
> >>> > http://stanfordnlp.github.io/CoreNLP/
> >>> >
> >>> > Best,
> >>> > Jay
> >>> >
> >>> >
> >>> > On Thu, Jul 7, 2016 at 9:52 PM, Puneet Pawaia <
> [hidden email] <javascript:;>
> >>> >
> >>> > wrote:
> >>> >
> >>> > > Hi Jay
> >>> > > Any place I can learn more on this method of integration?
> >>> > > Thanks
> >>> > > Puneet
> >>> > >
> >>> > > On 8 Jul 2016 02:58, "Jay Urbain" <[hidden email]
> <javascript:;>> wrote:
> >>> > >
> >>> > > > I use Stanford NLP and cTakes (based on OpenNLP) while indexing
> >>> with a
> >>> > > > SOLRJ application.
> >>> > > >
> >>> > > > Best,
> >>> > > > Jay
> >>> > > >
> >>> > > > On Thu, Jul 7, 2016 at 12:09 PM, Puneet Pawaia <
> >>> > [hidden email] <javascript:;>>
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Hi
> >>> > > > >
> >>> > > > > I am currently using Solr 5.5.x to test but can upgrade to Solr
> >>> 6.x
> >>> > if
> >>> > > > > required.
> >>> > > > > I am working on a POC for natural language query using Solr.
> >>> Should I
> >>> > > use
> >>> > > > > the Stanford libraries or are there any other libraries having
> >>> > > > integration
> >>> > > > > with Solr already available.
> >>> > > > > Any direction in how to do this would be most appreciated. How
> >>> > should I
> >>> > > > > process the query to give relevant results.
> >>> > > > >
> >>> > > > > Regards
> >>> > > > > Puneet
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> --------------------------
> >>>
> >>> Benedetti Alessandro
> >>> Visiting card : http://about.me/alessandro_benedetti
> >>>
> >>> "Tyger, tyger burning bright
> >>> In the forests of the night,
> >>> What immortal hand or eye
> >>> Could frame thy fearful symmetry?"
> >>>
> >>> William Blake - Songs of Experience -1794 England
> >>>
> >>
> >>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Puneet Pawaia
Hi Yangrui
We are testing the Rank and Retrieve as well as the NLP interface. However
this is being done by another team and so I would not be able to comment
further on it.
I would like to know what kind of Solr field you are using for storing the
output from your classes. And also what function output you are putting
into that field.
Thanks
Puneet

On 10 Jul 2016 00:17, "Yangrui Guo" <[hidden email]> wrote:

> Hi Puneet,
>
> I only use Watson's text to speech as user interface, because a lot of
> people think NLP is the same as voice recognition. If you don't need voice
> recognition you could remove Watson from it. Stanford has better dependency
> parsing and can be used offline. However it seems you are using Watson's
> retrieve and rank API, which is based on Solr, am I correct?
>
> Yangrui
>
> On Saturday, July 9, 2016, Puneet Pawaia <[hidden email]> wrote:
>
> > Hi Yangrui,
> >
> > I have been looking at your code for squery.
> > Unfortunately, I am not very conversant with SolrJ.  I seem to be missing
> > how and what data is added to the Solr index.
> > Also, I see some references to IBM Watson in your code. Are you using IBM
> > Watson? If yes, then why use the Stanford NLP if you can use the Watson
> > NLP?
> >
> > Regards
> > Puneet
> >
> >
> > On Sat, Jul 9, 2016 at 11:37 AM, Puneet Pawaia <[hidden email]
> > <javascript:;>>
> > wrote:
> >
> > > Hi Alessandro
> > >
> > > I am looking at being able to answer questions like "Can a non-compete
> > > clause in an employment agreement be enforced after the expiry of the
> > > agreement?"
> > > We are doing some testing with IBM Watson and with a sample test data,
> we
> > > are able to get relevant replies to the above question. Since IBM
> Watson
> > > uses Solr at its backend, I was wondering if we can get the same
> working
> > at
> > > the Solr level without having to use Watson.
> > >
> > > Regards
> > > Puneet
> > >
> > > On Sat, Jul 9, 2016 at 11:34 AM, Puneet Pawaia <
> [hidden email]
> > <javascript:;>>
> > > wrote:
> > >
> > >> Hi Alessandro
> > >>
> > >> I am looking at being able to answer questions like "Can a non-compete
> > >> clause in an employment agreement be enforced after the expiry of the
> > >> agreement?"
> > >>
> > >> On Sat, Jul 9, 2016 at 4:34 AM, Alessandro Benedetti <
> > >> [hidden email] <javascript:;>> wrote:
> > >>
> > >>> Hi Puneet,
> > >>> your requirement :
> > >>> "I would like users to be able to write queries in natural language
> > >>> rather
> > >>> than keyword based search."
> > >>>
> > >>> Is really really vague :(
> > >>> Can you try to help us with some specific example, starting of course
> > >>> from
> > >>> the simplest use cases you have initially in mind ?
> > >>>
> > >>> Moving from keyword based search to natural language is a really
> > complex
> > >>> task.
> > >>> Proceeding step by step can help you.
> > >>>
> > >>> Do you want for example to set up a Q&A basic system ?
> > >>> In that case you should take care of query rewriting.
> > >>> You need basically to identify your base requirement and then build a
> > >>> specific parser for that.
> > >>> You can use triple stores and knowledge bases to enrich both your
> query
> > >>> and
> > >>> your index, but let's start from the basis, what is your simplest
> > >>> requirement ?
> > >>>
> > >>> On Fri, Jul 8, 2016 at 1:56 PM, Jay Urbain <[hidden email]
> > <javascript:;>> wrote:
> > >>>
> > >>> > I've added multivalued fields within my SOLR schema for indexing
> > >>> entities
> > >>> > extracted using NLP methods applied to the text I'm indexing, along
> > >>> with
> > >>> > fields for other discrete data extracted from relational databases.
> > >>> >
> > >>> > A Java application reads data out of multiple relational databases,
> > >>> uses
> > >>> > NLP on the text and indexes each document (de-normalized) using
> > SOLRJ.
> > >>> >
> > >>> > I initially tried doing this with content handlers, but found it
> much
> > >>> > easier to just write a Java application.
> > >>> >
> > >>> > SOLRJ Java API reference:
> > >>> > https://cwiki.apache.org/confluence/display/solr/Using+SolrJ
> > >>> >
> > >>> > Stanford NLP:
> > >>> > http://stanfordnlp.github.io/CoreNLP/
> > >>> >
> > >>> > Best,
> > >>> > Jay
> > >>> >
> > >>> >
> > >>> > On Thu, Jul 7, 2016 at 9:52 PM, Puneet Pawaia <
> > [hidden email] <javascript:;>
> > >>> >
> > >>> > wrote:
> > >>> >
> > >>> > > Hi Jay
> > >>> > > Any place I can learn more on this method of integration?
> > >>> > > Thanks
> > >>> > > Puneet
> > >>> > >
> > >>> > > On 8 Jul 2016 02:58, "Jay Urbain" <[hidden email]
> > <javascript:;>> wrote:
> > >>> > >
> > >>> > > > I use Stanford NLP and cTakes (based on OpenNLP) while indexing
> > >>> with a
> > >>> > > > SOLRJ application.
> > >>> > > >
> > >>> > > > Best,
> > >>> > > > Jay
> > >>> > > >
> > >>> > > > On Thu, Jul 7, 2016 at 12:09 PM, Puneet Pawaia <
> > >>> > [hidden email] <javascript:;>>
> > >>> > > > wrote:
> > >>> > > >
> > >>> > > > > Hi
> > >>> > > > >
> > >>> > > > > I am currently using Solr 5.5.x to test but can upgrade to
> Solr
> > >>> 6.x
> > >>> > if
> > >>> > > > > required.
> > >>> > > > > I am working on a POC for natural language query using Solr.
> > >>> Should I
> > >>> > > use
> > >>> > > > > the Stanford libraries or are there any other libraries
> having
> > >>> > > > integration
> > >>> > > > > with Solr already available.
> > >>> > > > > Any direction in how to do this would be most appreciated.
> How
> > >>> > should I
> > >>> > > > > process the query to give relevant results.
> > >>> > > > >
> > >>> > > > > Regards
> > >>> > > > > Puneet
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> --------------------------
> > >>>
> > >>> Benedetti Alessandro
> > >>> Visiting card : http://about.me/alessandro_benedetti
> > >>>
> > >>> "Tyger, tyger burning bright
> > >>> In the forests of the night,
> > >>> What immortal hand or eye
> > >>> Could frame thy fearful symmetry?"
> > >>>
> > >>> William Blake - Songs of Experience -1794 England
> > >>>
> > >>
> > >>
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Integrating Stanford NLP or any other NLP for Natural Language Query

Yangrui Guo
Hi my solution uses multi value text fields for storing data objects. It
works best with relational search using natural language. For example,
"car", "automobile", "vehicle" may denote the same class, but they are not
equivalent under certain contexts. Multiple value attributes could help the
search engine better interpret different expression by user referring to
the same concept.

On Sunday, July 10, 2016, Puneet Pawaia <[hidden email]> wrote:

> Hi Yangrui
> We are testing the Rank and Retrieve as well as the NLP interface. However
> this is being done by another team and so I would not be able to comment
> further on it.
> I would like to know what kind of Solr field you are using for storing the
> output from your classes. And also what function output you are putting
> into that field.
> Thanks
> Puneet
>
> On 10 Jul 2016 00:17, "Yangrui Guo" <[hidden email] <javascript:;>>
> wrote:
>
> > Hi Puneet,
> >
> > I only use Watson's text to speech as user interface, because a lot of
> > people think NLP is the same as voice recognition. If you don't need
> voice
> > recognition you could remove Watson from it. Stanford has better
> dependency
> > parsing and can be used offline. However it seems you are using Watson's
> > retrieve and rank API, which is based on Solr, am I correct?
> >
> > Yangrui
> >
> > On Saturday, July 9, 2016, Puneet Pawaia <[hidden email]
> <javascript:;>> wrote:
> >
> > > Hi Yangrui,
> > >
> > > I have been looking at your code for squery.
> > > Unfortunately, I am not very conversant with SolrJ.  I seem to be
> missing
> > > how and what data is added to the Solr index.
> > > Also, I see some references to IBM Watson in your code. Are you using
> IBM
> > > Watson? If yes, then why use the Stanford NLP if you can use the Watson
> > > NLP?
> > >
> > > Regards
> > > Puneet
> > >
> > >
> > > On Sat, Jul 9, 2016 at 11:37 AM, Puneet Pawaia <
> [hidden email] <javascript:;>
> > > <javascript:;>>
> > > wrote:
> > >
> > > > Hi Alessandro
> > > >
> > > > I am looking at being able to answer questions like "Can a
> non-compete
> > > > clause in an employment agreement be enforced after the expiry of the
> > > > agreement?"
> > > > We are doing some testing with IBM Watson and with a sample test
> data,
> > we
> > > > are able to get relevant replies to the above question. Since IBM
> > Watson
> > > > uses Solr at its backend, I was wondering if we can get the same
> > working
> > > at
> > > > the Solr level without having to use Watson.
> > > >
> > > > Regards
> > > > Puneet
> > > >
> > > > On Sat, Jul 9, 2016 at 11:34 AM, Puneet Pawaia <
> > [hidden email] <javascript:;>
> > > <javascript:;>>
> > > > wrote:
> > > >
> > > >> Hi Alessandro
> > > >>
> > > >> I am looking at being able to answer questions like "Can a
> non-compete
> > > >> clause in an employment agreement be enforced after the expiry of
> the
> > > >> agreement?"
> > > >>
> > > >> On Sat, Jul 9, 2016 at 4:34 AM, Alessandro Benedetti <
> > > >> [hidden email] <javascript:;> <javascript:;>> wrote:
> > > >>
> > > >>> Hi Puneet,
> > > >>> your requirement :
> > > >>> "I would like users to be able to write queries in natural language
> > > >>> rather
> > > >>> than keyword based search."
> > > >>>
> > > >>> Is really really vague :(
> > > >>> Can you try to help us with some specific example, starting of
> course
> > > >>> from
> > > >>> the simplest use cases you have initially in mind ?
> > > >>>
> > > >>> Moving from keyword based search to natural language is a really
> > > complex
> > > >>> task.
> > > >>> Proceeding step by step can help you.
> > > >>>
> > > >>> Do you want for example to set up a Q&A basic system ?
> > > >>> In that case you should take care of query rewriting.
> > > >>> You need basically to identify your base requirement and then
> build a
> > > >>> specific parser for that.
> > > >>> You can use triple stores and knowledge bases to enrich both your
> > query
> > > >>> and
> > > >>> your index, but let's start from the basis, what is your simplest
> > > >>> requirement ?
> > > >>>
> > > >>> On Fri, Jul 8, 2016 at 1:56 PM, Jay Urbain <[hidden email]
> <javascript:;>
> > > <javascript:;>> wrote:
> > > >>>
> > > >>> > I've added multivalued fields within my SOLR schema for indexing
> > > >>> entities
> > > >>> > extracted using NLP methods applied to the text I'm indexing,
> along
> > > >>> with
> > > >>> > fields for other discrete data extracted from relational
> databases.
> > > >>> >
> > > >>> > A Java application reads data out of multiple relational
> databases,
> > > >>> uses
> > > >>> > NLP on the text and indexes each document (de-normalized) using
> > > SOLRJ.
> > > >>> >
> > > >>> > I initially tried doing this with content handlers, but found it
> > much
> > > >>> > easier to just write a Java application.
> > > >>> >
> > > >>> > SOLRJ Java API reference:
> > > >>> > https://cwiki.apache.org/confluence/display/solr/Using+SolrJ
> > > >>> >
> > > >>> > Stanford NLP:
> > > >>> > http://stanfordnlp.github.io/CoreNLP/
> > > >>> >
> > > >>> > Best,
> > > >>> > Jay
> > > >>> >
> > > >>> >
> > > >>> > On Thu, Jul 7, 2016 at 9:52 PM, Puneet Pawaia <
> > > [hidden email] <javascript:;> <javascript:;>
> > > >>> >
> > > >>> > wrote:
> > > >>> >
> > > >>> > > Hi Jay
> > > >>> > > Any place I can learn more on this method of integration?
> > > >>> > > Thanks
> > > >>> > > Puneet
> > > >>> > >
> > > >>> > > On 8 Jul 2016 02:58, "Jay Urbain" <[hidden email]
> <javascript:;>
> > > <javascript:;>> wrote:
> > > >>> > >
> > > >>> > > > I use Stanford NLP and cTakes (based on OpenNLP) while
> indexing
> > > >>> with a
> > > >>> > > > SOLRJ application.
> > > >>> > > >
> > > >>> > > > Best,
> > > >>> > > > Jay
> > > >>> > > >
> > > >>> > > > On Thu, Jul 7, 2016 at 12:09 PM, Puneet Pawaia <
> > > >>> > [hidden email] <javascript:;> <javascript:;>>
> > > >>> > > > wrote:
> > > >>> > > >
> > > >>> > > > > Hi
> > > >>> > > > >
> > > >>> > > > > I am currently using Solr 5.5.x to test but can upgrade to
> > Solr
> > > >>> 6.x
> > > >>> > if
> > > >>> > > > > required.
> > > >>> > > > > I am working on a POC for natural language query using
> Solr.
> > > >>> Should I
> > > >>> > > use
> > > >>> > > > > the Stanford libraries or are there any other libraries
> > having
> > > >>> > > > integration
> > > >>> > > > > with Solr already available.
> > > >>> > > > > Any direction in how to do this would be most appreciated.
> > How
> > > >>> > should I
> > > >>> > > > > process the query to give relevant results.
> > > >>> > > > >
> > > >>> > > > > Regards
> > > >>> > > > > Puneet
> > > >>> > > > >
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> --------------------------
> > > >>>
> > > >>> Benedetti Alessandro
> > > >>> Visiting card : http://about.me/alessandro_benedetti
> > > >>>
> > > >>> "Tyger, tyger burning bright
> > > >>> In the forests of the night,
> > > >>> What immortal hand or eye
> > > >>> Could frame thy fearful symmetry?"
> > > >>>
> > > >>> William Blake - Songs of Experience -1794 England
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>