Re: Solr 8.3 Solrj streaming expressions do not return all field values

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.3 Solrj streaming expressions do not return all field values

Jörn Franke

On Wed, Nov 6, 2019 at 10:45 AM Jörn Franke <[hidden email]> wrote:
I have checked now Solr 8.3 server in admin UI. Same issue.

Reproduction:
select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”, qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found)

In 8.3 it returns only the id field.
In 8.2 it returns id,found field.

Since found is generated by select (and not coming from the collection) there must be an issue with select.

Any idea why this is happening.

Debug logs do not show any error and the expression is correctly received by Solr.

Thank you.

Best regards

> Am 05.11.2019 um 14:59 schrieb Jörn Franke <[hidden email]>:
>
> Thanks I will check and come back to you. As far as I remember (but have to check) the queries generated by Solr were correct
>
> Just to be clear the same thing works with Solr 8.2 server and Solr 8.2 client.
>
> It show the odd behaviour with Solr 8.2 server and Solr 8.3 client.
>
>> Am 05.11.2019 um 14:49 schrieb Joel Bernstein <[hidden email]>:
>>
>> I'll probably need some more details. One thing that's useful is to look at
>> the logs and see the underlying Solr queries that are generated. Then try
>> those underlying queries against the Solr index and see what comes back. If
>> you're not seeing the fields with the plain Solr queries then we know it's
>> something going on below streaming expressions. If you are seeing the
>> fields then it's the expressions themselves that are not handling the data
>> as expected.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>>>> On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke <[hidden email]> wrote:
>>>
>>> Most likely this issue can bei also reproduced in the admin UI for the
>>> streaming handler of a collection.
>>>
>>>>> Am 04.11.2019 um 13:32 schrieb Jörn Franke <[hidden email]>:
>>>>
>>>> Hi,
>>>>
>>>> I use streaming expressions, e.g.
>>>> Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
>>>> (Using export handler, sort is not really mandatory , I will remove it
>>> later anyway)
>>>>
>>>> This works perfectly fine if I use Solr 8.2.0 (server + client). It
>>> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
>>>>
>>>> However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the
>>> above statement only returns the id field, but not the found field.
>>>>
>>>> Questions:
>>>> 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
>>> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
>>>> 2) has the syntax for the above expression changed? If so how?
>>>> 3) is this not expected behavior and I should create a Jira for it?
>>>>
>>>> Thank you.
>>>> Best regards
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.3 Solrj streaming expressions do not return all field values

Joel Bernstein
Before moving to a jira let's take a look at the underlying Solr queries in the log. The Streaming Expressions just creates solr queries, in this case queries to the /export handler. So when something is not working as expected we want to strip away the streaming expression and debug the actual queries that are being run.

You can find the Solr queries that appear in the log after running the expressions and then try running them outside of the expression as plain Solr queries.

You can also post the Solr queries to this thread and we discuss what the logs say.

In these cases the logs always are the way to debug whats going on.


On Wed, Nov 6, 2019 at 4:14 PM Jörn Franke <[hidden email]> wrote:

On Wed, Nov 6, 2019 at 10:45 AM Jörn Franke <[hidden email]> wrote:
I have checked now Solr 8.3 server in admin UI. Same issue.

Reproduction:
select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”, qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found)

In 8.3 it returns only the id field.
In 8.2 it returns id,found field.

Since found is generated by select (and not coming from the collection) there must be an issue with select.

Any idea why this is happening.

Debug logs do not show any error and the expression is correctly received by Solr.

Thank you.

Best regards

> Am 05.11.2019 um 14:59 schrieb Jörn Franke <[hidden email]>:
>
> Thanks I will check and come back to you. As far as I remember (but have to check) the queries generated by Solr were correct
>
> Just to be clear the same thing works with Solr 8.2 server and Solr 8.2 client.
>
> It show the odd behaviour with Solr 8.2 server and Solr 8.3 client.
>
>> Am 05.11.2019 um 14:49 schrieb Joel Bernstein <[hidden email]>:
>>
>> I'll probably need some more details. One thing that's useful is to look at
>> the logs and see the underlying Solr queries that are generated. Then try
>> those underlying queries against the Solr index and see what comes back. If
>> you're not seeing the fields with the plain Solr queries then we know it's
>> something going on below streaming expressions. If you are seeing the
>> fields then it's the expressions themselves that are not handling the data
>> as expected.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>>>> On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke <[hidden email]> wrote:
>>>
>>> Most likely this issue can bei also reproduced in the admin UI for the
>>> streaming handler of a collection.
>>>
>>>>> Am 04.11.2019 um 13:32 schrieb Jörn Franke <[hidden email]>:
>>>>
>>>> Hi,
>>>>
>>>> I use streaming expressions, e.g.
>>>> Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
>>>> (Using export handler, sort is not really mandatory , I will remove it
>>> later anyway)
>>>>
>>>> This works perfectly fine if I use Solr 8.2.0 (server + client). It
>>> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
>>>>
>>>> However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the
>>> above statement only returns the id field, but not the found field.
>>>>
>>>> Questions:
>>>> 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
>>> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
>>>> 2) has the syntax for the above expression changed? If so how?
>>>> 3) is this not expected behavior and I should create a Jira for it?
>>>>
>>>> Thank you.
>>>> Best regards
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.3 Solrj streaming expressions do not return all field values

Jörn Franke
thanks i will find another installation of Solr, but last time the underlying queries to the export handler were correct. What was wrong was the generated field "found" based on the if statement based NOT on any data in the collectioj.  As said, in Solr 8.3 the id fields are returned, but not the "found" field, which is generated by the selected statement:
select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found)

returns a tuple 
id: "blabla"

but not the part generated by the if(eq(1,1),Y,N) as found. In Solr 8.2 the found field is returned:
id: "blabla"
found: "Y"
To my best knowledge this is a fully generated field => it does not depend on the underlying collection data and the export handler. This was done to exclude any issues with this one.
it states simply if 1=1 return Y else N


On Fri, Nov 8, 2019 at 3:20 PM Joel Bernstein <[hidden email]> wrote:
Before moving to a jira let's take a look at the underlying Solr queries in the log. The Streaming Expressions just creates solr queries, in this case queries to the /export handler. So when something is not working as expected we want to strip away the streaming expression and debug the actual queries that are being run.

You can find the Solr queries that appear in the log after running the expressions and then try running them outside of the expression as plain Solr queries.

You can also post the Solr queries to this thread and we discuss what the logs say.

In these cases the logs always are the way to debug whats going on.


On Wed, Nov 6, 2019 at 4:14 PM Jörn Franke <[hidden email]> wrote:

On Wed, Nov 6, 2019 at 10:45 AM Jörn Franke <[hidden email]> wrote:
I have checked now Solr 8.3 server in admin UI. Same issue.

Reproduction:
select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”, qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found)

In 8.3 it returns only the id field.
In 8.2 it returns id,found field.

Since found is generated by select (and not coming from the collection) there must be an issue with select.

Any idea why this is happening.

Debug logs do not show any error and the expression is correctly received by Solr.

Thank you.

Best regards

> Am 05.11.2019 um 14:59 schrieb Jörn Franke <[hidden email]>:
>
> Thanks I will check and come back to you. As far as I remember (but have to check) the queries generated by Solr were correct
>
> Just to be clear the same thing works with Solr 8.2 server and Solr 8.2 client.
>
> It show the odd behaviour with Solr 8.2 server and Solr 8.3 client.
>
>> Am 05.11.2019 um 14:49 schrieb Joel Bernstein <[hidden email]>:
>>
>> I'll probably need some more details. One thing that's useful is to look at
>> the logs and see the underlying Solr queries that are generated. Then try
>> those underlying queries against the Solr index and see what comes back. If
>> you're not seeing the fields with the plain Solr queries then we know it's
>> something going on below streaming expressions. If you are seeing the
>> fields then it's the expressions themselves that are not handling the data
>> as expected.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>>>> On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke <[hidden email]> wrote:
>>>
>>> Most likely this issue can bei also reproduced in the admin UI for the
>>> streaming handler of a collection.
>>>
>>>>> Am 04.11.2019 um 13:32 schrieb Jörn Franke <[hidden email]>:
>>>>
>>>> Hi,
>>>>
>>>> I use streaming expressions, e.g.
>>>> Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
>>>> (Using export handler, sort is not really mandatory , I will remove it
>>> later anyway)
>>>>
>>>> This works perfectly fine if I use Solr 8.2.0 (server + client). It
>>> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
>>>>
>>>> However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the
>>> above statement only returns the id field, but not the found field.
>>>>
>>>> Questions:
>>>> 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
>>> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
>>>> 2) has the syntax for the above expression changed? If so how?
>>>> 3) is this not expected behavior and I should create a Jira for it?
>>>>
>>>> Thank you.
>>>> Best regards
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.3 Solrj streaming expressions do not return all field values

Jörn Franke
Could it be that Solr 8.3 is more strict on the if statement?
the statement if(eq(1,1),Y,N)
is supposed to return the character Y (not the field). In Solr 8.2 it returns the character Y, but in Solr 8.3 not. 

On Tue, Nov 19, 2019 at 11:21 PM Jörn Franke <[hidden email]> wrote:
thanks i will find another installation of Solr, but last time the underlying queries to the export handler were correct. What was wrong was the generated field "found" based on the if statement based NOT on any data in the collectioj.  As said, in Solr 8.3 the id fields are returned, but not the "found" field, which is generated by the selected statement:
select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found)

returns a tuple 
id: "blabla"

but not the part generated by the if(eq(1,1),Y,N) as found. In Solr 8.2 the found field is returned:
id: "blabla"
found: "Y"
To my best knowledge this is a fully generated field => it does not depend on the underlying collection data and the export handler. This was done to exclude any issues with this one.
it states simply if 1=1 return Y else N


On Fri, Nov 8, 2019 at 3:20 PM Joel Bernstein <[hidden email]> wrote:
Before moving to a jira let's take a look at the underlying Solr queries in the log. The Streaming Expressions just creates solr queries, in this case queries to the /export handler. So when something is not working as expected we want to strip away the streaming expression and debug the actual queries that are being run.

You can find the Solr queries that appear in the log after running the expressions and then try running them outside of the expression as plain Solr queries.

You can also post the Solr queries to this thread and we discuss what the logs say.

In these cases the logs always are the way to debug whats going on.


On Wed, Nov 6, 2019 at 4:14 PM Jörn Franke <[hidden email]> wrote:

On Wed, Nov 6, 2019 at 10:45 AM Jörn Franke <[hidden email]> wrote:
I have checked now Solr 8.3 server in admin UI. Same issue.

Reproduction:
select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”, qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found)

In 8.3 it returns only the id field.
In 8.2 it returns id,found field.

Since found is generated by select (and not coming from the collection) there must be an issue with select.

Any idea why this is happening.

Debug logs do not show any error and the expression is correctly received by Solr.

Thank you.

Best regards

> Am 05.11.2019 um 14:59 schrieb Jörn Franke <[hidden email]>:
>
> Thanks I will check and come back to you. As far as I remember (but have to check) the queries generated by Solr were correct
>
> Just to be clear the same thing works with Solr 8.2 server and Solr 8.2 client.
>
> It show the odd behaviour with Solr 8.2 server and Solr 8.3 client.
>
>> Am 05.11.2019 um 14:49 schrieb Joel Bernstein <[hidden email]>:
>>
>> I'll probably need some more details. One thing that's useful is to look at
>> the logs and see the underlying Solr queries that are generated. Then try
>> those underlying queries against the Solr index and see what comes back. If
>> you're not seeing the fields with the plain Solr queries then we know it's
>> something going on below streaming expressions. If you are seeing the
>> fields then it's the expressions themselves that are not handling the data
>> as expected.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>>>> On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke <[hidden email]> wrote:
>>>
>>> Most likely this issue can bei also reproduced in the admin UI for the
>>> streaming handler of a collection.
>>>
>>>>> Am 04.11.2019 um 13:32 schrieb Jörn Franke <[hidden email]>:
>>>>
>>>> Hi,
>>>>
>>>> I use streaming expressions, e.g.
>>>> Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
>>>> (Using export handler, sort is not really mandatory , I will remove it
>>> later anyway)
>>>>
>>>> This works perfectly fine if I use Solr 8.2.0 (server + client). It
>>> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
>>>>
>>>> However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the
>>> above statement only returns the id field, but not the found field.
>>>>
>>>> Questions:
>>>> 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
>>> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
>>>> 2) has the syntax for the above expression changed? If so how?
>>>> 3) is this not expected behavior and I should create a Jira for it?
>>>>
>>>> Thank you.
>>>> Best regards
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.3 Solrj streaming expressions do not return all field values

Jörn Franke
Ok, solved.  Solr 8.2 accepted this statement:
select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found)

and return to me the expected results. Note that around Y and N there is no ". Solr 8.3 requires the following statement
select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),"Y","N") as found)
to return the exact results. I do not know why I did originally not include the quotation marks around Y and N, but it seems that 8.2 accepted this and 8.3 not. I will update the Jira.



On Tue, Nov 19, 2019 at 11:25 PM Jörn Franke <[hidden email]> wrote:
Could it be that Solr 8.3 is more strict on the if statement?
the statement if(eq(1,1),Y,N)
is supposed to return the character Y (not the field). In Solr 8.2 it returns the character Y, but in Solr 8.3 not. 

On Tue, Nov 19, 2019 at 11:21 PM Jörn Franke <[hidden email]> wrote:
thanks i will find another installation of Solr, but last time the underlying queries to the export handler were correct. What was wrong was the generated field "found" based on the if statement based NOT on any data in the collectioj.  As said, in Solr 8.3 the id fields are returned, but not the "found" field, which is generated by the selected statement:
select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found)

returns a tuple 
id: "blabla"

but not the part generated by the if(eq(1,1),Y,N) as found. In Solr 8.2 the found field is returned:
id: "blabla"
found: "Y"
To my best knowledge this is a fully generated field => it does not depend on the underlying collection data and the export handler. This was done to exclude any issues with this one.
it states simply if 1=1 return Y else N


On Fri, Nov 8, 2019 at 3:20 PM Joel Bernstein <[hidden email]> wrote:
Before moving to a jira let's take a look at the underlying Solr queries in the log. The Streaming Expressions just creates solr queries, in this case queries to the /export handler. So when something is not working as expected we want to strip away the streaming expression and debug the actual queries that are being run.

You can find the Solr queries that appear in the log after running the expressions and then try running them outside of the expression as plain Solr queries.

You can also post the Solr queries to this thread and we discuss what the logs say.

In these cases the logs always are the way to debug whats going on.


On Wed, Nov 6, 2019 at 4:14 PM Jörn Franke <[hidden email]> wrote:

On Wed, Nov 6, 2019 at 10:45 AM Jörn Franke <[hidden email]> wrote:
I have checked now Solr 8.3 server in admin UI. Same issue.

Reproduction:
select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”, qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found)

In 8.3 it returns only the id field.
In 8.2 it returns id,found field.

Since found is generated by select (and not coming from the collection) there must be an issue with select.

Any idea why this is happening.

Debug logs do not show any error and the expression is correctly received by Solr.

Thank you.

Best regards

> Am 05.11.2019 um 14:59 schrieb Jörn Franke <[hidden email]>:
>
> Thanks I will check and come back to you. As far as I remember (but have to check) the queries generated by Solr were correct
>
> Just to be clear the same thing works with Solr 8.2 server and Solr 8.2 client.
>
> It show the odd behaviour with Solr 8.2 server and Solr 8.3 client.
>
>> Am 05.11.2019 um 14:49 schrieb Joel Bernstein <[hidden email]>:
>>
>> I'll probably need some more details. One thing that's useful is to look at
>> the logs and see the underlying Solr queries that are generated. Then try
>> those underlying queries against the Solr index and see what comes back. If
>> you're not seeing the fields with the plain Solr queries then we know it's
>> something going on below streaming expressions. If you are seeing the
>> fields then it's the expressions themselves that are not handling the data
>> as expected.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>>>> On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke <[hidden email]> wrote:
>>>
>>> Most likely this issue can bei also reproduced in the admin UI for the
>>> streaming handler of a collection.
>>>
>>>>> Am 04.11.2019 um 13:32 schrieb Jörn Franke <[hidden email]>:
>>>>
>>>> Hi,
>>>>
>>>> I use streaming expressions, e.g.
>>>> Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
>>>> (Using export handler, sort is not really mandatory , I will remove it
>>> later anyway)
>>>>
>>>> This works perfectly fine if I use Solr 8.2.0 (server + client). It
>>> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
>>>>
>>>> However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the
>>> above statement only returns the id field, but not the found field.
>>>>
>>>> Questions:
>>>> 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
>>> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
>>>> 2) has the syntax for the above expression changed? If so how?
>>>> 3) is this not expected behavior and I should create a Jira for it?
>>>>
>>>> Thank you.
>>>> Best regards
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.3 Solrj streaming expressions do not return all field values

Joel Bernstein
Ah, I should have noticed that. Yes, there was a change in how string literals are handled. Quotes are now required for string literals in Math Expressions. I suspect this change is going to cause problems for other people as well, but it needed to be done to clarify aspects of the language. Sorry for the confusion. I'll make a point of updating the documentation to make sure all examples in the documentation are correct.




On Tue, Nov 19, 2019 at 5:56 PM Jörn Franke <[hidden email]> wrote:
Ok, solved.  Solr 8.2 accepted this statement:
select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found)

and return to me the expected results. Note that around Y and N there is no ". Solr 8.3 requires the following statement
select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),"Y","N") as found)
to return the exact results. I do not know why I did originally not include the quotation marks around Y and N, but it seems that 8.2 accepted this and 8.3 not. I will update the Jira.



On Tue, Nov 19, 2019 at 11:25 PM Jörn Franke <[hidden email]> wrote:
Could it be that Solr 8.3 is more strict on the if statement?
the statement if(eq(1,1),Y,N)
is supposed to return the character Y (not the field). In Solr 8.2 it returns the character Y, but in Solr 8.3 not. 

On Tue, Nov 19, 2019 at 11:21 PM Jörn Franke <[hidden email]> wrote:
thanks i will find another installation of Solr, but last time the underlying queries to the export handler were correct. What was wrong was the generated field "found" based on the if statement based NOT on any data in the collectioj.  As said, in Solr 8.3 the id fields are returned, but not the "found" field, which is generated by the selected statement:
select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found)

returns a tuple 
id: "blabla"

but not the part generated by the if(eq(1,1),Y,N) as found. In Solr 8.2 the found field is returned:
id: "blabla"
found: "Y"
To my best knowledge this is a fully generated field => it does not depend on the underlying collection data and the export handler. This was done to exclude any issues with this one.
it states simply if 1=1 return Y else N


On Fri, Nov 8, 2019 at 3:20 PM Joel Bernstein <[hidden email]> wrote:
Before moving to a jira let's take a look at the underlying Solr queries in the log. The Streaming Expressions just creates solr queries, in this case queries to the /export handler. So when something is not working as expected we want to strip away the streaming expression and debug the actual queries that are being run.

You can find the Solr queries that appear in the log after running the expressions and then try running them outside of the expression as plain Solr queries.

You can also post the Solr queries to this thread and we discuss what the logs say.

In these cases the logs always are the way to debug whats going on.


On Wed, Nov 6, 2019 at 4:14 PM Jörn Franke <[hidden email]> wrote:

On Wed, Nov 6, 2019 at 10:45 AM Jörn Franke <[hidden email]> wrote:
I have checked now Solr 8.3 server in admin UI. Same issue.

Reproduction:
select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”, qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found)

In 8.3 it returns only the id field.
In 8.2 it returns id,found field.

Since found is generated by select (and not coming from the collection) there must be an issue with select.

Any idea why this is happening.

Debug logs do not show any error and the expression is correctly received by Solr.

Thank you.

Best regards

> Am 05.11.2019 um 14:59 schrieb Jörn Franke <[hidden email]>:
>
> Thanks I will check and come back to you. As far as I remember (but have to check) the queries generated by Solr were correct
>
> Just to be clear the same thing works with Solr 8.2 server and Solr 8.2 client.
>
> It show the odd behaviour with Solr 8.2 server and Solr 8.3 client.
>
>> Am 05.11.2019 um 14:49 schrieb Joel Bernstein <[hidden email]>:
>>
>> I'll probably need some more details. One thing that's useful is to look at
>> the logs and see the underlying Solr queries that are generated. Then try
>> those underlying queries against the Solr index and see what comes back. If
>> you're not seeing the fields with the plain Solr queries then we know it's
>> something going on below streaming expressions. If you are seeing the
>> fields then it's the expressions themselves that are not handling the data
>> as expected.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>>>> On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke <[hidden email]> wrote:
>>>
>>> Most likely this issue can bei also reproduced in the admin UI for the
>>> streaming handler of a collection.
>>>
>>>>> Am 04.11.2019 um 13:32 schrieb Jörn Franke <[hidden email]>:
>>>>
>>>> Hi,
>>>>
>>>> I use streaming expressions, e.g.
>>>> Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
>>>> (Using export handler, sort is not really mandatory , I will remove it
>>> later anyway)
>>>>
>>>> This works perfectly fine if I use Solr 8.2.0 (server + client). It
>>> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
>>>>
>>>> However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the
>>> above statement only returns the id field, but not the found field.
>>>>
>>>> Questions:
>>>> 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
>>> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
>>>> 2) has the syntax for the above expression changed? If so how?
>>>> 3) is this not expected behavior and I should create a Jira for it?
>>>>
>>>> Thank you.
>>>> Best regards
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.3 Solrj streaming expressions do not return all field values

Jörn Franke
No problem, but thanks a lot for the feedback and considerations. I will try to get into the process on how to update the Solr ref guide. I found a lot of things that I could solve to facilitate others to do a solution based on Solr.

Am 20.11.2019 um 13:04 schrieb Joel Bernstein <[hidden email]>:


Ah, I should have noticed that. Yes, there was a change in how string literals are handled. Quotes are now required for string literals in Math Expressions. I suspect this change is going to cause problems for other people as well, but it needed to be done to clarify aspects of the language. Sorry for the confusion. I'll make a point of updating the documentation to make sure all examples in the documentation are correct.




On Tue, Nov 19, 2019 at 5:56 PM Jörn Franke <[hidden email]> wrote:
Ok, solved.  Solr 8.2 accepted this statement:
select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found)

and return to me the expected results. Note that around Y and N there is no ". Solr 8.3 requires the following statement
select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),"Y","N") as found)
to return the exact results. I do not know why I did originally not include the quotation marks around Y and N, but it seems that 8.2 accepted this and 8.3 not. I will update the Jira.



On Tue, Nov 19, 2019 at 11:25 PM Jörn Franke <[hidden email]> wrote:
Could it be that Solr 8.3 is more strict on the if statement?
the statement if(eq(1,1),Y,N)
is supposed to return the character Y (not the field). In Solr 8.2 it returns the character Y, but in Solr 8.3 not. 

On Tue, Nov 19, 2019 at 11:21 PM Jörn Franke <[hidden email]> wrote:
thanks i will find another installation of Solr, but last time the underlying queries to the export handler were correct. What was wrong was the generated field "found" based on the if statement based NOT on any data in the collectioj.  As said, in Solr 8.3 the id fields are returned, but not the "found" field, which is generated by the selected statement:
select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found)

returns a tuple 
id: "blabla"

but not the part generated by the if(eq(1,1),Y,N) as found. In Solr 8.2 the found field is returned:
id: "blabla"
found: "Y"
To my best knowledge this is a fully generated field => it does not depend on the underlying collection data and the export handler. This was done to exclude any issues with this one.
it states simply if 1=1 return Y else N


On Fri, Nov 8, 2019 at 3:20 PM Joel Bernstein <[hidden email]> wrote:
Before moving to a jira let's take a look at the underlying Solr queries in the log. The Streaming Expressions just creates solr queries, in this case queries to the /export handler. So when something is not working as expected we want to strip away the streaming expression and debug the actual queries that are being run.

You can find the Solr queries that appear in the log after running the expressions and then try running them outside of the expression as plain Solr queries.

You can also post the Solr queries to this thread and we discuss what the logs say.

In these cases the logs always are the way to debug whats going on.


On Wed, Nov 6, 2019 at 4:14 PM Jörn Franke <[hidden email]> wrote:

On Wed, Nov 6, 2019 at 10:45 AM Jörn Franke <[hidden email]> wrote:
I have checked now Solr 8.3 server in admin UI. Same issue.

Reproduction:
select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”, qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found)

In 8.3 it returns only the id field.
In 8.2 it returns id,found field.

Since found is generated by select (and not coming from the collection) there must be an issue with select.

Any idea why this is happening.

Debug logs do not show any error and the expression is correctly received by Solr.

Thank you.

Best regards

> Am 05.11.2019 um 14:59 schrieb Jörn Franke <[hidden email]>:
>
> Thanks I will check and come back to you. As far as I remember (but have to check) the queries generated by Solr were correct
>
> Just to be clear the same thing works with Solr 8.2 server and Solr 8.2 client.
>
> It show the odd behaviour with Solr 8.2 server and Solr 8.3 client.
>
>> Am 05.11.2019 um 14:49 schrieb Joel Bernstein <[hidden email]>:
>>
>> I'll probably need some more details. One thing that's useful is to look at
>> the logs and see the underlying Solr queries that are generated. Then try
>> those underlying queries against the Solr index and see what comes back. If
>> you're not seeing the fields with the plain Solr queries then we know it's
>> something going on below streaming expressions. If you are seeing the
>> fields then it's the expressions themselves that are not handling the data
>> as expected.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>>>> On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke <[hidden email]> wrote:
>>>
>>> Most likely this issue can bei also reproduced in the admin UI for the
>>> streaming handler of a collection.
>>>
>>>>> Am 04.11.2019 um 13:32 schrieb Jörn Franke <[hidden email]>:
>>>>
>>>> Hi,
>>>>
>>>> I use streaming expressions, e.g.
>>>> Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A asc”)
>>>> (Using export handler, sort is not really mandatory , I will remove it
>>> later anyway)
>>>>
>>>> This works perfectly fine if I use Solr 8.2.0 (server + client). It
>>> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
>>>>
>>>> However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then the
>>> above statement only returns the id field, but not the found field.
>>>>
>>>> Questions:
>>>> 1) is this expected behavior, ie Solr client 8.3.0 is in this case not
>>> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix this?
>>>> 2) has the syntax for the above expression changed? If so how?
>>>> 3) is this not expected behavior and I should create a Jira for it?
>>>>
>>>> Thank you.
>>>> Best regards
>>>