Dealing with null values in streaming rollup

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Dealing with null values in streaming rollup

RAUNAK AGRAWAL
Hi,

I am trying to use streaming rollup expression to aggregate the sales
values over week. Here is the query:

curl http://localhost:8983/solr/metrics_data/stream -d 'expr=rollup(
   search(metrics_data, q=id:123, fl="week_no,sales,qty", qt="/export",
sort="week_no desc"),
  over="week",
   sum(sales),
   sum(qty)
)'

But I am getting exception like:

{
"result-set": {
"docs": [{
"EXCEPTION": null,
"EOF": true,
"RESPONSE_TIME": 169
}]
}
}

The reason being some of the documents are having null as sales. One option
is to wrap the search with select expression
with replace(field,null,withValue=0). Is there any other way for rollup to
ignore those docs which has some fields as null?

Thanks in advance
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with null values in streaming rollup

Jan Høydahl / Cominvent
Have you tried with Solr 7.5? I think it may have been fixed in that version? At least for the timeseries() expression...

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 18. okt. 2018 kl. 05:35 skrev RAUNAK AGRAWAL <[hidden email]>:
>
> Hi,
>
> I am trying to use streaming rollup expression to aggregate the sales
> values over week. Here is the query:
>
> curl http://localhost:8983/solr/metrics_data/stream -d 'expr=rollup(
>   search(metrics_data, q=id:123, fl="week_no,sales,qty", qt="/export",
> sort="week_no desc"),
>  over="week",
>   sum(sales),
>   sum(qty)
> )'
>
> But I am getting exception like:
>
> {
> "result-set": {
> "docs": [{
> "EXCEPTION": null,
> "EOF": true,
> "RESPONSE_TIME": 169
> }]
> }
> }
>
> The reason being some of the documents are having null as sales. One option
> is to wrap the search with select expression
> with replace(field,null,withValue=0). Is there any other way for rollup to
> ignore those docs which has some fields as null?
>
> Thanks in advance

Reply | Threaded
Open this post in threaded view
|

Re: Dealing with null values in streaming rollup

RAUNAK AGRAWAL
Thanks a lot Jan. Will try with 7.5

I am currently using 7.2.1 version. Is there a way to fix it?

On Fri, Oct 19, 2018 at 12:31 AM Jan Høydahl <[hidden email]> wrote:

> Have you tried with Solr 7.5? I think it may have been fixed in that
> version? At least for the timeseries() expression...
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 18. okt. 2018 kl. 05:35 skrev RAUNAK AGRAWAL <[hidden email]>:
> >
> > Hi,
> >
> > I am trying to use streaming rollup expression to aggregate the sales
> > values over week. Here is the query:
> >
> > curl http://localhost:8983/solr/metrics_data/stream -d 'expr=rollup(
> >   search(metrics_data, q=id:123, fl="week_no,sales,qty", qt="/export",
> > sort="week_no desc"),
> >  over="week",
> >   sum(sales),
> >   sum(qty)
> > )'
> >
> > But I am getting exception like:
> >
> > {
> > "result-set": {
> > "docs": [{
> > "EXCEPTION": null,
> > "EOF": true,
> > "RESPONSE_TIME": 169
> > }]
> > }
> > }
> >
> > The reason being some of the documents are having null as sales. One
> option
> > is to wrap the search with select expression
> > with replace(field,null,withValue=0). Is there any other way for rollup
> to
> > ignore those docs which has some fields as null?
> >
> > Thanks in advance
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with null values in streaming rollup

Kojo
I think that you can use stream evaluators in your expressions to filter
the values you want:

https://lucene.apache.org/solr/guide/6_6/stream-evaluators.html





Em seg, 22 de out de 2018 às 12:10, RAUNAK AGRAWAL <[hidden email]>
escreveu:

> Thanks a lot Jan. Will try with 7.5
>
> I am currently using 7.2.1 version. Is there a way to fix it?
>
> On Fri, Oct 19, 2018 at 12:31 AM Jan Høydahl <[hidden email]>
> wrote:
>
> > Have you tried with Solr 7.5? I think it may have been fixed in that
> > version? At least for the timeseries() expression...
> >
> > --
> > Jan Høydahl, search solution architect
> > Cominvent AS - www.cominvent.com
> >
> > > 18. okt. 2018 kl. 05:35 skrev RAUNAK AGRAWAL <[hidden email]
> >:
> > >
> > > Hi,
> > >
> > > I am trying to use streaming rollup expression to aggregate the sales
> > > values over week. Here is the query:
> > >
> > > curl http://localhost:8983/solr/metrics_data/stream -d 'expr=rollup(
> > >   search(metrics_data, q=id:123, fl="week_no,sales,qty", qt="/export",
> > > sort="week_no desc"),
> > >  over="week",
> > >   sum(sales),
> > >   sum(qty)
> > > )'
> > >
> > > But I am getting exception like:
> > >
> > > {
> > > "result-set": {
> > > "docs": [{
> > > "EXCEPTION": null,
> > > "EOF": true,
> > > "RESPONSE_TIME": 169
> > > }]
> > > }
> > > }
> > >
> > > The reason being some of the documents are having null as sales. One
> > option
> > > is to wrap the search with select expression
> > > with replace(field,null,withValue=0). Is there any other way for rollup
> > to
> > > ignore those docs which has some fields as null?
> > >
> > > Thanks in advance
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with null values in streaming rollup

Joel Bernstein
This sounds like a bug, please log a ticket.

I think the workaround you suggest is the only way to solve this problem
currently.

Joel Bernstein
http://joelsolr.blogspot.com/


On Mon, Oct 22, 2018 at 11:28 AM Kojo <[hidden email]> wrote:

> I think that you can use stream evaluators in your expressions to filter
> the values you want:
>
> https://lucene.apache.org/solr/guide/6_6/stream-evaluators.html
>
>
>
>
>
> Em seg, 22 de out de 2018 às 12:10, RAUNAK AGRAWAL <
> [hidden email]>
> escreveu:
>
> > Thanks a lot Jan. Will try with 7.5
> >
> > I am currently using 7.2.1 version. Is there a way to fix it?
> >
> > On Fri, Oct 19, 2018 at 12:31 AM Jan Høydahl <[hidden email]>
> > wrote:
> >
> > > Have you tried with Solr 7.5? I think it may have been fixed in that
> > > version? At least for the timeseries() expression...
> > >
> > > --
> > > Jan Høydahl, search solution architect
> > > Cominvent AS - www.cominvent.com
> > >
> > > > 18. okt. 2018 kl. 05:35 skrev RAUNAK AGRAWAL <
> [hidden email]
> > >:
> > > >
> > > > Hi,
> > > >
> > > > I am trying to use streaming rollup expression to aggregate the sales
> > > > values over week. Here is the query:
> > > >
> > > > curl http://localhost:8983/solr/metrics_data/stream -d 'expr=rollup(
> > > >   search(metrics_data, q=id:123, fl="week_no,sales,qty",
> qt="/export",
> > > > sort="week_no desc"),
> > > >  over="week",
> > > >   sum(sales),
> > > >   sum(qty)
> > > > )'
> > > >
> > > > But I am getting exception like:
> > > >
> > > > {
> > > > "result-set": {
> > > > "docs": [{
> > > > "EXCEPTION": null,
> > > > "EOF": true,
> > > > "RESPONSE_TIME": 169
> > > > }]
> > > > }
> > > > }
> > > >
> > > > The reason being some of the documents are having null as sales. One
> > > option
> > > > is to wrap the search with select expression
> > > > with replace(field,null,withValue=0). Is there any other way for
> rollup
> > > to
> > > > ignore those docs which has some fields as null?
> > > >
> > > > Thanks in advance
> > >
> > >
> >
>