Solr Stream vs Export Request Handlers

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr Stream vs Export Request Handlers

kamal11
Hi,

I have a requirement to fetch all data from a collection. One way is to use
streaming expression and other way is to use export.

Streaming expression documentation says *streaming functions are designed
to work with entire result sets rather then the top N results like normal
search. This is supported by the /export handler.*

Also, Export handler documentation says *this feature uses a stream sorting
technique that begins to send records within milliseconds and continues to
stream results until the entire result set has been sorted and exported.*

These two statements concludes to me that for fetching entire results
streaming expressions uses export handler and export handler uses stream,
so, whether I use streaming expression or export handler, they are
internally same and would have same performance. I am correct over here to
say so ?


Ref Links:

https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html


Regards
Kamal Kishore
Reply | Threaded
Open this post in threaded view
|

Re: Solr Stream vs Export Request Handlers

Jan Høydahl / Cominvent
Hi,

Yes, you can choose which to use, it should give  you about same result. If you already work with the Solr search API it would be the easiest for you to consume /export as you don't need to learn the new syntax and parse the Tuple response. However, if you need to do stuff with the docs as you stream them from Solr, then streaming expressions lets you enrich the docs, modify, join etc on the fly.

PS: When the /export docs says it uses a streaming tecnique, it does NOT mean that it has uses the solr feature streaming expressions :)

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <[hidden email]>:
>
> Hi,
>
> I have a requirement to fetch all data from a collection. One way is to use
> streaming expression and other way is to use export.
>
> Streaming expression documentation says *streaming functions are designed
> to work with entire result sets rather then the top N results like normal
> search. This is supported by the /export handler.*
>
> Also, Export handler documentation says *this feature uses a stream sorting
> technique that begins to send records within milliseconds and continues to
> stream results until the entire result set has been sorted and exported.*
>
> These two statements concludes to me that for fetching entire results
> streaming expressions uses export handler and export handler uses stream,
> so, whether I use streaming expression or export handler, they are
> internally same and would have same performance. I am correct over here to
> say so ?
>
>
> Ref Links:
>
> https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
>
>
> Regards
> Kamal Kishore

Reply | Threaded
Open this post in threaded view
|

Re: Solr Stream vs Export Request Handlers

Joel Bernstein
The export handler does not do distributed search. So if you have a
multi-shard collection you may have to use Streaming Expressions to get
exports from all shards.


Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <[hidden email]> wrote:

> Hi,
>
> Yes, you can choose which to use, it should give  you about same result.
> If you already work with the Solr search API it would be the easiest for
> you to consume /export as you don't need to learn the new syntax and parse
> the Tuple response. However, if you need to do stuff with the docs as you
> stream them from Solr, then streaming expressions lets you enrich the docs,
> modify, join etc on the fly.
>
> PS: When the /export docs says it uses a streaming tecnique, it does NOT
> mean that it has uses the solr feature streaming expressions :)
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> [hidden email]>:
> >
> > Hi,
> >
> > I have a requirement to fetch all data from a collection. One way is to
> use
> > streaming expression and other way is to use export.
> >
> > Streaming expression documentation says *streaming functions are designed
> > to work with entire result sets rather then the top N results like normal
> > search. This is supported by the /export handler.*
> >
> > Also, Export handler documentation says *this feature uses a stream
> sorting
> > technique that begins to send records within milliseconds and continues
> to
> > stream results until the entire result set has been sorted and exported.*
> >
> > These two statements concludes to me that for fetching entire results
> > streaming expressions uses export handler and export handler uses stream,
> > so, whether I use streaming expression or export handler, they are
> > internally same and would have same performance. I am correct over here
> to
> > say so ?
> >
> >
> > Ref Links:
> >
> > https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> > https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> >
> >
> > Regards
> > Kamal Kishore
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr Stream vs Export Request Handlers

kamal11
Hi,

Thanks Jan & Joel.

Though I will evaluate the performance over my data, but based on your
experience, which one of the two is better in performance ?. Please suggest

Yeah, I know export does not get the data from all shards, but we can write
code to aggregate the data from all shards. But only if export is better
over stream.

Thanks
Kamal Kishore


On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <[hidden email]> wrote:

> The export handler does not do distributed search. So if you have a
> multi-shard collection you may have to use Streaming Expressions to get
> exports from all shards.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <[hidden email]> wrote:
>
> > Hi,
> >
> > Yes, you can choose which to use, it should give  you about same result.
> > If you already work with the Solr search API it would be the easiest for
> > you to consume /export as you don't need to learn the new syntax and
> parse
> > the Tuple response. However, if you need to do stuff with the docs as you
> > stream them from Solr, then streaming expressions lets you enrich the
> docs,
> > modify, join etc on the fly.
> >
> > PS: When the /export docs says it uses a streaming tecnique, it does NOT
> > mean that it has uses the solr feature streaming expressions :)
> >
> > --
> > Jan Høydahl, search solution architect
> > Cominvent AS - www.cominvent.com
> >
> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> > [hidden email]>:
> > >
> > > Hi,
> > >
> > > I have a requirement to fetch all data from a collection. One way is to
> > use
> > > streaming expression and other way is to use export.
> > >
> > > Streaming expression documentation says *streaming functions are
> designed
> > > to work with entire result sets rather then the top N results like
> normal
> > > search. This is supported by the /export handler.*
> > >
> > > Also, Export handler documentation says *this feature uses a stream
> > sorting
> > > technique that begins to send records within milliseconds and continues
> > to
> > > stream results until the entire result set has been sorted and
> exported.*
> > >
> > > These two statements concludes to me that for fetching entire results
> > > streaming expressions uses export handler and export handler uses
> stream,
> > > so, whether I use streaming expression or export handler, they are
> > > internally same and would have same performance. I am correct over here
> > to
> > > say so ?
> > >
> > >
> > > Ref Links:
> > >
> > > https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> > > https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> > >
> > >
> > > Regards
> > > Kamal Kishore
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr Stream vs Export Request Handlers

kamal11
Hi,

After I performed the test on my data, I found out that direct /export and
streaming expression with export, both are giving almost same response
time. This was also pointed out by *Jan Høydahl* in his reply.

Also, the documentation says export feature uses stream sorting technique
and streaming expression also uses steam technique. So, are they internally
works in same fashion. Please confirm.

Regards
Kamal Kishore



On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <[hidden email]>
wrote:

> Hi,
>
> Thanks Jan & Joel.
>
> Though I will evaluate the performance over my data, but based on your
> experience, which one of the two is better in performance ?. Please suggest
>
> Yeah, I know export does not get the data from all shards, but we can
> write code to aggregate the data from all shards. But only if export is
> better over stream.
>
> Thanks
> Kamal Kishore
>
>
> On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <[hidden email]>
> wrote:
>
>> The export handler does not do distributed search. So if you have a
>> multi-shard collection you may have to use Streaming Expressions to get
>> exports from all shards.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <[hidden email]>
>> wrote:
>>
>> > Hi,
>> >
>> > Yes, you can choose which to use, it should give  you about same result.
>> > If you already work with the Solr search API it would be the easiest for
>> > you to consume /export as you don't need to learn the new syntax and
>> parse
>> > the Tuple response. However, if you need to do stuff with the docs as
>> you
>> > stream them from Solr, then streaming expressions lets you enrich the
>> docs,
>> > modify, join etc on the fly.
>> >
>> > PS: When the /export docs says it uses a streaming tecnique, it does NOT
>> > mean that it has uses the solr feature streaming expressions :)
>> >
>> > --
>> > Jan Høydahl, search solution architect
>> > Cominvent AS - www.cominvent.com
>> >
>> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
>> > [hidden email]>:
>> > >
>> > > Hi,
>> > >
>> > > I have a requirement to fetch all data from a collection. One way is
>> to
>> > use
>> > > streaming expression and other way is to use export.
>> > >
>> > > Streaming expression documentation says *streaming functions are
>> designed
>> > > to work with entire result sets rather then the top N results like
>> normal
>> > > search. This is supported by the /export handler.*
>> > >
>> > > Also, Export handler documentation says *this feature uses a stream
>> > sorting
>> > > technique that begins to send records within milliseconds and
>> continues
>> > to
>> > > stream results until the entire result set has been sorted and
>> exported.*
>> > >
>> > > These two statements concludes to me that for fetching entire results
>> > > streaming expressions uses export handler and export handler uses
>> stream,
>> > > so, whether I use streaming expression or export handler, they are
>> > > internally same and would have same performance. I am correct over
>> here
>> > to
>> > > say so ?
>> > >
>> > >
>> > > Ref Links:
>> > >
>> > > https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
>> > > https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
>> > >
>> > >
>> > > Regards
>> > > Kamal Kishore
>> >
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr Stream vs Export Request Handlers

Joel Bernstein
Yes this is correct. But keep in mind Streaming Expression has a wide range
of features that have nothing at all to do with the export handler. In
general with Streaming Expressions you want to find the functions that get
the job done using the least amount of work. The /export handler is often
not the best choice. You'll want to read through the various streaming
expressions to see if they might be more efficient for your use case.


Joel Bernstein
http://joelsolr.blogspot.com/


On Mon, Oct 15, 2018 at 12:05 PM Kamal Kishore Aggarwal <
[hidden email]> wrote:

> Hi,
>
> After I performed the test on my data, I found out that direct /export and
> streaming expression with export, both are giving almost same response
> time. This was also pointed out by *Jan Høydahl* in his reply.
>
> Also, the documentation says export feature uses stream sorting technique
> and streaming expression also uses steam technique. So, are they internally
> works in same fashion. Please confirm.
>
> Regards
> Kamal Kishore
>
>
>
> On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <
> [hidden email]>
> wrote:
>
> > Hi,
> >
> > Thanks Jan & Joel.
> >
> > Though I will evaluate the performance over my data, but based on your
> > experience, which one of the two is better in performance ?. Please
> suggest
> >
> > Yeah, I know export does not get the data from all shards, but we can
> > write code to aggregate the data from all shards. But only if export is
> > better over stream.
> >
> > Thanks
> > Kamal Kishore
> >
> >
> > On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <[hidden email]>
> > wrote:
> >
> >> The export handler does not do distributed search. So if you have a
> >> multi-shard collection you may have to use Streaming Expressions to get
> >> exports from all shards.
> >>
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >>
> >> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <[hidden email]>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > Yes, you can choose which to use, it should give  you about same
> result.
> >> > If you already work with the Solr search API it would be the easiest
> for
> >> > you to consume /export as you don't need to learn the new syntax and
> >> parse
> >> > the Tuple response. However, if you need to do stuff with the docs as
> >> you
> >> > stream them from Solr, then streaming expressions lets you enrich the
> >> docs,
> >> > modify, join etc on the fly.
> >> >
> >> > PS: When the /export docs says it uses a streaming tecnique, it does
> NOT
> >> > mean that it has uses the solr feature streaming expressions :)
> >> >
> >> > --
> >> > Jan Høydahl, search solution architect
> >> > Cominvent AS - www.cominvent.com
> >> >
> >> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> >> > [hidden email]>:
> >> > >
> >> > > Hi,
> >> > >
> >> > > I have a requirement to fetch all data from a collection. One way is
> >> to
> >> > use
> >> > > streaming expression and other way is to use export.
> >> > >
> >> > > Streaming expression documentation says *streaming functions are
> >> designed
> >> > > to work with entire result sets rather then the top N results like
> >> normal
> >> > > search. This is supported by the /export handler.*
> >> > >
> >> > > Also, Export handler documentation says *this feature uses a stream
> >> > sorting
> >> > > technique that begins to send records within milliseconds and
> >> continues
> >> > to
> >> > > stream results until the entire result set has been sorted and
> >> exported.*
> >> > >
> >> > > These two statements concludes to me that for fetching entire
> results
> >> > > streaming expressions uses export handler and export handler uses
> >> stream,
> >> > > so, whether I use streaming expression or export handler, they are
> >> > > internally same and would have same performance. I am correct over
> >> here
> >> > to
> >> > > say so ?
> >> > >
> >> > >
> >> > > Ref Links:
> >> > >
> >> > > https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> >> > > https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> >> > >
> >> > >
> >> > > Regards
> >> > > Kamal Kishore
> >> >
> >> >
> >>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr Stream vs Export Request Handlers

kamal11
Hi,

Thanks again Joel for your reply. I have noted your suggestions.

I observed one more thing while using solrj to fetch the data using /stream
with export and direct /export. The solr QTime is almost same, however
elapsed time(total time) to fetch response in streaming with export is
better than direct /export (Streaming export taking 30% less than /export).

Is this also expected ?

Regards
Kamal Kishore



On Tue, Oct 16, 2018 at 3:21 AM Joel Bernstein <[hidden email]> wrote:

> Yes this is correct. But keep in mind Streaming Expression has a wide range
> of features that have nothing at all to do with the export handler. In
> general with Streaming Expressions you want to find the functions that get
> the job done using the least amount of work. The /export handler is often
> not the best choice. You'll want to read through the various streaming
> expressions to see if they might be more efficient for your use case.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Mon, Oct 15, 2018 at 12:05 PM Kamal Kishore Aggarwal <
> [hidden email]> wrote:
>
> > Hi,
> >
> > After I performed the test on my data, I found out that direct /export
> and
> > streaming expression with export, both are giving almost same response
> > time. This was also pointed out by *Jan Høydahl* in his reply.
> >
> > Also, the documentation says export feature uses stream sorting technique
> > and streaming expression also uses steam technique. So, are they
> internally
> > works in same fashion. Please confirm.
> >
> > Regards
> > Kamal Kishore
> >
> >
> >
> > On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <
> > [hidden email]>
> > wrote:
> >
> > > Hi,
> > >
> > > Thanks Jan & Joel.
> > >
> > > Though I will evaluate the performance over my data, but based on your
> > > experience, which one of the two is better in performance ?. Please
> > suggest
> > >
> > > Yeah, I know export does not get the data from all shards, but we can
> > > write code to aggregate the data from all shards. But only if export is
> > > better over stream.
> > >
> > > Thanks
> > > Kamal Kishore
> > >
> > >
> > > On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <[hidden email]>
> > > wrote:
> > >
> > >> The export handler does not do distributed search. So if you have a
> > >> multi-shard collection you may have to use Streaming Expressions to
> get
> > >> exports from all shards.
> > >>
> > >>
> > >> Joel Bernstein
> > >> http://joelsolr.blogspot.com/
> > >>
> > >>
> > >> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <[hidden email]>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > Yes, you can choose which to use, it should give  you about same
> > result.
> > >> > If you already work with the Solr search API it would be the easiest
> > for
> > >> > you to consume /export as you don't need to learn the new syntax and
> > >> parse
> > >> > the Tuple response. However, if you need to do stuff with the docs
> as
> > >> you
> > >> > stream them from Solr, then streaming expressions lets you enrich
> the
> > >> docs,
> > >> > modify, join etc on the fly.
> > >> >
> > >> > PS: When the /export docs says it uses a streaming tecnique, it does
> > NOT
> > >> > mean that it has uses the solr feature streaming expressions :)
> > >> >
> > >> > --
> > >> > Jan Høydahl, search solution architect
> > >> > Cominvent AS - www.cominvent.com
> > >> >
> > >> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> > >> > [hidden email]>:
> > >> > >
> > >> > > Hi,
> > >> > >
> > >> > > I have a requirement to fetch all data from a collection. One way
> is
> > >> to
> > >> > use
> > >> > > streaming expression and other way is to use export.
> > >> > >
> > >> > > Streaming expression documentation says *streaming functions are
> > >> designed
> > >> > > to work with entire result sets rather then the top N results like
> > >> normal
> > >> > > search. This is supported by the /export handler.*
> > >> > >
> > >> > > Also, Export handler documentation says *this feature uses a
> stream
> > >> > sorting
> > >> > > technique that begins to send records within milliseconds and
> > >> continues
> > >> > to
> > >> > > stream results until the entire result set has been sorted and
> > >> exported.*
> > >> > >
> > >> > > These two statements concludes to me that for fetching entire
> > results
> > >> > > streaming expressions uses export handler and export handler uses
> > >> stream,
> > >> > > so, whether I use streaming expression or export handler, they are
> > >> > > internally same and would have same performance. I am correct over
> > >> here
> > >> > to
> > >> > > say so ?
> > >> > >
> > >> > >
> > >> > > Ref Links:
> > >> > >
> > >> > >
> https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> > >> > >
> https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> > >> > >
> > >> > >
> > >> > > Regards
> > >> > > Kamal Kishore
> > >> >
> > >> >
> > >>
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr Stream vs Export Request Handlers

kamal11
Any update on this.

Regards
Kamal

On Thu, Oct 18, 2018 at 11:50 AM Kamal Kishore Aggarwal <
[hidden email]> wrote:

> Hi,
>
> Thanks again Joel for your reply. I have noted your suggestions.
>
> I observed one more thing while using solrj to fetch the data using
> /stream with export and direct /export. The solr QTime is almost same,
> however elapsed time(total time) to fetch response in streaming with export
> is better than direct /export (Streaming export taking 30% less than
> /export).
>
> Is this also expected ?
>
> Regards
> Kamal Kishore
>
>
>
> On Tue, Oct 16, 2018 at 3:21 AM Joel Bernstein <[hidden email]> wrote:
>
>> Yes this is correct. But keep in mind Streaming Expression has a wide
>> range
>> of features that have nothing at all to do with the export handler. In
>> general with Streaming Expressions you want to find the functions that get
>> the job done using the least amount of work. The /export handler is often
>> not the best choice. You'll want to read through the various streaming
>> expressions to see if they might be more efficient for your use case.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>> On Mon, Oct 15, 2018 at 12:05 PM Kamal Kishore Aggarwal <
>> [hidden email]> wrote:
>>
>> > Hi,
>> >
>> > After I performed the test on my data, I found out that direct /export
>> and
>> > streaming expression with export, both are giving almost same response
>> > time. This was also pointed out by *Jan Høydahl* in his reply.
>> >
>> > Also, the documentation says export feature uses stream sorting
>> technique
>> > and streaming expression also uses steam technique. So, are they
>> internally
>> > works in same fashion. Please confirm.
>> >
>> > Regards
>> > Kamal Kishore
>> >
>> >
>> >
>> > On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <
>> > [hidden email]>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > Thanks Jan & Joel.
>> > >
>> > > Though I will evaluate the performance over my data, but based on your
>> > > experience, which one of the two is better in performance ?. Please
>> > suggest
>> > >
>> > > Yeah, I know export does not get the data from all shards, but we can
>> > > write code to aggregate the data from all shards. But only if export
>> is
>> > > better over stream.
>> > >
>> > > Thanks
>> > > Kamal Kishore
>> > >
>> > >
>> > > On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <[hidden email]>
>> > > wrote:
>> > >
>> > >> The export handler does not do distributed search. So if you have a
>> > >> multi-shard collection you may have to use Streaming Expressions to
>> get
>> > >> exports from all shards.
>> > >>
>> > >>
>> > >> Joel Bernstein
>> > >> http://joelsolr.blogspot.com/
>> > >>
>> > >>
>> > >> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <[hidden email]>
>> > >> wrote:
>> > >>
>> > >> > Hi,
>> > >> >
>> > >> > Yes, you can choose which to use, it should give  you about same
>> > result.
>> > >> > If you already work with the Solr search API it would be the
>> easiest
>> > for
>> > >> > you to consume /export as you don't need to learn the new syntax
>> and
>> > >> parse
>> > >> > the Tuple response. However, if you need to do stuff with the docs
>> as
>> > >> you
>> > >> > stream them from Solr, then streaming expressions lets you enrich
>> the
>> > >> docs,
>> > >> > modify, join etc on the fly.
>> > >> >
>> > >> > PS: When the /export docs says it uses a streaming tecnique, it
>> does
>> > NOT
>> > >> > mean that it has uses the solr feature streaming expressions :)
>> > >> >
>> > >> > --
>> > >> > Jan Høydahl, search solution architect
>> > >> > Cominvent AS - www.cominvent.com
>> > >> >
>> > >> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
>> > >> > [hidden email]>:
>> > >> > >
>> > >> > > Hi,
>> > >> > >
>> > >> > > I have a requirement to fetch all data from a collection. One
>> way is
>> > >> to
>> > >> > use
>> > >> > > streaming expression and other way is to use export.
>> > >> > >
>> > >> > > Streaming expression documentation says *streaming functions are
>> > >> designed
>> > >> > > to work with entire result sets rather then the top N results
>> like
>> > >> normal
>> > >> > > search. This is supported by the /export handler.*
>> > >> > >
>> > >> > > Also, Export handler documentation says *this feature uses a
>> stream
>> > >> > sorting
>> > >> > > technique that begins to send records within milliseconds and
>> > >> continues
>> > >> > to
>> > >> > > stream results until the entire result set has been sorted and
>> > >> exported.*
>> > >> > >
>> > >> > > These two statements concludes to me that for fetching entire
>> > results
>> > >> > > streaming expressions uses export handler and export handler uses
>> > >> stream,
>> > >> > > so, whether I use streaming expression or export handler, they
>> are
>> > >> > > internally same and would have same performance. I am correct
>> over
>> > >> here
>> > >> > to
>> > >> > > say so ?
>> > >> > >
>> > >> > >
>> > >> > > Ref Links:
>> > >> > >
>> > >> > >
>> https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
>> > >> > >
>> https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
>> > >> > >
>> > >> > >
>> > >> > > Regards
>> > >> > > Kamal Kishore
>> > >> >
>> > >> >
>> > >>
>> > >
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr Stream vs Export Request Handlers

Joel Bernstein
I'm not sure why /stream is exporting faster then /export. It may be that
the different approaches in the client are the reason for the difference.
But the /export handler would be used in both scenarios if you specify
qt=/export in the search() Streaming Expression.


Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Oct 25, 2018 at 3:07 AM Kamal Kishore Aggarwal <
[hidden email]> wrote:

> Any update on this.
>
> Regards
> Kamal
>
> On Thu, Oct 18, 2018 at 11:50 AM Kamal Kishore Aggarwal <
> [hidden email]> wrote:
>
> > Hi,
> >
> > Thanks again Joel for your reply. I have noted your suggestions.
> >
> > I observed one more thing while using solrj to fetch the data using
> > /stream with export and direct /export. The solr QTime is almost same,
> > however elapsed time(total time) to fetch response in streaming with
> export
> > is better than direct /export (Streaming export taking 30% less than
> > /export).
> >
> > Is this also expected ?
> >
> > Regards
> > Kamal Kishore
> >
> >
> >
> > On Tue, Oct 16, 2018 at 3:21 AM Joel Bernstein <[hidden email]>
> wrote:
> >
> >> Yes this is correct. But keep in mind Streaming Expression has a wide
> >> range
> >> of features that have nothing at all to do with the export handler. In
> >> general with Streaming Expressions you want to find the functions that
> get
> >> the job done using the least amount of work. The /export handler is
> often
> >> not the best choice. You'll want to read through the various streaming
> >> expressions to see if they might be more efficient for your use case.
> >>
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >>
> >> On Mon, Oct 15, 2018 at 12:05 PM Kamal Kishore Aggarwal <
> >> [hidden email]> wrote:
> >>
> >> > Hi,
> >> >
> >> > After I performed the test on my data, I found out that direct /export
> >> and
> >> > streaming expression with export, both are giving almost same response
> >> > time. This was also pointed out by *Jan Høydahl* in his reply.
> >> >
> >> > Also, the documentation says export feature uses stream sorting
> >> technique
> >> > and streaming expression also uses steam technique. So, are they
> >> internally
> >> > works in same fashion. Please confirm.
> >> >
> >> > Regards
> >> > Kamal Kishore
> >> >
> >> >
> >> >
> >> > On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <
> >> > [hidden email]>
> >> > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > Thanks Jan & Joel.
> >> > >
> >> > > Though I will evaluate the performance over my data, but based on
> your
> >> > > experience, which one of the two is better in performance ?. Please
> >> > suggest
> >> > >
> >> > > Yeah, I know export does not get the data from all shards, but we
> can
> >> > > write code to aggregate the data from all shards. But only if export
> >> is
> >> > > better over stream.
> >> > >
> >> > > Thanks
> >> > > Kamal Kishore
> >> > >
> >> > >
> >> > > On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein <[hidden email]
> >
> >> > > wrote:
> >> > >
> >> > >> The export handler does not do distributed search. So if you have a
> >> > >> multi-shard collection you may have to use Streaming Expressions to
> >> get
> >> > >> exports from all shards.
> >> > >>
> >> > >>
> >> > >> Joel Bernstein
> >> > >> http://joelsolr.blogspot.com/
> >> > >>
> >> > >>
> >> > >> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl <[hidden email]
> >
> >> > >> wrote:
> >> > >>
> >> > >> > Hi,
> >> > >> >
> >> > >> > Yes, you can choose which to use, it should give  you about same
> >> > result.
> >> > >> > If you already work with the Solr search API it would be the
> >> easiest
> >> > for
> >> > >> > you to consume /export as you don't need to learn the new syntax
> >> and
> >> > >> parse
> >> > >> > the Tuple response. However, if you need to do stuff with the
> docs
> >> as
> >> > >> you
> >> > >> > stream them from Solr, then streaming expressions lets you enrich
> >> the
> >> > >> docs,
> >> > >> > modify, join etc on the fly.
> >> > >> >
> >> > >> > PS: When the /export docs says it uses a streaming tecnique, it
> >> does
> >> > NOT
> >> > >> > mean that it has uses the solr feature streaming expressions :)
> >> > >> >
> >> > >> > --
> >> > >> > Jan Høydahl, search solution architect
> >> > >> > Cominvent AS - www.cominvent.com
> >> > >> >
> >> > >> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> >> > >> > [hidden email]>:
> >> > >> > >
> >> > >> > > Hi,
> >> > >> > >
> >> > >> > > I have a requirement to fetch all data from a collection. One
> >> way is
> >> > >> to
> >> > >> > use
> >> > >> > > streaming expression and other way is to use export.
> >> > >> > >
> >> > >> > > Streaming expression documentation says *streaming functions
> are
> >> > >> designed
> >> > >> > > to work with entire result sets rather then the top N results
> >> like
> >> > >> normal
> >> > >> > > search. This is supported by the /export handler.*
> >> > >> > >
> >> > >> > > Also, Export handler documentation says *this feature uses a
> >> stream
> >> > >> > sorting
> >> > >> > > technique that begins to send records within milliseconds and
> >> > >> continues
> >> > >> > to
> >> > >> > > stream results until the entire result set has been sorted and
> >> > >> exported.*
> >> > >> > >
> >> > >> > > These two statements concludes to me that for fetching entire
> >> > results
> >> > >> > > streaming expressions uses export handler and export handler
> uses
> >> > >> stream,
> >> > >> > > so, whether I use streaming expression or export handler, they
> >> are
> >> > >> > > internally same and would have same performance. I am correct
> >> over
> >> > >> here
> >> > >> > to
> >> > >> > > say so ?
> >> > >> > >
> >> > >> > >
> >> > >> > > Ref Links:
> >> > >> > >
> >> > >> > >
> >> https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> >> > >> > >
> >> https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> >> > >> > >
> >> > >> > >
> >> > >> > > Regards
> >> > >> > > Kamal Kishore
> >> > >> >
> >> > >> >
> >> > >>
> >> > >
> >> >
> >>
> >
>