Monitoring decisions taken by IndexOrDocValuesQuery

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Monitoring decisions taken by IndexOrDocValuesQuery

Egor Moraru
Hi,

At my current project we wanted to monitor for a specific field the
fraction of indexed vs doc values queries executed by IndexOrDocValuesQuery.

We ended up forking IndexOrDocValuesQuery and passing a listener that
is notified when the query execution path is decided.

Do you think this is something the community might be interested in?

Kind regards,
Egor Moraru.
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring decisions taken by IndexOrDocValuesQuery

Adrien Grand
FWIW a related PR was just merged that allows to introspect query
execution: https://issues.apache.org/jira/browse/LUCENE-9965. It's
different from your use-case though in that it is debugging information for
a single query rather than statistical information across lots of user
queries (and the approach on that other issue makes things much slower so
you wouldn't like to enable it in production).

Out of curiosity, what are you doing with this information about which
execution path is chosen?

On Wed, Jun 9, 2021 at 2:14 PM Egor Moraru <[hidden email]> wrote:

> Hi,
>
> At my current project we wanted to monitor for a specific field the
> fraction of indexed vs doc values queries executed by
> IndexOrDocValuesQuery.
>
> We ended up forking IndexOrDocValuesQuery and passing a listener that
> is notified when the query execution path is decided.
>
> Do you think this is something the community might be interested in?
>
> Kind regards,
> Egor Moraru.
>


--
Adrien
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring decisions taken by IndexOrDocValuesQuery

Egor Moraru
Hi Adrien,

In this specific use case our data is encoded as points and also is stored
as doc values.

We use information about which execution path is chosen to decide
if we can get away with storing this data only once and using one of
the queries.

On Wed, Jun 9, 2021 at 10:39 PM Adrien Grand <[hidden email]> wrote:

> FWIW a related PR was just merged that allows to introspect query
> execution: https://issues.apache.org/jira/browse/LUCENE-9965. It's
> different from your use-case though in that it is debugging information for
> a single query rather than statistical information across lots of user
> queries (and the approach on that other issue makes things much slower so
> you wouldn't like to enable it in production).
>
> Out of curiosity, what are you doing with this information about which
> execution path is chosen?
>
> On Wed, Jun 9, 2021 at 2:14 PM Egor Moraru <[hidden email]> wrote:
>
> > Hi,
> >
> > At my current project we wanted to monitor for a specific field the
> > fraction of indexed vs doc values queries executed by
> > IndexOrDocValuesQuery.
> >
> > We ended up forking IndexOrDocValuesQuery and passing a listener that
> > is notified when the query execution path is decided.
> >
> > Do you think this is something the community might be interested in?
> >
> > Kind regards,
> > Egor Moraru.
> >
>
>
> --
> Adrien
>


--

Kind regards,
Egor Moraru.
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring decisions taken by IndexOrDocValuesQuery

Adrien Grand
This sounds useful indeed!

Is there a way we could do it that wouldn't require forking
IndexOrDocValuesQuery? E.g. could we have query wrappers that we would use
on both the index query and the doc-value query in order to be able to
count how many times they have been used? We could add something like that
to lucene/sandbox.

On Thu, Jun 10, 2021 at 2:51 PM Egor Moraru <[hidden email]> wrote:

> Hi Adrien,
>
> In this specific use case our data is encoded as points and also is stored
> as doc values.
>
> We use information about which execution path is chosen to decide
> if we can get away with storing this data only once and using one of
> the queries.
>
> On Wed, Jun 9, 2021 at 10:39 PM Adrien Grand <[hidden email]> wrote:
>
> > FWIW a related PR was just merged that allows to introspect query
> > execution: https://issues.apache.org/jira/browse/LUCENE-9965. It's
> > different from your use-case though in that it is debugging information
> for
> > a single query rather than statistical information across lots of user
> > queries (and the approach on that other issue makes things much slower so
> > you wouldn't like to enable it in production).
> >
> > Out of curiosity, what are you doing with this information about which
> > execution path is chosen?
> >
> > On Wed, Jun 9, 2021 at 2:14 PM Egor Moraru <[hidden email]>
> wrote:
> >
> > > Hi,
> > >
> > > At my current project we wanted to monitor for a specific field the
> > > fraction of indexed vs doc values queries executed by
> > > IndexOrDocValuesQuery.
> > >
> > > We ended up forking IndexOrDocValuesQuery and passing a listener that
> > > is notified when the query execution path is decided.
> > >
> > > Do you think this is something the community might be interested in?
> > >
> > > Kind regards,
> > > Egor Moraru.
> > >
> >
> >
> > --
> > Adrien
> >
>
>
> --
>
> Kind regards,
> Egor Moraru.
>


--
Adrien