gatherNodes question. Is this a bug?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

gatherNodes question. Is this a bug?

Kojo
Hello everybody I have a question about Streaming Expression/Graph
Traversal.

The following pseudocode works fine:

complement( search(),
    sort(
        gatherNodes( collection, search())
    ),
)


However, when I feed the SE resultset above to another gatherNodes
function, I have a result different from what I expected. It returns the
root nodes (branches) of the inner gatherNodes:

gatherNodes( collection,
    complement( search(),
        sort(
           gatherNodes( collection, search())
        ),
    ),
)

In the case I tested, the outer gatherNodes does not have leaves. I was
waiting to have the result from the "complement" function as the root nodes
of the outter gatherNodes function. Do you know how can I achieve this?

Thank you,
Reply | Threaded
Open this post in threaded view
|

Re: gatherNodes question. Is this a bug?

Joel Bernstein
What you're trying to do should work. Possibly of you provide more detail
like the full query with some sample outputs I might be able to see what
the issue is.

Joel Bernstein
http://joelsolr.blogspot.com/


On Wed, Apr 10, 2019 at 10:55 AM Kojo <[hidden email]> wrote:

> Hello everybody I have a question about Streaming Expression/Graph
> Traversal.
>
> The following pseudocode works fine:
>
> complement( search(),
>     sort(
>         gatherNodes( collection, search())
>     ),
> )
>
>
> However, when I feed the SE resultset above to another gatherNodes
> function, I have a result different from what I expected. It returns the
> root nodes (branches) of the inner gatherNodes:
>
> gatherNodes( collection,
>     complement( search(),
>         sort(
>            gatherNodes( collection, search())
>         ),
>     ),
> )
>
> In the case I tested, the outer gatherNodes does not have leaves. I was
> waiting to have the result from the "complement" function as the root nodes
> of the outter gatherNodes function. Do you know how can I achieve this?
>
> Thank you,
>
Reply | Threaded
Open this post in threaded view
|

Re: gatherNodes question. Is this a bug?

Kojo
Joel,
thank you in advance.

Follows SE code and resultset for each piece. I only change some resultset
values, without change the logic.
I am on Solr 6.6.2.

1. First SE - Inner gatherNodes:

        sort(
            gatherNodes( my_collection, gatherNodes( my_collection,
                search(my_collection, qt="/export", q=*:*,
fl="process_number", sort="process_number asc",  fq=-ancestor:*,
fq=situacao:("On going"), fq=area_pt:("Ciências Exatas e da Terra"),
fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
Brasil")),
            walk="process_number->ancestor", trackTraversal="true",
gather="process_number"), walk="node->ancestor", trackTraversal="true",
gather="process_number", scatter="branches, leaves"),
        by="node asc")


{
  "result-set": {
    "docs": [
      {
        "node": "01/09299-2",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/11664-0",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/19764-4",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/20577-4",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/21532-4",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "EOF": true,
        "RESPONSE_TIME": 13
      }
    ]
  }
}


2. SE enclosed by "complement"

complement(
    search(my_collection, qt="/export", q=*:*, fl="process_number",
sort="process_number asc",  fq=ancestor:*, fq=situacao:("On going"),
fq=area_pt:("Ciências Exatas e da Terra"),
fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
Brasil")),
    sort(
        gatherNodes( my_collection, gatherNodes( my_collection,
            search(my_collection, qt="/export", q=*:*, fl="process_number",
sort="process_number asc",  fq=-ancestor:*, fq=situacao:("On going"),
fq=area_pt:("Ciências Exatas e da Terra"),
fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
Brasil")),
        walk="process_number->ancestor", trackTraversal="true",
gather="process_number"), walk="node->ancestor", trackTraversal="true",
gather="process_number", scatter="branches, leaves"),
    by="node asc"),
on="process_number=node")


{
  "result-set": {
    "docs": [
      {
        "process_number": "01/02444-7"
      },
      {
        "process_number": "01/08149-7"
      },
      {
        "process_number": "01/21749-3"
      },
      {
        "process_number": "01/22503-8"
      },
      {
        "EOF": true,
        "RESPONSE_TIME": 18
      }
    ]
  }
}

3. Fetch some extra data:

fetch(my_collection, select(
    complement(
        search(my_collection, qt="/export", q=*:*, fl="process_number",
sort="process_number asc",  fq=ancestor:*, fq=situacao:("On going"),
fq=area_pt:("Ciências Exatas e da Terra"),
fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
Brasil")),
        sort(
            gatherNodes( my_collection, gatherNodes( my_collection,
                search(my_collection, qt="/export", q=*:*,
fl="process_number", sort="process_number asc",  fq=-ancestor:*,
fq=situacao:("On going"), fq=area_pt:("Ciências Exatas e da Terra"),
fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
Brasil")),
            walk="process_number->ancestor", trackTraversal="true",
gather="process_number"), walk="node->ancestor", trackTraversal="true",
gather="process_number", scatter="branches, leaves"),
        by="node asc"),
    on="process_number=node"),
process_number as node),fl="type_status_facet , amount",
on="node=process_number",)

{
  "result-set": {
    "docs": [
      {
        "node": "01/02444-7",
        "type_status_facet ": "Ongoing projetcs",
        "amount": 154620
      },
      {
        "node": "01/08149-7",
        "type_status_facet ": "Ongoing projetcs",
        "amount": 131115
      },
      {
        "node": "01/21749-3",
        "type_status_facet ": "Ongoing projetcs",
        "amount": 157300
      },
      {
        "node": "01/22503-8",
        "type_status_facet ": "Ongoing projetcs",
        "amount": 154800
      },
      {
        "EOF": true,
        "RESPONSE_TIME": 24
      }
    ]
  }
}

4. Trying to gather more nodes. Here is the problem, the resultset comes
from the inner gatherNodes.

gatherNodes( my_collection, gatherNodes( my_collection,
    fetch(my_collection, select(
        complement(
            search(my_collection, qt="/export", q=*:*, fl="process_number",
sort="process_number asc",  fq=ancestor:*, fq=situacao:("On going"),
fq=area_pt:("Ciências Exatas e da Terra"),
fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
Brasil")),
            sort(
                gatherNodes( my_collection, gatherNodes( my_collection,
                    search(my_collection, qt="/export", q=*:*,
fl="process_number", sort="process_number asc",  fq=-ancestor:*,
fq=situacao:("On going"), fq=area_pt:("Ciências Exatas e da Terra"),
fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
Brasil")),
                walk="process_number->ancestor", trackTraversal="true",
gather="process_number"), walk="node->ancestor", trackTraversal="true",
gather="process_number", scatter="branches, leaves"),
            by="node asc"),
        on="process_number=node"),
    process_number as node),fl="type_status_facet , amount",
on="node=process_number"),
walk="node->ancestor", trackTraversal="true", gather="process_number"),
walk="node->ancestor", trackTraversal="true", gather="process_number",
scatter="branches, leaves")

{
  "result-set": {
    "docs": [
      {
        "node": "01/20577-4",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/19764-4",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/09299-2",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/21532-4",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/11664-0",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "EOF": true,
        "RESPONSE_TIME": 30
      }
    ]
  }
}




5. Just one more information, maybe not necessary.
 Enclose step 3 wiht rollup. Check "level" field on the result-set, it is
NULL


rollup(having(sort(
    fetch(my_collection, select(
        complement(
            search(my_collection, qt="/export", q=*:*, fl="process_number",
sort="process_number asc",  fq=ancestor:*, fq=situacao:("On going"),
fq=area_pt:("Ciências Exatas e da Terra"),
fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
Brasil")),
            sort(
                gatherNodes( my_collection, gatherNodes( my_collection,
                    search(my_collection, qt="/export", q=*:*,
fl="process_number", sort="process_number asc",  fq=-ancestor:*,
fq=situacao:("On going"), fq=area_pt:("Ciências Exatas e da Terra"),
fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
Brasil")),
                walk="process_number->ancestor", trackTraversal="true",
gather="process_number"), walk="node->ancestor", trackTraversal="true",
gather="process_number", scatter="branches, leaves"),
            by="node asc"),
        on="process_number=node"),
    process_number as node),fl="type_status_facet , amount",
on="node=process_number"),
by="level asc, type_status_facet  asc"),gt(if(eq(amount,null), 0,
amount),0)), over="level, type_status_facet ", sum(amount), count(*))

{
  "result-set": {
    "docs": [
      {
        "sum(amount)": 597835,
        "count(*)": 4,
        "type_status_facet ": "Ongoing projetcs",
        "level": "NULL"
      },
      {
        "EOF": true,
        "RESPONSE_TIME": 26
      }
    ]
  }
}

Em qua, 10 de abr de 2019 às 16:49, Joel Bernstein <[hidden email]>
escreveu:

> What you're trying to do should work. Possibly of you provide more detail
> like the full query with some sample outputs I might be able to see what
> the issue is.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Wed, Apr 10, 2019 at 10:55 AM Kojo <[hidden email]> wrote:
>
> > Hello everybody I have a question about Streaming Expression/Graph
> > Traversal.
> >
> > The following pseudocode works fine:
> >
> > complement( search(),
> >     sort(
> >         gatherNodes( collection, search())
> >     ),
> > )
> >
> >
> > However, when I feed the SE resultset above to another gatherNodes
> > function, I have a result different from what I expected. It returns the
> > root nodes (branches) of the inner gatherNodes:
> >
> > gatherNodes( collection,
> >     complement( search(),
> >         sort(
> >            gatherNodes( collection, search())
> >         ),
> >     ),
> > )
> >
> > In the case I tested, the outer gatherNodes does not have leaves. I was
> > waiting to have the result from the "complement" function as the root
> nodes
> > of the outter gatherNodes function. Do you know how can I achieve this?
> >
> > Thank you,
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: gatherNodes question. Is this a bug?

Joel Bernstein
The outer gather nodes is bringing back a different set of nodes:

{
  "result-set": {
    "docs": [
      {
        "node": "01/20577-4",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/19764-4",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/09299-2",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/21532-4",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "node": "01/11664-0",
        "collection": "my_collection",
        "field": "process_number",
        "ancestors": [],
        "level": 0
      },
      {
        "EOF": true,
        "RESPONSE_TIME": 30
      }
    ]
  }
}

Here is the inner node set:

{
  "result-set": {
    "docs": [
      {
        "node": "01/02444-7",
        "type_status_facet ": "Ongoing projetcs",
        "amount": 154620
      },
      {
        "node": "01/08149-7",
        "type_status_facet ": "Ongoing projetcs",
        "amount": 131115
      },
      {
        "node": "01/21749-3",
        "type_status_facet ": "Ongoing projetcs",
        "amount": 157300
      },
      {
        "node": "01/22503-8",
        "type_status_facet ": "Ongoing projetcs",
        "amount": 154800
      },
      {
        "EOF": true,
        "RESPONSE_TIME": 24
      }
    ]
  }
}

This is to be expected.

Try turning on tracking, with trackTraversal="true".

This will show the ancestors and hopefully make things more clear.


Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Apr 11, 2019 at 12:07 PM Kojo <[hidden email]> wrote:

> Joel,
> thank you in advance.
>
> Follows SE code and resultset for each piece. I only change some resultset
> values, without change the logic.
> I am on Solr 6.6.2.
>
> 1. First SE - Inner gatherNodes:
>
>         sort(
>             gatherNodes( my_collection, gatherNodes( my_collection,
>                 search(my_collection, qt="/export", q=*:*,
> fl="process_number", sort="process_number asc",  fq=-ancestor:*,
> fq=situacao:("On going"), fq=area_pt:("Ciências Exatas e da Terra"),
> fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
> Brasil")),
>             walk="process_number->ancestor", trackTraversal="true",
> gather="process_number"), walk="node->ancestor", trackTraversal="true",
> gather="process_number", scatter="branches, leaves"),
>         by="node asc")
>
>
> {
>   "result-set": {
>     "docs": [
>       {
>         "node": "01/09299-2",
>         "collection": "my_collection",
>         "field": "process_number",
>         "ancestors": [],
>         "level": 0
>       },
>       {
>         "node": "01/11664-0",
>         "collection": "my_collection",
>         "field": "process_number",
>         "ancestors": [],
>         "level": 0
>       },
>       {
>         "node": "01/19764-4",
>         "collection": "my_collection",
>         "field": "process_number",
>         "ancestors": [],
>         "level": 0
>       },
>       {
>         "node": "01/20577-4",
>         "collection": "my_collection",
>         "field": "process_number",
>         "ancestors": [],
>         "level": 0
>       },
>       {
>         "node": "01/21532-4",
>         "collection": "my_collection",
>         "field": "process_number",
>         "ancestors": [],
>         "level": 0
>       },
>       {
>         "EOF": true,
>         "RESPONSE_TIME": 13
>       }
>     ]
>   }
> }
>
>
> 2. SE enclosed by "complement"
>
> complement(
>     search(my_collection, qt="/export", q=*:*, fl="process_number",
> sort="process_number asc",  fq=ancestor:*, fq=situacao:("On going"),
> fq=area_pt:("Ciências Exatas e da Terra"),
> fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
> Brasil")),
>     sort(
>         gatherNodes( my_collection, gatherNodes( my_collection,
>             search(my_collection, qt="/export", q=*:*, fl="process_number",
> sort="process_number asc",  fq=-ancestor:*, fq=situacao:("On going"),
> fq=area_pt:("Ciências Exatas e da Terra"),
> fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
> Brasil")),
>         walk="process_number->ancestor", trackTraversal="true",
> gather="process_number"), walk="node->ancestor", trackTraversal="true",
> gather="process_number", scatter="branches, leaves"),
>     by="node asc"),
> on="process_number=node")
>
>
> {
>   "result-set": {
>     "docs": [
>       {
>         "process_number": "01/02444-7"
>       },
>       {
>         "process_number": "01/08149-7"
>       },
>       {
>         "process_number": "01/21749-3"
>       },
>       {
>         "process_number": "01/22503-8"
>       },
>       {
>         "EOF": true,
>         "RESPONSE_TIME": 18
>       }
>     ]
>   }
> }
>
> 3. Fetch some extra data:
>
> fetch(my_collection, select(
>     complement(
>         search(my_collection, qt="/export", q=*:*, fl="process_number",
> sort="process_number asc",  fq=ancestor:*, fq=situacao:("On going"),
> fq=area_pt:("Ciências Exatas e da Terra"),
> fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
> Brasil")),
>         sort(
>             gatherNodes( my_collection, gatherNodes( my_collection,
>                 search(my_collection, qt="/export", q=*:*,
> fl="process_number", sort="process_number asc",  fq=-ancestor:*,
> fq=situacao:("On going"), fq=area_pt:("Ciências Exatas e da Terra"),
> fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
> Brasil")),
>             walk="process_number->ancestor", trackTraversal="true",
> gather="process_number"), walk="node->ancestor", trackTraversal="true",
> gather="process_number", scatter="branches, leaves"),
>         by="node asc"),
>     on="process_number=node"),
> process_number as node),fl="type_status_facet , amount",
> on="node=process_number",)
>
> {
>   "result-set": {
>     "docs": [
>       {
>         "node": "01/02444-7",
>         "type_status_facet ": "Ongoing projetcs",
>         "amount": 154620
>       },
>       {
>         "node": "01/08149-7",
>         "type_status_facet ": "Ongoing projetcs",
>         "amount": 131115
>       },
>       {
>         "node": "01/21749-3",
>         "type_status_facet ": "Ongoing projetcs",
>         "amount": 157300
>       },
>       {
>         "node": "01/22503-8",
>         "type_status_facet ": "Ongoing projetcs",
>         "amount": 154800
>       },
>       {
>         "EOF": true,
>         "RESPONSE_TIME": 24
>       }
>     ]
>   }
> }
>
> 4. Trying to gather more nodes. Here is the problem, the resultset comes
> from the inner gatherNodes.
>
> gatherNodes( my_collection, gatherNodes( my_collection,
>     fetch(my_collection, select(
>         complement(
>             search(my_collection, qt="/export", q=*:*, fl="process_number",
> sort="process_number asc",  fq=ancestor:*, fq=situacao:("On going"),
> fq=area_pt:("Ciências Exatas e da Terra"),
> fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
> Brasil")),
>             sort(
>                 gatherNodes( my_collection, gatherNodes( my_collection,
>                     search(my_collection, qt="/export", q=*:*,
> fl="process_number", sort="process_number asc",  fq=-ancestor:*,
> fq=situacao:("On going"), fq=area_pt:("Ciências Exatas e da Terra"),
> fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
> Brasil")),
>                 walk="process_number->ancestor", trackTraversal="true",
> gather="process_number"), walk="node->ancestor", trackTraversal="true",
> gather="process_number", scatter="branches, leaves"),
>             by="node asc"),
>         on="process_number=node"),
>     process_number as node),fl="type_status_facet , amount",
> on="node=process_number"),
> walk="node->ancestor", trackTraversal="true", gather="process_number"),
> walk="node->ancestor", trackTraversal="true", gather="process_number",
> scatter="branches, leaves")
>
> {
>   "result-set": {
>     "docs": [
>       {
>         "node": "01/20577-4",
>         "collection": "my_collection",
>         "field": "process_number",
>         "ancestors": [],
>         "level": 0
>       },
>       {
>         "node": "01/19764-4",
>         "collection": "my_collection",
>         "field": "process_number",
>         "ancestors": [],
>         "level": 0
>       },
>       {
>         "node": "01/09299-2",
>         "collection": "my_collection",
>         "field": "process_number",
>         "ancestors": [],
>         "level": 0
>       },
>       {
>         "node": "01/21532-4",
>         "collection": "my_collection",
>         "field": "process_number",
>         "ancestors": [],
>         "level": 0
>       },
>       {
>         "node": "01/11664-0",
>         "collection": "my_collection",
>         "field": "process_number",
>         "ancestors": [],
>         "level": 0
>       },
>       {
>         "EOF": true,
>         "RESPONSE_TIME": 30
>       }
>     ]
>   }
> }
>
>
>
>
> 5. Just one more information, maybe not necessary.
>  Enclose step 3 wiht rollup. Check "level" field on the result-set, it is
> NULL
>
>
> rollup(having(sort(
>     fetch(my_collection, select(
>         complement(
>             search(my_collection, qt="/export", q=*:*, fl="process_number",
> sort="process_number asc",  fq=ancestor:*, fq=situacao:("On going"),
> fq=area_pt:("Ciências Exatas e da Terra"),
> fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
> Brasil")),
>             sort(
>                 gatherNodes( my_collection, gatherNodes( my_collection,
>                     search(my_collection, qt="/export", q=*:*,
> fl="process_number", sort="process_number asc",  fq=-ancestor:*,
> fq=situacao:("On going"), fq=area_pt:("Ciências Exatas e da Terra"),
> fq=auxilio_pesquisa_pt:("07;Auxílio Pesquisador|00;Auxilio Pesquisador -
> Brasil")),
>                 walk="process_number->ancestor", trackTraversal="true",
> gather="process_number"), walk="node->ancestor", trackTraversal="true",
> gather="process_number", scatter="branches, leaves"),
>             by="node asc"),
>         on="process_number=node"),
>     process_number as node),fl="type_status_facet , amount",
> on="node=process_number"),
> by="level asc, type_status_facet  asc"),gt(if(eq(amount,null), 0,
> amount),0)), over="level, type_status_facet ", sum(amount), count(*))
>
> {
>   "result-set": {
>     "docs": [
>       {
>         "sum(amount)": 597835,
>         "count(*)": 4,
>         "type_status_facet ": "Ongoing projetcs",
>         "level": "NULL"
>       },
>       {
>         "EOF": true,
>         "RESPONSE_TIME": 26
>       }
>     ]
>   }
> }
>
> Em qua, 10 de abr de 2019 às 16:49, Joel Bernstein <[hidden email]>
> escreveu:
>
> > What you're trying to do should work. Possibly of you provide more detail
> > like the full query with some sample outputs I might be able to see what
> > the issue is.
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> >
> > On Wed, Apr 10, 2019 at 10:55 AM Kojo <[hidden email]> wrote:
> >
> > > Hello everybody I have a question about Streaming Expression/Graph
> > > Traversal.
> > >
> > > The following pseudocode works fine:
> > >
> > > complement( search(),
> > >     sort(
> > >         gatherNodes( collection, search())
> > >     ),
> > > )
> > >
> > >
> > > However, when I feed the SE resultset above to another gatherNodes
> > > function, I have a result different from what I expected. It returns
> the
> > > root nodes (branches) of the inner gatherNodes:
> > >
> > > gatherNodes( collection,
> > >     complement( search(),
> > >         sort(
> > >            gatherNodes( collection, search())
> > >         ),
> > >     ),
> > > )
> > >
> > > In the case I tested, the outer gatherNodes does not have leaves. I was
> > > waiting to have the result from the "complement" function as the root
> > nodes
> > > of the outter gatherNodes function. Do you know how can I achieve this?
> > >
> > > Thank you,
> > >
> >
>