How to query against dynamic fields without listing them all?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

How to query against dynamic fields without listing them all?

Steven White
Hi everyone,

In my schema, I have the following field:

  <dynamicField name="CC_COMP_NAME_*" type="file_text" multiValued="true"
indexed="true" required="false" stored="false"/>

When I index, I create dynamic fields and index into it like so:

  doc.addField("CC_COMP_NAME_" + componentName.toUpperCase(),
ccAllFieldsDataValue);

In my query handler, I have this:

  {"requestHandler":{"/select_hcl":{
      "class":"solr.SearchHandler",
      "name":"/select_hcl",
      "defaults":{
        "defType":"edismax",
        "echoParams":"explicit",
        "fl":"CC_UNIQUE_FIELD,CC_FILE_PATH,score",
        "indent":"true",
        "qf":"CC_COMP_NAME_*",
        "rows":"100",
        "wt":"xml"}}}}

My expectation was when i query using this handler, it will include all the
dynamic fields with the prefix of "CC_COMP_NAME_" however, that is not
happening and I'm getting 0 hits.  But when I use the full field name, such
as CC_COMP_NAME_1 or CC_COMP_NAME_2, that works so I know my data is
indexed, it is just that Solr not paying attention to the dynamic field
syntax in "qf".

I don't want to keep a list of those dynamic fields and pass them to my
handler, but if I must, than I must.  If so, how can I get the list of
those dynamic fields from Solr so that I don't have to maintain and sync-up
the list myself.

Thanks

Steven
Reply | Threaded
Open this post in threaded view
|

Re: How to query against dynamic fields without listing them all?

David Santamauro
Hi Steven,

You can dump all the dynamic fields into a copyField

    <field name="CC_COMP_NAME_ALL"  type="file _text" indexed="true" stored="false" multiValued="true" />
    <copyField source="CC_COMP_NAME_*"    dest="CC_COMP_NAME_ALL" />

Then you can just set
  "qf":"CC_COMP_NAME_ALL"


On 7/14/19, 10:42 AM, "Steven White" <[hidden email]> wrote:

    Hi everyone,
   
    In my schema, I have the following field:
   
      <dynamicField name="CC_COMP_NAME_*" type="file_text" multiValued="true"
    indexed="true" required="false" stored="false"/>
   
    When I index, I create dynamic fields and index into it like so:
   
      doc.addField("CC_COMP_NAME_" + componentName.toUpperCase(),
    ccAllFieldsDataValue);
   
    In my query handler, I have this:
   
      {"requestHandler":{"/select_hcl":{
          "class":"solr.SearchHandler",
          "name":"/select_hcl",
          "defaults":{
            "defType":"edismax",
            "echoParams":"explicit",
            "fl":"CC_UNIQUE_FIELD,CC_FILE_PATH,score",
            "indent":"true",
            "qf":"CC_COMP_NAME_*",
            "rows":"100",
            "wt":"xml"}}}}
   
    My expectation was when i query using this handler, it will include all the
    dynamic fields with the prefix of "CC_COMP_NAME_" however, that is not
    happening and I'm getting 0 hits.  But when I use the full field name, such
    as CC_COMP_NAME_1 or CC_COMP_NAME_2, that works so I know my data is
    indexed, it is just that Solr not paying attention to the dynamic field
    syntax in "qf".
   
    I don't want to keep a list of those dynamic fields and pass them to my
    handler, but if I must, than I must.  If so, how can I get the list of
    those dynamic fields from Solr so that I don't have to maintain and sync-up
    the list myself.
   
    Thanks
   
    Steven
   
Reply | Threaded
Open this post in threaded view
|

Re: How to query against dynamic fields without listing them all?

Steven White
Thanks David.  Yes, I see that would work, but wouldn't that increase my
index size?  It would be best if Solr supported dynamic field in "qf".

Steven.

On Sun, Jul 14, 2019 at 11:02 AM David Santamauro <
[hidden email]> wrote:

> Hi Steven,
>
> You can dump all the dynamic fields into a copyField
>
>     <field name="CC_COMP_NAME_ALL"  type="file _text" indexed="true"
> stored="false" multiValued="true" />
>     <copyField source="CC_COMP_NAME_*"    dest="CC_COMP_NAME_ALL" />
>
> Then you can just set
>   "qf":"CC_COMP_NAME_ALL"
>
>
> On 7/14/19, 10:42 AM, "Steven White" <[hidden email]> wrote:
>
>     Hi everyone,
>
>     In my schema, I have the following field:
>
>       <dynamicField name="CC_COMP_NAME_*" type="file_text"
> multiValued="true"
>     indexed="true" required="false" stored="false"/>
>
>     When I index, I create dynamic fields and index into it like so:
>
>       doc.addField("CC_COMP_NAME_" + componentName.toUpperCase(),
>     ccAllFieldsDataValue);
>
>     In my query handler, I have this:
>
>       {"requestHandler":{"/select_hcl":{
>           "class":"solr.SearchHandler",
>           "name":"/select_hcl",
>           "defaults":{
>             "defType":"edismax",
>             "echoParams":"explicit",
>             "fl":"CC_UNIQUE_FIELD,CC_FILE_PATH,score",
>             "indent":"true",
>             "qf":"CC_COMP_NAME_*",
>             "rows":"100",
>             "wt":"xml"}}}}
>
>     My expectation was when i query using this handler, it will include
> all the
>     dynamic fields with the prefix of "CC_COMP_NAME_" however, that is not
>     happening and I'm getting 0 hits.  But when I use the full field name,
> such
>     as CC_COMP_NAME_1 or CC_COMP_NAME_2, that works so I know my data is
>     indexed, it is just that Solr not paying attention to the dynamic field
>     syntax in "qf".
>
>     I don't want to keep a list of those dynamic fields and pass them to my
>     handler, but if I must, than I must.  If so, how can I get the list of
>     those dynamic fields from Solr so that I don't have to maintain and
> sync-up
>     the list myself.
>
>     Thanks
>
>     Steven
>
>
Reply | Threaded
Open this post in threaded view
|

Re: How to query against dynamic fields without listing them all?

Alexandre Rafalovitch
The other options is to use query field alias:
https://lucene.apache.org/solr/guide/8_1/the-extended-dismax-query-parser.html#field-aliasing-using-per-field-qf-overrides
(example: https://github.com/arafalov/solr-indexing-book/blob/master/published/languages/conf/solrconfig.xml#L20
). This still does not allow wildcards (I think, check!), but at least
it allows to isolate field names as a reference in a separate
variable.
And you can update that variable using Request Parameter API:
https://lucene.apache.org/solr/guide/8_1/request-parameters-api.html#request-parameters-api

You can periodically get the list of all fields, using Luke endpoint.
I don't remember if/where it is documented, but you can examine the
traffic between Admin UI (Analysis or Schema screen) and backend to
see how it works.

You may even have a custom postCommit listener defined that could
check the current list of dynamic fields and write appropriate
mapping: https://lucene.apache.org/solr/guide/8_1/updatehandlers-in-solrconfig.html#event-listeners

Regards,
   Alex.
P.s. If you use copyField and the target does not store content, just
index - then the overhead is not too bad. And if your original fields
are store-only (because they are never searched directly) and the
copyField target is index-only, then there is no extra overhead.







On Sun, 14 Jul 2019 at 16:37, Steven White <[hidden email]> wrote:

>
> Thanks David.  Yes, I see that would work, but wouldn't that increase my
> index size?  It would be best if Solr supported dynamic field in "qf".
>
> Steven.
>
> On Sun, Jul 14, 2019 at 11:02 AM David Santamauro <
> [hidden email]> wrote:
>
> > Hi Steven,
> >
> > You can dump all the dynamic fields into a copyField
> >
> >     <field name="CC_COMP_NAME_ALL"  type="file _text" indexed="true"
> > stored="false" multiValued="true" />
> >     <copyField source="CC_COMP_NAME_*"    dest="CC_COMP_NAME_ALL" />
> >
> > Then you can just set
> >   "qf":"CC_COMP_NAME_ALL"
> >
> >
> > On 7/14/19, 10:42 AM, "Steven White" <[hidden email]> wrote:
> >
> >     Hi everyone,
> >
> >     In my schema, I have the following field:
> >
> >       <dynamicField name="CC_COMP_NAME_*" type="file_text"
> > multiValued="true"
> >     indexed="true" required="false" stored="false"/>
> >
> >     When I index, I create dynamic fields and index into it like so:
> >
> >       doc.addField("CC_COMP_NAME_" + componentName.toUpperCase(),
> >     ccAllFieldsDataValue);
> >
> >     In my query handler, I have this:
> >
> >       {"requestHandler":{"/select_hcl":{
> >           "class":"solr.SearchHandler",
> >           "name":"/select_hcl",
> >           "defaults":{
> >             "defType":"edismax",
> >             "echoParams":"explicit",
> >             "fl":"CC_UNIQUE_FIELD,CC_FILE_PATH,score",
> >             "indent":"true",
> >             "qf":"CC_COMP_NAME_*",
> >             "rows":"100",
> >             "wt":"xml"}}}}
> >
> >     My expectation was when i query using this handler, it will include
> > all the
> >     dynamic fields with the prefix of "CC_COMP_NAME_" however, that is not
> >     happening and I'm getting 0 hits.  But when I use the full field name,
> > such
> >     as CC_COMP_NAME_1 or CC_COMP_NAME_2, that works so I know my data is
> >     indexed, it is just that Solr not paying attention to the dynamic field
> >     syntax in "qf".
> >
> >     I don't want to keep a list of those dynamic fields and pass them to my
> >     handler, but if I must, than I must.  If so, how can I get the list of
> >     those dynamic fields from Solr so that I don't have to maintain and
> > sync-up
> >     the list myself.
> >
> >     Thanks
> >
> >     Steven
> >
> >