[jira] Created: (SOLR-43) query parameter overhaul

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (SOLR-43) query parameter overhaul

Chris Mattmann (Jira)
query parameter overhaul
------------------------

                 Key: SOLR-43
                 URL: http://issues.apache.org/jira/browse/SOLR-43
             Project: Solr
          Issue Type: New Feature
            Reporter: Yonik Seeley
         Assigned To: Yonik Seeley


Goals:
- per field parameters that fall back to global values
- defaults in solrconfig.xml per request handler, overridable per

This is desirable for highlighting additions: http://issues.apache.org/jira/browse/SOLR-37 

last email thread: http://www.nabble.com/parameter-defaults-and-config-tf2020863.html#a5556298

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-43) query parameter overhaul

Chris Mattmann (Jira)
     [ http://issues.apache.org/jira/browse/SOLR-43?page=all ]

Yonik Seeley updated SOLR-43:
-----------------------------

    Attachment: solrparams.patch

prototype implementation for feedback/discussion:
- changed standard request handler, but not plugin utils it calls
- untested for new functionallity, but all current tests pass


> query parameter overhaul
> ------------------------
>
>                 Key: SOLR-43
>                 URL: http://issues.apache.org/jira/browse/SOLR-43
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Yonik Seeley
>         Assigned To: Yonik Seeley
>         Attachments: solrparams.patch
>
>
> Goals:
> - per field parameters that fall back to global values
> - defaults in solrconfig.xml per request handler, overridable per
> This is desirable for highlighting additions: http://issues.apache.org/jira/browse/SOLR-37 
> last email thread: http://www.nabble.com/parameter-defaults-and-config-tf2020863.html#a5556298

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Work started: (SOLR-43) query parameter overhaul

Chris Mattmann (Jira)
In reply to this post by Chris Mattmann (Jira)
     [ http://issues.apache.org/jira/browse/SOLR-43?page=all ]

Work on SOLR-43 started by Yonik Seeley.

> query parameter overhaul
> ------------------------
>
>                 Key: SOLR-43
>                 URL: http://issues.apache.org/jira/browse/SOLR-43
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Yonik Seeley
>         Assigned To: Yonik Seeley
>         Attachments: solrparams.patch
>
>
> Goals:
> - per field parameters that fall back to global values
> - defaults in solrconfig.xml per request handler, overridable per
> This is desirable for highlighting additions: http://issues.apache.org/jira/browse/SOLR-37 
> last email thread: http://www.nabble.com/parameter-defaults-and-config-tf2020863.html#a5556298

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-43) query parameter overhaul

Chris Mattmann (Jira)
In reply to this post by Chris Mattmann (Jira)
     [ http://issues.apache.org/jira/browse/SOLR-43?page=all ]


    Attachment: solrparams.patch

> query parameter overhaul
> ------------------------
>
>                 Key: SOLR-43
>                 URL: http://issues.apache.org/jira/browse/SOLR-43
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Yonik Seeley
>         Assigned To: Yonik Seeley
>         Attachments: solrparams.patch, solrparams.patch
>
>
> Goals:
> - per field parameters that fall back to global values
> - defaults in solrconfig.xml per request handler, overridable per
> This is desirable for highlighting additions: http://issues.apache.org/jira/browse/SOLR-37 
> last email thread: http://www.nabble.com/parameter-defaults-and-config-tf2020863.html#a5556298

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-43) query parameter overhaul

Chris Mattmann (Jira)
In reply to this post by Chris Mattmann (Jira)
    [ http://issues.apache.org/jira/browse/SOLR-43?page=comments#action_12429821 ]
           
Yonik Seeley commented on SOLR-43:
----------------------------------

Attached refresh.

Unless someone thinks this is going off in the wrong direction, I'll commit this soon so others can work on it also.

> query parameter overhaul
> ------------------------
>
>                 Key: SOLR-43
>                 URL: http://issues.apache.org/jira/browse/SOLR-43
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Yonik Seeley
>         Assigned To: Yonik Seeley
>         Attachments: solrparams.patch, solrparams.patch
>
>
> Goals:
> - per field parameters that fall back to global values
> - defaults in solrconfig.xml per request handler, overridable per
> This is desirable for highlighting additions: http://issues.apache.org/jira/browse/SOLR-37 
> last email thread: http://www.nabble.com/parameter-defaults-and-config-tf2020863.html#a5556298

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Updated: (SOLR-43) query parameter overhaul

Yonik Seeley-2
In reply to this post by Chris Mattmann (Jira)
On 8/22/06, Anonymous (JIRA) <[hidden email]> wrote:
>      [ http://issues.apache.org/jira/browse/SOLR-43?page=all ]
>
>     Attachment: solrparams.patch

Anonymous, huh?  I would have figured one would need to be logged in
to attach something...

Anyway, for the record, it was me :-)

-Yonik
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-43) query parameter overhaul

Chris Mattmann (Jira)
In reply to this post by Chris Mattmann (Jira)
    [ http://issues.apache.org/jira/browse/SOLR-43?page=comments#action_12429863 ]
           
Mike Klaas commented on SOLR-43:
--------------------------------

Looks good!  I especially like how trivial it is to create arbitrarily-nested parameter chains.  One thing I believe that was lost in this patch is static (source-level) defaults for parameters.  Presumably these would be defined using another level of defaul parameters which is a static member of CommonParams or somesuch?

Also, should SolrParams.parseBool() perhaps do a case-insensitive test?

> query parameter overhaul
> ------------------------
>
>                 Key: SOLR-43
>                 URL: http://issues.apache.org/jira/browse/SOLR-43
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Yonik Seeley
>         Assigned To: Yonik Seeley
>         Attachments: solrparams.patch, solrparams.patch
>
>
> Goals:
> - per field parameters that fall back to global values
> - defaults in solrconfig.xml per request handler, overridable per
> This is desirable for highlighting additions: http://issues.apache.org/jira/browse/SOLR-37 
> last email thread: http://www.nabble.com/parameter-defaults-and-config-tf2020863.html#a5556298

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (SOLR-43) query parameter overhaul

Chris Hostetter-3
In reply to this post by Chris Mattmann (Jira)

: Attached refresh.

I've been too busy the past two days to play with this -- or even read
it carefully, but here's some impressions as i skim it...


1) This looks like an idiom in the making ... it might be worth
while to go ahead and refactor into a static utility now
(ie: "SolrParams p = SolrParams.wrapDefaults(req,defaults)") ...

+      SolrParams p = req.getParams();
+      if (defaults != null) {
+        p = new DefaultSolrParams(p,defaults);
+        // set params so they will be visible to other components such as the response writer
+        req.setParams(p);
+      }


2) ...

+      // TODO: We have some constants in SolrQueryRequestBase, and some in CommonParams...

...i vote for pulling them out of SolrQueryRequestBase ... i would go so
far as to recommend deprecating the named accessors like getQueryString,
getLimit, and getStart -- the only params that should be treated special
are "qt" and "wt".  As for where they should live, ... I'm thinking
CommonParams is headed the way of the Dodo, SolrParams seems like the
right place to me.


3) in this method, would it be better to let getOriginalParams() return
null? .. as is information (the lack of params when the request was
constructed) can be lost if setParams is called more then once...

+  public void setParams(SolrParams params) {
+    if (this.origParams==null) this.origParams=params;
+    this.params = params;
+  }






-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (SOLR-43) query parameter overhaul

Yonik Seeley
In reply to this post by Chris Mattmann (Jira)
Mike Klaas commented on SOLR-43:
>  One thing I believe that was lost in this patch is static (source-level) defaults for parameters.  Presumably these would be defined using another level of defaul parameters which is a static member of CommonParams or somesuch?

I assume you mean the default for things like "start" or "fl" if no
handler defaults are defined?  That would be one way to handle it...
but most defaults are null anyway I think.  And it's not too bad just
having some defaults hardcoded like params.getInt(CommonParams.START,
0);

If we did this, one might want to merge the global defaults with the
handler defaults to eliminate the additional level of lookup.

> Also, should SolrParams.parseBool() perhaps do a case-insensitive test?

I'm not opposed... is it needed though?

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (SOLR-43) query parameter overhaul

Yonik Seeley-2
In reply to this post by Chris Hostetter-3
On 8/23/06, Chris Hostetter <[hidden email]> wrote:
> +      // TODO: We have some constants in SolrQueryRequestBase, and some in CommonParams...
>
> ...i vote for pulling them out of SolrQueryRequestBase ... i would go so
> far as to recommend deprecating the named accessors like getQueryString,
> getLimit, and getStart -- the only params that should be treated special
> are "qt" and "wt".  As for where they should live, ... I'm thinking
> CommonParams is headed the way of the Dodo, SolrParams seems like the
> right place to me.

I think SolrParams is fine, I'll move them there unless someone has a
better idea.

> 3) in this method, would it be better to let getOriginalParams() return
> null? .. as is information (the lack of params when the request was
> constructed) can be lost if setParams is called more then once...
>
> +  public void setParams(SolrParams params) {
> +    if (this.origParams==null) this.origParams=params;
> +    this.params = params;
> +  }

The original lack of params wasn't meaningful before (see
LocalSolrQuery constructor which called the superclass constructor
first (as it must) and then created the parameters and called
setParams()).  I can refactor out the parameter construction into a
static method so it can be passed to the superclass constructor
though.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (SOLR-43) query parameter overhaul

Yonik Seeley-2
JIRA didn't send my comments to the list for some reason (or I just
never received it).  I'll cc here:

Committed current version.

Left to do off the top of my head:
 - deprecate methods dealing with params in PluginUtils
 - change use of deprecated methods (including dismax handler)
 - dismax handler: were to get defaults from solrconfig.xml... the
base level, or "defaults". If the latter, provide some backward compat
for existing configs?

Highlighter stuff:
 - allow specification of markup
 - allow fragsize per-field
 - keep in mind recent highlighter work going on in Lucene... we
should try and specify what instead of how (not use exact class names,
etc)
 - start using "hl" namespace for highlighter params... this is just a
convention to help clarify the semantics of a parameter at a glance.
   - for consistency, should "highlight" => "hl", "highlightFields" =>
"hl.fields" or "hl.fl", "maxSnippets" => "hl.snippets"?
    Normally backward compatibility is very important for the external
interfaces, *but* things will change while a feature is in
development... every commit does not constitute a release. Is
highlighting new enough that we can change these parameters? Is anyone
using these parameters in production where it would be a burden if we
changed these?

Examples of potential highlighter param names:
hl=true
hl.fl=name,title,body
hl.snippets=4
hl.fragsize=100
hl.formatter=simple
hl.simple.pre=<em>
hl.simple.post=</em>

And per field params:
f.title.hl.fragsize=0 // overrides fragsize only for field 'title'
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-43) query parameter overhaul

Chris Mattmann (Jira)
In reply to this post by Chris Mattmann (Jira)
    [ http://issues.apache.org/jira/browse/SOLR-43?page=comments#action_12430086 ]
           
Yonik Seeley commented on SOLR-43:
----------------------------------

Committed current version.

Left to do off the top of my head:
 - deprecate methods dealing with params in PluginUtils
 - change use of deprecated methods (including dismax handler)
 - dismax handler: were to get defaults from solrconfig.xml... the base level, or "defaults".  If the latter, provide some backward compat for existing configs?

Highlighter stuff:
 - allow specification of markup
 - allow fragsize per-field
 - keep in mind recent highlighter work going on in Lucene... we should try and specify what instead of how (not use exact class names, etc)
 - start using "hl" namespace for highlighter params... this is just a convention to help clarify the semantics of a parameter at a glance.
   - for consistency, should "highlight" => "hl", "highlightFields" => "hl.fields" or "hl.fl", "maxSnippets" => "hl.snippets"?
    Normally backward compatibility is very important for the external interfaces, *but* things will change while a feature is in development... every commit does not constitute a release.  Is highlighting new enough that we can change these parameters?  Is anyone using these parameters in production where it would be a burden if we changed these?

Examples of potential highlighter param names:
hl=true
hl.fl=name,title,body
hl.snippets=4
hl.fragsize=100
hl.formatter=simple
hl.simple.pre=<em>  
hl.simple.post=</em>

And per field params:
f.title.hl.fragsize=0  // overrides fragsize only for field 'title'



> query parameter overhaul
> ------------------------
>
>                 Key: SOLR-43
>                 URL: http://issues.apache.org/jira/browse/SOLR-43
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Yonik Seeley
>         Assigned To: Yonik Seeley
>         Attachments: solrparams.patch, solrparams.patch
>
>
> Goals:
> - per field parameters that fall back to global values
> - defaults in solrconfig.xml per request handler, overridable per
> This is desirable for highlighting additions: http://issues.apache.org/jira/browse/SOLR-37 
> last email thread: http://www.nabble.com/parameter-defaults-and-config-tf2020863.html#a5556298

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (SOLR-43) query parameter overhaul

Andrew May
In reply to this post by Yonik Seeley-2
Yonik Seeley wrote:

> Highlighter stuff:
> - allow specification of markup
> - allow fragsize per-field
> - keep in mind recent highlighter work going on in Lucene... we
> should try and specify what instead of how (not use exact class names,
> etc)
> - start using "hl" namespace for highlighter params... this is just a
> convention to help clarify the semantics of a parameter at a glance.
>   - for consistency, should "highlight" => "hl", "highlightFields" =>
> "hl.fields" or "hl.fl", "maxSnippets" => "hl.snippets"?
>    Normally backward compatibility is very important for the external
> interfaces, *but* things will change while a feature is in
> development... every commit does not constitute a release. Is
> highlighting new enough that we can change these parameters? Is anyone
> using these parameters in production where it would be a burden if we
> changed these?
>

I think the introduction of a bunch of new highlighter parameters is as good a time as any
to break backwards compatibility, even if we're keeping the same default behaviour - but
we're not using Solr in production yet, so it's easy for me to say that.

> Examples of potential highlighter param names:
> hl=true
> hl.fl=name,title,body
> hl.snippets=4
> hl.fragsize=100
> hl.formatter=simple
> hl.simple.pre=<em>
> hl.simple.post=</em>
>
> And per field params:
> f.title.hl.fragsize=0 // overrides fragsize only for field 'title'

These all sound good.

I was going to try and rewrite my highlighting patch to use the new facilities. I can also
try and update SolrPluginUtils at the same time, as I'll need to make changes to that as
well.

I'm hoping that the lucene highlighter can be updated soon to pick up the fix for
http://issues.apache.org/jira/browse/LUCENE-645 (although it sounds like there might be
further changes ahead). I've currently got a bunch of code that tries to handle this
wandering punctuation in the highlights.

-Andrew
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (SOLR-43) query parameter overhaul

Mike Klaas
On 8/24/06, Andrew May <[hidden email]> wrote:

> I think the introduction of a bunch of new highlighter parameters is as good a time as any
> to break backwards compatibility, even if we're keeping the same default behaviour - but
> we're not using Solr in production yet, so it's easy for me to say that.

We're also not using solr in production yet, so switching is easy.
Highlighting, I think, should remain an experimental feature subject
to modification until at least the lucene highlighting approach has
stabilized.

> I was going to try and rewrite my highlighting patch to use the new facilities. I can also
> try and update SolrPluginUtils at the same time, as I'll need to make changes to that as
> well.

I had started modifying the patch accordingly but hadn't got past the
point of making superficial changes, so I'd be happy to review yours.

-Mike
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (SOLR-43) query parameter overhaul

Andrew May
Mike Klaas wrote:
>> I was going to try and rewrite my highlighting patch to use the new
>> facilities. I can also
>> try and update SolrPluginUtils at the same time, as I'll need to make
>> changes to that as
>> well.
>
> I had started modifying the patch accordingly but hadn't got past the
> point of making superficial changes, so I'd be happy to review yours.
>

The updated patch is now attached to http://issues.apache.org/jira/browse/SOLR-37 - seems
like emails from JIRA are taking a while at the moment.

-Andrew
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-43) query parameter overhaul

Chris Mattmann (Jira)
In reply to this post by Chris Mattmann (Jira)
     [ http://issues.apache.org/jira/browse/SOLR-43?page=all ]

Hoss Man updated SOLR-43:
-------------------------

    Attachment: dismax-solrparams.patch

Overhaul of DisMax handler to use SolrParams, and deprecated all of the SolrPluginUtil.get*Param methods.

Backwards compatible support was added to treat all init params as defaults if no init param named "defaults" was specified.

If no one spots any problems with this patch I'd like to commit it tomorow (Wed).

> query parameter overhaul
> ------------------------
>
>                 Key: SOLR-43
>                 URL: http://issues.apache.org/jira/browse/SOLR-43
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Yonik Seeley
>         Assigned To: Yonik Seeley
>         Attachments: dismax-solrparams.patch, solrparams.patch, solrparams.patch
>
>
> Goals:
> - per field parameters that fall back to global values
> - defaults in solrconfig.xml per request handler, overridable per
> This is desirable for highlighting additions: http://issues.apache.org/jira/browse/SOLR-37 
> last email thread: http://www.nabble.com/parameter-defaults-and-config-tf2020863.html#a5556298

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-43) query parameter overhaul

Chris Mattmann (Jira)
In reply to this post by Chris Mattmann (Jira)
    [ http://issues.apache.org/jira/browse/SOLR-43?page=comments#action_12432727 ]
           
Mike Klaas commented on SOLR-43:
--------------------------------

Looks fine after a cursory examination.

> query parameter overhaul
> ------------------------
>
>                 Key: SOLR-43
>                 URL: http://issues.apache.org/jira/browse/SOLR-43
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Yonik Seeley
>         Assigned To: Yonik Seeley
>         Attachments: dismax-solrparams.patch, solrparams.patch, solrparams.patch
>
>
> Goals:
> - per field parameters that fall back to global values
> - defaults in solrconfig.xml per request handler, overridable per
> This is desirable for highlighting additions: http://issues.apache.org/jira/browse/SOLR-37 
> last email thread: http://www.nabble.com/parameter-defaults-and-config-tf2020863.html#a5556298

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-43) query parameter overhaul

Chris Mattmann (Jira)
In reply to this post by Chris Mattmann (Jira)
    [ http://issues.apache.org/jira/browse/SOLR-43?page=comments#action_12432946 ]
           
Hoss Man commented on SOLR-43:
------------------------------

commited dismax-solrparams.patch

Still todo:  Completely remove refrences to deprecated methods (ie: req.getParam should be req.getParams().get) from within the core code base.

 

> query parameter overhaul
> ------------------------
>
>                 Key: SOLR-43
>                 URL: http://issues.apache.org/jira/browse/SOLR-43
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Yonik Seeley
>         Assigned To: Yonik Seeley
>         Attachments: dismax-solrparams.patch, solrparams.patch, solrparams.patch
>
>
> Goals:
> - per field parameters that fall back to global values
> - defaults in solrconfig.xml per request handler, overridable per
> This is desirable for highlighting additions: http://issues.apache.org/jira/browse/SOLR-37 
> last email thread: http://www.nabble.com/parameter-defaults-and-config-tf2020863.html#a5556298

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (SOLR-43) query parameter overhaul

Chris Mattmann (Jira)
In reply to this post by Chris Mattmann (Jira)
     [ http://issues.apache.org/jira/browse/SOLR-43?page=all ]

Yonik Seeley resolved SOLR-43.
------------------------------

    Resolution: Fixed

closing... the removal of deprecated methods should probably be more tied to releases.

> query parameter overhaul
> ------------------------
>
>                 Key: SOLR-43
>                 URL: http://issues.apache.org/jira/browse/SOLR-43
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Yonik Seeley
>         Assigned To: Yonik Seeley
>         Attachments: dismax-solrparams.patch, solrparams.patch, solrparams.patch
>
>
> Goals:
> - per field parameters that fall back to global values
> - defaults in solrconfig.xml per request handler, overridable per
> This is desirable for highlighting additions: http://issues.apache.org/jira/browse/SOLR-37 
> last email thread: http://www.nabble.com/parameter-defaults-and-config-tf2020863.html#a5556298

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira