Double Solr Installation on Single Tomcat (or Double Index)

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Got it working! And some questions

Yonik Seeley-2
On 9/9/06, Michael Imbeault <[hidden email]> wrote:
> The main problem was that addIndex was sending 1 doc at a time to solr;
> it would cause a problem after a few thousand docs because i was running
> out of resources.

Sending one doc at a time should be fine... you shouldn't run out of
resources.
There must be a bug somewhere...

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Got it working! And some questions

Erik Hatcher
In reply to this post by Michael Imbeault

On Sep 10, 2006, at 10:47 PM, Michael Imbeault wrote:
>  I'm still a little disappointed that I can't change the OR/AND  
> parsing by just changing some parameter (like I can do for the  
> number of results returned, for example); adding a OR between each  
> word in the text i want to compare sounds suboptimal, but i'll  
> probably do it that way; its a very minor nitpick, solr is awesome,  
> as I said before.

I'm the one that added support for controlling the default operator  
of Solr's query parser, and I hadn't considered the use case of  
controlling that setting from a request parameter.  It should be easy  
enough to add.  I'll take a look at adding that support and commit it  
once I have it working.

What parameter name should be used for this?    do=[AND|OR] (for  
default operator)?  We have df for default field.

        Erik

Reply | Threaded
Open this post in threaded view
|

Re: Got it working! And some questions

Michael Imbeault
Hello Erik,

Thanks for add that feature! "do" is fine with me, if "op" is already
used (not sure about this one).

Erik Hatcher wrote:

>
> On Sep 10, 2006, at 10:47 PM, Michael Imbeault wrote:
>>  I'm still a little disappointed that I can't change the OR/AND
>> parsing by just changing some parameter (like I can do for the number
>> of results returned, for example); adding a OR between each word in
>> the text i want to compare sounds suboptimal, but i'll probably do it
>> that way; its a very minor nitpick, solr is awesome, as I said before.
>
> I'm the one that added support for controlling the default operator of
> Solr's query parser, and I hadn't considered the use case of
> controlling that setting from a request parameter.  It should be easy
> enough to add.  I'll take a look at adding that support and commit it
> once I have it working.
>
> What parameter name should be used for this?    do=[AND|OR] (for
> default operator)?  We have df for default field.
>
>     Erik
>
--
Michael Imbeault
CHUL Research Center (CHUQ)
2705 boul. Laurier
Ste-Foy, QC, Canada, G1V 4G2
Tel: (418) 654-2705, Fax: (418) 654-2212

Reply | Threaded
Open this post in threaded view
|

Re: Got it working! And some questions

Yonik Seeley-2
In reply to this post by Erik Hatcher
On 9/11/06, Erik Hatcher <[hidden email]> wrote:

>
> On Sep 10, 2006, at 10:47 PM, Michael Imbeault wrote:
> >  I'm still a little disappointed that I can't change the OR/AND
> > parsing by just changing some parameter (like I can do for the
> > number of results returned, for example); adding a OR between each
> > word in the text i want to compare sounds suboptimal, but i'll
> > probably do it that way; its a very minor nitpick, solr is awesome,
> > as I said before.
>
> I'm the one that added support for controlling the default operator
> of Solr's query parser, and I hadn't considered the use case of
> controlling that setting from a request parameter.  It should be easy
> enough to add.  I'll take a look at adding that support and commit it
> once I have it working.
>
> What parameter name should be used for this?    do=[AND|OR] (for
> default operator)?  We have df for default field.

Maybe something like q.op or q.oper if it *only* applies to q.  Which
begs the question... what *does* it apply to?  At first blush, it
doesn't seem like it should apply to other queries like fq, facet
queries, and esp queries defined in solrconfig.xml.  I think that
would be very surprising.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Got it working! And some questions

Chris Hostetter-3

: Maybe something like q.op or q.oper if it *only* applies to q.  Which
: begs the question... what *does* it apply to?  At first blush, it
: doesn't seem like it should apply to other queries like fq, facet
: queries, and esp queries defined in solrconfig.xml.  I think that
: would be very surprising.

agreed not the comment i put into SolrPluginUtils.parseFilterQueries when
i add fq support to StandardRequestHandler...

    /* Ignore SolrParams.DF - could have init param FQs assuming the
     * schema default with query param DF intented to only affect Q.
     * If user doesn't want schema default, they should be explicit in the FQ.
     */

... i would think a "do" or "op" or "q.op" param should *definitely* only
influence the "q" param.





-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Got it working! And some questions

Erik Hatcher
In reply to this post by Yonik Seeley-2

On Sep 11, 2006, at 2:52 PM, Yonik Seeley wrote:

> On 9/11/06, Erik Hatcher <[hidden email]> wrote:
>>
>> On Sep 10, 2006, at 10:47 PM, Michael Imbeault wrote:
>> >  I'm still a little disappointed that I can't change the OR/AND
>> > parsing by just changing some parameter (like I can do for the
>> > number of results returned, for example); adding a OR between each
>> > word in the text i want to compare sounds suboptimal, but i'll
>> > probably do it that way; its a very minor nitpick, solr is awesome,
>> > as I said before.
>>
>> I'm the one that added support for controlling the default operator
>> of Solr's query parser, and I hadn't considered the use case of
>> controlling that setting from a request parameter.  It should be easy
>> enough to add.  I'll take a look at adding that support and commit it
>> once I have it working.
>>
>> What parameter name should be used for this?    do=[AND|OR] (for
>> default operator)?  We have df for default field.
>
> Maybe something like q.op or q.oper if it *only* applies to q.  Which
> begs the question... what *does* it apply to?  At first blush, it
> doesn't seem like it should apply to other queries like fq, facet
> queries, and esp queries defined in solrconfig.xml.  I think that
> would be very surprising.

I've implemented the ability to override the default operator with  
q.op=AND|OR.  The patch is pasted below for your review.

The one thing I don't like is that QueryParsing.parseQuery(String qs,  
String defaultField, SolrParams params, IndexSchema schema) is a bit  
redundant in that it takes defaultField which can also be gleaned  
from params, but StandardRequestHandler uses "df" for highlighting also.

I'm happy to commit this if there are no objections or suggestions  
for improvement (and of course update the wiki documentation for the  
parameters).

        Erik



Index: src/java/org/apache/solr/search/SolrQueryParser.java
===================================================================
--- src/java/org/apache/solr/search/SolrQueryParser.java (revision  
442689)
+++ src/java/org/apache/solr/search/SolrQueryParser.java (working copy)
@@ -34,10 +34,14 @@
    protected final IndexSchema schema;
    public SolrQueryParser(IndexSchema schema, String defaultField) {
+    this(schema, defaultField, QueryParser.Operator.OR);
+  }
+
+  public SolrQueryParser(IndexSchema schema, String defaultField,  
QueryParser.Operator defaultOperator) {
      super(defaultField == null ? schema.getDefaultSearchFieldName
() : defaultField, schema.getQueryAnalyzer());
      this.schema = schema;
      setLowercaseExpandedTerms(false);
-    setDefaultOperator("AND".equals
(schema.getQueryParserDefaultOperator()) ? QueryParser.Operator.AND :  
QueryParser.Operator.OR);
+    setDefaultOperator(defaultOperator);
    }
    protected Query getFieldQuery(String field, String queryText)  
throws ParseException {
Index: src/java/org/apache/solr/search/QueryParsing.java
===================================================================
--- src/java/org/apache/solr/search/QueryParsing.java (revision 442689)
+++ src/java/org/apache/solr/search/QueryParsing.java (working copy)
@@ -19,6 +19,7 @@
import org.apache.lucene.search.*;
import org.apache.solr.search.function.*;
import org.apache.lucene.queryParser.ParseException;
+import org.apache.lucene.queryParser.QueryParser;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.Term;
import org.apache.solr.core.SolrCore;
@@ -26,6 +27,7 @@
import org.apache.solr.schema.IndexSchema;
import org.apache.solr.schema.SchemaField;
import org.apache.solr.schema.FieldType;
+import org.apache.solr.request.SolrParams;
import java.util.ArrayList;
import java.util.regex.Pattern;
@@ -37,6 +39,7 @@
   * @version $Id$
   */
public class QueryParsing {
+  public static final String OP = "q.op";
    public static Query parseQuery(String qs, IndexSchema schema) {
      return parseQuery(qs, null, schema);
@@ -58,8 +61,24 @@
      }
    }
+  public static Query parseQuery(String qs, String defaultField,  
SolrParams params, IndexSchema schema) {
+    try {
+      String opParam = params.get(OP,  
schema.getQueryParserDefaultOperator());
+      QueryParser.Operator defaultOperator = "AND".equals(opParam) ?  
QueryParser.Operator.AND : QueryParser.Operator.OR;
+      Query query = new SolrQueryParser(schema, defaultField,  
defaultOperator).parse(qs);
+      if (SolrCore.log.isLoggable(Level.FINEST)) {
+        SolrCore.log.finest("After QueryParser:" + query);
+      }
+      return query;
+
+    } catch (ParseException e) {
+      SolrCore.log(e);
+      throw new SolrException(400,"Error parsing Lucene query",e);
+    }
+  }
+
    /***
     * SortSpec encapsulates a Lucene Sort and a count of the number  
of documents
     * to return.
Index: src/java/org/apache/solr/request/StandardRequestHandler.java
===================================================================
--- src/java/org/apache/solr/request/StandardRequestHandler.java
(revision 442689)
+++ src/java/org/apache/solr/request/StandardRequestHandler.java
(working copy)
@@ -94,7 +94,7 @@
        List<String> commands = StrUtils.splitSmart(sreq,';');
        String qs = commands.size() >= 1 ? commands.get(0) : "";
-      Query query = QueryParsing.parseQuery(qs, defaultField,  
req.getSchema());
+      Query query = QueryParsing.parseQuery(qs, defaultField, p,  
req.getSchema());
        // If the first non-query, non-filter command is a simple  
sort on an indexed field, then
        // we can use the Lucene sort ability.

Reply | Threaded
Open this post in threaded view
|

Re: Got it working! And some questions

Chris Hostetter-3

: I've implemented the ability to override the default operator with
: q.op=AND|OR.  The patch is pasted below for your review.

if i'm reading that right, one subtlety is that "new
SolrQueryParser(schema,field)" no longer pas attention to
schema.getQueryParserDefaultOperator() -- that only only becomes
applicable when using QueryParsing.parseQuery

...i am very okay with this change, i wasn't really a fan of the fact that
the SolrQueryParser pulled that info out of the IndexSchema in it's
constructor previously, i just wanted to point out that this patch would
change that.

Perhaps the constructor for SolrQueryParser shouldn't be aware of the op
at all (either from the schema or from the SolrParams) -- and setting it
should be left to QueryParsing.parseQuery (or some other utility in the
QueryParsing class) ... personally i'm a fan of leaving SolrQueryParser as
much like QueryParser as possible -- with the only real change being the
knowledege of hte individual field formats.


: Index: src/java/org/apache/solr/search/SolrQueryParser.java
: ===================================================================
: --- src/java/org/apache/solr/search/SolrQueryParser.java (revision
: 442689)
: +++ src/java/org/apache/solr/search/SolrQueryParser.java (working copy)
: @@ -34,10 +34,14 @@
:     protected final IndexSchema schema;
:     public SolrQueryParser(IndexSchema schema, String defaultField) {
: +    this(schema, defaultField, QueryParser.Operator.OR);
: +  }
: +
: +  public SolrQueryParser(IndexSchema schema, String defaultField,
: QueryParser.Operator defaultOperator) {
:       super(defaultField == null ? schema.getDefaultSearchFieldName
: () : defaultField, schema.getQueryAnalyzer());
:       this.schema = schema;
:       setLowercaseExpandedTerms(false);
: -    setDefaultOperator("AND".equals
: (schema.getQueryParserDefaultOperator()) ? QueryParser.Operator.AND :
: QueryParser.Operator.OR);
: +    setDefaultOperator(defaultOperator);
:     }
:     protected Query getFieldQuery(String field, String queryText)
: throws ParseException {
: Index: src/java/org/apache/solr/search/QueryParsing.java
: ===================================================================
: --- src/java/org/apache/solr/search/QueryParsing.java (revision 442689)
: +++ src/java/org/apache/solr/search/QueryParsing.java (working copy)
: @@ -19,6 +19,7 @@
: import org.apache.lucene.search.*;
: import org.apache.solr.search.function.*;
: import org.apache.lucene.queryParser.ParseException;
: +import org.apache.lucene.queryParser.QueryParser;
: import org.apache.lucene.document.Field;
: import org.apache.lucene.index.Term;
: import org.apache.solr.core.SolrCore;
: @@ -26,6 +27,7 @@
: import org.apache.solr.schema.IndexSchema;
: import org.apache.solr.schema.SchemaField;
: import org.apache.solr.schema.FieldType;
: +import org.apache.solr.request.SolrParams;
: import java.util.ArrayList;
: import java.util.regex.Pattern;
: @@ -37,6 +39,7 @@
:    * @version $Id$
:    */
: public class QueryParsing {
: +  public static final String OP = "q.op";
:     public static Query parseQuery(String qs, IndexSchema schema) {
:       return parseQuery(qs, null, schema);
: @@ -58,8 +61,24 @@
:       }
:     }
: +  public static Query parseQuery(String qs, String defaultField,
: SolrParams params, IndexSchema schema) {
: +    try {
: +      String opParam = params.get(OP,
: schema.getQueryParserDefaultOperator());
: +      QueryParser.Operator defaultOperator = "AND".equals(opParam) ?
: QueryParser.Operator.AND : QueryParser.Operator.OR;
: +      Query query = new SolrQueryParser(schema, defaultField,
: defaultOperator).parse(qs);
: +      if (SolrCore.log.isLoggable(Level.FINEST)) {
: +        SolrCore.log.finest("After QueryParser:" + query);
: +      }
: +      return query;
: +
: +    } catch (ParseException e) {
: +      SolrCore.log(e);
: +      throw new SolrException(400,"Error parsing Lucene query",e);
: +    }
: +  }
: +
:     /***
:      * SortSpec encapsulates a Lucene Sort and a count of the number
: of documents
:      * to return.
: Index: src/java/org/apache/solr/request/StandardRequestHandler.java
: ===================================================================
: --- src/java/org/apache/solr/request/StandardRequestHandler.java
: (revision 442689)
: +++ src/java/org/apache/solr/request/StandardRequestHandler.java
: (working copy)
: @@ -94,7 +94,7 @@
:         List<String> commands = StrUtils.splitSmart(sreq,';');
:         String qs = commands.size() >= 1 ? commands.get(0) : "";
: -      Query query = QueryParsing.parseQuery(qs, defaultField,
: req.getSchema());
: +      Query query = QueryParsing.parseQuery(qs, defaultField, p,
: req.getSchema());
:         // If the first non-query, non-filter command is a simple
: sort on an indexed field, then
:         // we can use the Lucene sort ability.
:



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Got it working! And some questions

Erik Hatcher

On Sep 12, 2006, at 4:47 PM, Chris Hostetter wrote:

> : I've implemented the ability to override the default operator with
> : q.op=AND|OR.  The patch is pasted below for your review.
>
> if i'm reading that right, one subtlety is that "new
> SolrQueryParser(schema,field)" no longer pas attention to
> schema.getQueryParserDefaultOperator() -- that only only becomes
> applicable when using QueryParsing.parseQuery
>
> ...i am very okay with this change, i wasn't really a fan of the  
> fact that
> the SolrQueryParser pulled that info out of the IndexSchema in it's
> constructor previously, i just wanted to point out that this patch  
> would
> change that.
>
> Perhaps the constructor for SolrQueryParser shouldn't be aware of  
> the op
> at all (either from the schema or from the SolrParams) -- and  
> setting it
> should be left to QueryParsing.parseQuery (or some other utility in  
> the
> QueryParsing class) ... personally i'm a fan of leaving  
> SolrQueryParser as
> much like QueryParser as possible -- with the only real change  
> being the
> knowledege of hte individual field formats.

I've reworked it based on your feedback.  The patch is pasted below.

SolrQueryParser now knows nothing about the default operator, it is  
set from QueryParsing.parseQuery() when passed a SolrParams.

QueryParsing.parseQuery() methods could be simplified, perhaps even  
into a single method, that took a query expression and a  
SolrQueryRequest, where it can get the SolrParams and  IndexSchema.  
It could even get the "q" parameter from there, but there is code  
that passes expressions that don't come from "q".  Maybe we could  
have two parseQuery() methods:  parseQuery(String expression,  
SolrQueryRequest req) and parseQuery(SolrQueryRequest req), and for  
the latter the "q" parameter is pulled from the request and used as  
the expression.

As it is, the patch below works fine and I'm happy to commit it, but  
am happy to rework this sort of thing to get it as clean as others like.

        Erik


Index: src/java/org/apache/solr/search/SolrQueryParser.java
===================================================================
--- src/java/org/apache/solr/search/SolrQueryParser.java (revision  
442772)
+++ src/java/org/apache/solr/search/SolrQueryParser.java (working copy)
@@ -37,7 +37,6 @@
      super(defaultField == null ? schema.getDefaultSearchFieldName
() : defaultField, schema.getQueryAnalyzer());
      this.schema = schema;
      setLowercaseExpandedTerms(false);
-    setDefaultOperator("AND".equals
(schema.getQueryParserDefaultOperator()) ? QueryParser.Operator.AND :  
QueryParser.Operator.OR);
    }
    protected Query getFieldQuery(String field, String queryText)  
throws ParseException {
Index: src/java/org/apache/solr/search/QueryParsing.java
===================================================================
--- src/java/org/apache/solr/search/QueryParsing.java (revision 442772)
+++ src/java/org/apache/solr/search/QueryParsing.java (working copy)
@@ -19,6 +19,7 @@
import org.apache.lucene.search.*;
import org.apache.solr.search.function.*;
import org.apache.lucene.queryParser.ParseException;
+import org.apache.lucene.queryParser.QueryParser;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.Term;
import org.apache.solr.core.SolrCore;
@@ -26,6 +27,7 @@
import org.apache.solr.schema.IndexSchema;
import org.apache.solr.schema.SchemaField;
import org.apache.solr.schema.FieldType;
+import org.apache.solr.request.SolrParams;
import java.util.ArrayList;
import java.util.regex.Pattern;
@@ -37,6 +39,7 @@
   * @version $Id$
   */
public class QueryParsing {
+  public static final String OP = "q.op";
    public static Query parseQuery(String qs, IndexSchema schema) {
      return parseQuery(qs, null, schema);
@@ -58,8 +61,26 @@
      }
    }
+  public static Query parseQuery(String qs, String defaultField,  
SolrParams params, IndexSchema schema) {
+    try {
+      String opParam = params.get(OP,  
schema.getQueryParserDefaultOperator());
+      QueryParser.Operator defaultOperator = "AND".equals(opParam) ?  
QueryParser.Operator.AND : QueryParser.Operator.OR;
+      SolrQueryParser parser = new SolrQueryParser(schema,  
defaultField);
+      parser.setDefaultOperator(defaultOperator);
+      Query query = parser.parse(qs);
+      if (SolrCore.log.isLoggable(Level.FINEST)) {
+        SolrCore.log.finest("After QueryParser:" + query);
+      }
+      return query;
+
+    } catch (ParseException e) {
+      SolrCore.log(e);
+      throw new SolrException(400,"Error parsing Lucene query",e);
+    }
+  }
+
    /***
     * SortSpec encapsulates a Lucene Sort and a count of the number  
of documents
     * to return.
Index: src/java/org/apache/solr/request/StandardRequestHandler.java
===================================================================
--- src/java/org/apache/solr/request/StandardRequestHandler.java
(revision 442772)
+++ src/java/org/apache/solr/request/StandardRequestHandler.java
(working copy)
@@ -105,7 +105,7 @@
        List<String> commands = StrUtils.splitSmart(sreq,';');
        String qs = commands.size() >= 1 ? commands.get(0) : "";
-      Query query = QueryParsing.parseQuery(qs, defaultField,  
req.getSchema());
+      Query query = QueryParsing.parseQuery(qs, defaultField, p,  
req.getSchema());
        // If the first non-query, non-filter command is a simple  
sort on an indexed field, then
        // we can use the Lucene sort ability.

Reply | Threaded
Open this post in threaded view
|

Re: Got it working! And some questions

Chris Hostetter-3

: SolrQueryParser now knows nothing about the default operator, it is
: set from QueryParsing.parseQuery() when passed a SolrParams.

i didn't test it, but it looks clean to me.

the only other thing i would do is beaf up the javadocs for
SolrQueryParser (to clarify that IndexSchema is only used for determining
field format) and QueryParsing.parseQuery (to clarify that it *does* use
IndexSearcher to get extra parsing options).

: QueryParsing.parseQuery() methods could be simplified, perhaps even
        ...
: It could even get the "q" parameter from there, but there is code
: that passes expressions that don't come from "q".  Maybe we could

...yeha, it's utility for simple queries regardless of the "primary"
language of a request handler is key.

: have two parseQuery() methods:  parseQuery(String expression,
: SolrQueryRequest req) and parseQuery(SolrQueryRequest req), and for
: the latter the "q" parameter is pulled from the request and used as
: the expression.

That sounds good to me ... but it doesn't seem critical ... clean house as
much as you want, but i don't think anybody else will mind a bit of dust
on the window sills.



-Hoss

12