Phrase search error

[Jason]

Hi, all

When I queried a phrase search "test mp3", I got some error below.
I think that the problem is because of WordDelimiterFilter.
In WordDelimiterFilter 'mp3' is splited pos1:mp, pos2:(3, mp3).
In such a case, the positions of subword and catenateword are incremented.
If this is not phrase search or WordDelimiterFilterFactory options just set catenateAll=1, no problems.
But If WordDelimiterFilterFactory options set like below 'My Schema.xml', occured error.

How can I solve this problem?
Give me any idea.

Thanks in advance.
Jason


[Error Message]
==============================================================================
Unknown query type "org.apache.lucene.search.MultiPhraseQuery" found in phrase query string "test mp3"

java.lang.IllegalArgumentException: Unknown query type "org.apache.lucene.search.MultiPhraseQuery" found in phrase query string "test mp3"
        at org.apache.lucene.queryParser.ComplexPhraseQueryParser$ComplexPhraseQuery.rewrite(ComplexPhraseQueryParser.java:300)
        at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:307)
        at org.apache.lucene.search.Query.weight(Query.java:98)
        at org.apache.lucene.search.Searcher.createWeight(Searcher.java:230)
        at org.apache.lucene.search.Searcher.search(Searcher.java:171)
        at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988)
        at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884)
        at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
        at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:341)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:244)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
        at org.mortbay.jetty.Server.handle(Server.java:285)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:821)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
==============================================================================


[My Schema.xml]
==============================================================================
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
     <analyzer type="index">
        <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt" />
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="0" catenateWords="1" catenateNumbers="1" catenateAll="1" splitOnCaseChange="1" />
        <filter class="solr.LowerCaseFilterFactory" />
        <filter class="solr.SnowballPorterWithUnstemFilterFactory" language="English" protected="protwords.txt" />
     </analyzer>
     <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="0" catenateWords="1" catenateNumbers="1" catenateAll="1" splitOnCaseChange="0" />
        <filter class="solr.LowerCaseFilterFactory" />
        <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt" />
     </analyzer>
</fieldType>
==============================================================================

[lboutros]

Hi Jason,

you could add this filter to the end of your analyzer :

http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.PositionFilterFactory

That should solve your problem.

Ludovic.

[Jason]

Hi, Ludovic

That's just what I'm looking for.
You're been a big help.
Thank you so much.

Jason.

[Rob Brown]

Hi,

I'm running Solr on a machine with 16 CPU cores, yet watching "top" shows that java is only apparently using 1 and maxing it out.

Is there anything that can be done to take advantage of more CPU cores?

Solr 3.4 under Tomcat

[root@solr01 ~]# java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.8) (rhel-1.22.1.9.8.el5_6-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)


top - 14:36:18 up 22 days, 21:54,  4 users,  load average: 1.89, 1.24, 1.08
Tasks: 317 total,   1 running, 315 sleeping,   0 stopped,   1 zombie
Cpu0  :  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  : 99.6%us,  0.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu12 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu13 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu14 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu15 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  132088928k total, 23760584k used, 108328344k free,   318228k buffers
Swap: 25920868k total,        0k used, 25920868k free, 18371128k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                            
 4466 tomcat    20   0 31.2g 4.0g 171m S 101.0  3.2   2909:38 java                                                                                                                                                                              
 6495 root      15   0 42416 3892 1740 S  0.4  0.0   9:34.71 openvpn                                                                                                                                                                            
11456 root      16   0 12892 1312  836 R  0.4  0.0   0:00.08 top                                                                                                                                                                                
    1 root      15   0 10368  632  536 S  0.0  0.0   0:04.69 init                                            

[Chris Hostetter-3]

: Subject: Multi CPU Cores
: In-Reply-To: <[hidden email]>
: References: <[hidden email]>

http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists

When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email.  Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention.   It makes following discussions in the mailing list archives
particularly difficult.



-Hoss

[Li Li]

for indexing, your can make use of multi cores easily by call
IndexWriter.addDocument with multi-threads
as far as I know, for searching, if there is only one request, you can't
make good use of cpus.

On Sat, Oct 15, 2011 at 9:37 PM, Rob Brown <[hidden email]> wrote:

> Hi,
>
> I'm running Solr on a machine with 16 CPU cores, yet watching "top" shows
> that java is only apparently using 1 and maxing it out.
>
> Is there anything that can be done to take advantage of more CPU cores?
>
> Solr 3.4 under Tomcat
>
> [root@solr01 ~]# java -version
> java version "1.6.0_20"
> OpenJDK Runtime Environment (IcedTea6 1.9.8) (rhel-1.22.1.9.8.el5_6-x86_64)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
>
>
> top - 14:36:18 up 22 days, 21:54,  4 users,  load average: 1.89, 1.24, 1.08
> Tasks: 317 total,   1 running, 315 sleeping,   0 stopped,   1 zombie
> Cpu0  :  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu6  : 99.6%us,  0.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu8  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu9  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu10 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu11 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu12 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu13 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu14 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Cpu15 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Mem:  132088928k total, 23760584k used, 108328344k free,   318228k buffers
> Swap: 25920868k total,        0k used, 25920868k free, 18371128k cached
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  4466 tomcat    20   0 31.2g 4.0g 171m S 101.0  3.2   2909:38 java
>  6495 root      15   0 42416 3892 1740 S  0.4  0.0   9:34.71 openvpn
> 11456 root      16   0 12892 1312  836 R  0.4  0.0   0:00.08 top
>    1 root      15   0 10368  632  536 S  0.0  0.0   0:04.69 init
>
>