lucene java OOM while sorting more than one field

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

lucene java OOM while sorting more than one field

GURUPRASAD MS
 Lucene Index contains 2.1 Million records (indexed from 2.1 million records
from sqlserver DB).
Lucene Index file Size 256MB
Lucene version: 2.3
Searching works fine when we sort the results on a single field. However, if
the search results is sorted on more than one field we get Out of Memory
exception.
We restrict the number of search results to 250

Out of Memory exception is quite consistent in 2.3. We recently moved from
2.0 to 2.3.
Version 2.0 also used to give the OOM but not this frequent.


*Code Snippet:*

final QueryParser parser;
StandardAnalyzer analyzer = new StandardAnalyzer();
String sSearchQuery = "MUSIC";
Sort oSort = null;
public static final String G2KEYFIELDS = "G2KEYFIELDS";
oSort = new Sort(new SortField[]{ new SortField (
GroupsConstants.LAST_MODIFIED,SortField.INT,true), new SortField(
GroupsConstants.GROUPNAME) });

parser = new QueryParser(G2KEYFIELDS, analyzer);
final Vector ids = new Vector();
FSDirectory dir = null;
IndexSearcher searcher = null;
try {
dir = FSDirectory.getDirectory(index);
searcher = new IndexSearcher(dir);
Query query = parser.parse(sSearchQuery);
Hits hits = searcher.search(query, oSort);
for (int i = 0; i != hits.length() && i != 250; ++i) {
final Document doc = hits.doc;
Integer oiGroupId=new Integer(doc.getField (GroupsConstants.IDENTITY
).stringValue());
if(!ids.contains(oiGroupId)){ ids.addElement(oiGroupId); }
}
searcher.close();
dir.close();


Thanks in Advance
Reply | Threaded
Open this post in threaded view
|

Re: lucene java OOM while sorting more than one field

Erick Erickson
The first question is always "how much memory are you giving
your JVM?".

A 256M index is pretty small, I wouldn't be surprised if your JVM is using
some vary small default....

Best
Erick

On Wed, Feb 27, 2008 at 6:23 AM, GURUPRASAD MS <[hidden email]>
wrote:

>  Lucene Index contains 2.1 Million records (indexed from 2.1 million
> records
> from sqlserver DB).
> Lucene Index file Size 256MB
> Lucene version: 2.3
> Searching works fine when we sort the results on a single field. However,
> if
> the search results is sorted on more than one field we get Out of Memory
> exception.
> We restrict the number of search results to 250
>
> Out of Memory exception is quite consistent in 2.3. We recently moved from
> 2.0 to 2.3.
> Version 2.0 also used to give the OOM but not this frequent.
>
>
> *Code Snippet:*
>
> final QueryParser parser;
> StandardAnalyzer analyzer = new StandardAnalyzer();
> String sSearchQuery = "MUSIC";
> Sort oSort = null;
> public static final String G2KEYFIELDS = "G2KEYFIELDS";
> oSort = new Sort(new SortField[]{ new SortField (
> GroupsConstants.LAST_MODIFIED,SortField.INT,true), new SortField(
> GroupsConstants.GROUPNAME) });
>
> parser = new QueryParser(G2KEYFIELDS, analyzer);
> final Vector ids = new Vector();
> FSDirectory dir = null;
> IndexSearcher searcher = null;
> try {
> dir = FSDirectory.getDirectory(index);
> searcher = new IndexSearcher(dir);
> Query query = parser.parse(sSearchQuery);
> Hits hits = searcher.search(query, oSort);
> for (int i = 0; i != hits.length() && i != 250; ++i) {
> final Document doc = hits.doc;
> Integer oiGroupId=new Integer(doc.getField (GroupsConstants.IDENTITY
> ).stringValue());
> if(!ids.contains(oiGroupId)){ ids.addElement(oiGroupId); }
> }
> searcher.close();
> dir.close();
>
>
> Thanks in Advance
>