[jira] Created: (LUCENE-1193) OOM exception while sorting on more than 1 field

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-1193) OOM exception while sorting on more than 1 field

Tim Allison (Jira)
OOM exception while sorting on more than 1 field
------------------------------------------------

                 Key: LUCENE-1193
                 URL: https://issues.apache.org/jira/browse/LUCENE-1193
             Project: Lucene - Java
          Issue Type: Bug
          Components: Search
    Affects Versions: 2.3
         Environment: LINUX, JRUN4, APACHE, 3GB RAM
            Reporter: vaidy


Lucene Index contains 2.1 Million records (indexed from 2.1 million records from sqlserver DB).
Lucene Index file Size 256MB
Lucene version: 2.3
Searching works fine when we sort the results on a single field. However, if the search results is sorted on more than one field we get Out of Memory exception.
We restrict the number of search results to 250

Out of Memory exception is quite consistent in 2.3.  We recently moved from 2.0 to 2.3.  
Version 2.0 also used to give the OOM but not this frequent.

{color:red}
*Code Snippet:*

        final QueryParser parser;
        StandardAnalyzer analyzer = new StandardAnalyzer();
        String sSearchQuery = "MUSIC";
        Sort oSort = null;
        public static final String G2KEYFIELDS = "G2KEYFIELDS";
            oSort =  new Sort(new SortField[]{
                                  new SortField (GroupsConstants.LAST_MODIFIED,SortField.INT,true),
                                  new SortField(GroupsConstants.GROUPNAME)
                                });
           
  parser = new QueryParser(G2KEYFIELDS, analyzer);
        final Vector ids = new Vector();
        FSDirectory dir = null;
        IndexSearcher searcher = null;
        try {
            dir = FSDirectory.getDirectory(index);
            searcher = new IndexSearcher(dir);
            Query query = parser.parse(sSearchQuery);
            Hits hits = searcher.search(query, oSort);
            for (int i = 0; i != hits.length() && i != 250; ++i) {
                final Document doc = hits.doc(i);
                Integer oiGroupId=new Integer(doc.getField (GroupsConstants.IDENTITY).stringValue());
                if(!ids.contains(oiGroupId)){
                    ids.addElement(oiGroupId);
                }
            }
            searcher.close();
            dir.close();
{color}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCENE-1193) OOM exception while sorting on more than 1 field

Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Otis Gospodnetic resolved LUCENE-1193.
--------------------------------------

    Resolution: Invalid

> OOM exception while sorting on more than 1 field
> ------------------------------------------------
>
>                 Key: LUCENE-1193
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1193
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 2.3
>         Environment: LINUX, JRUN4, APACHE, 3GB RAM
>            Reporter: vaidy
>
> Lucene Index contains 2.1 Million records (indexed from 2.1 million records from sqlserver DB).
> Lucene Index file Size 256MB
> Lucene version: 2.3
> Searching works fine when we sort the results on a single field. However, if the search results is sorted on more than one field we get Out of Memory exception.
> We restrict the number of search results to 250
> Out of Memory exception is quite consistent in 2.3.  We recently moved from 2.0 to 2.3.  
> Version 2.0 also used to give the OOM but not this frequent.
> {color:red}
> *Code Snippet:*
> final QueryParser parser;
>         StandardAnalyzer analyzer = new StandardAnalyzer();
>         String sSearchQuery = "MUSIC";
>         Sort oSort = null;
> public static final String G2KEYFIELDS = "G2KEYFIELDS";
>             oSort =  new Sort(new SortField[]{
>                                   new SortField (GroupsConstants.LAST_MODIFIED,SortField.INT,true),
>                                   new SortField(GroupsConstants.GROUPNAME)
>                                 });
>            
>   parser = new QueryParser(G2KEYFIELDS, analyzer);
>         final Vector ids = new Vector();
>         FSDirectory dir = null;
>         IndexSearcher searcher = null;
>         try {
>             dir = FSDirectory.getDirectory(index);
>             searcher = new IndexSearcher(dir);
>             Query query = parser.parse(sSearchQuery);
>             Hits hits = searcher.search(query, oSort);
>             for (int i = 0; i != hits.length() && i != 250; ++i) {
>                 final Document doc = hits.doc(i);
>                 Integer oiGroupId=new Integer(doc.getField (GroupsConstants.IDENTITY).stringValue());
>                 if(!ids.contains(oiGroupId)){
>                     ids.addElement(oiGroupId);
>                 }
>             }
>             searcher.close();
>             dir.close();
> {color}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCENE-1193) OOM exception while sorting on more than 1 field

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Otis Gospodnetic resolved LUCENE-1193.
--------------------------------------

    Resolution: Fixed

Sounds like your JVM heap is too small for sorting by 2 fields (causes more data to be rad from Lucene index into memory), that's all.  You can bring this up on java-user@lucene list.

> OOM exception while sorting on more than 1 field
> ------------------------------------------------
>
>                 Key: LUCENE-1193
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1193
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 2.3
>         Environment: LINUX, JRUN4, APACHE, 3GB RAM
>            Reporter: vaidy
>
> Lucene Index contains 2.1 Million records (indexed from 2.1 million records from sqlserver DB).
> Lucene Index file Size 256MB
> Lucene version: 2.3
> Searching works fine when we sort the results on a single field. However, if the search results is sorted on more than one field we get Out of Memory exception.
> We restrict the number of search results to 250
> Out of Memory exception is quite consistent in 2.3.  We recently moved from 2.0 to 2.3.  
> Version 2.0 also used to give the OOM but not this frequent.
> {color:red}
> *Code Snippet:*
> final QueryParser parser;
>         StandardAnalyzer analyzer = new StandardAnalyzer();
>         String sSearchQuery = "MUSIC";
>         Sort oSort = null;
> public static final String G2KEYFIELDS = "G2KEYFIELDS";
>             oSort =  new Sort(new SortField[]{
>                                   new SortField (GroupsConstants.LAST_MODIFIED,SortField.INT,true),
>                                   new SortField(GroupsConstants.GROUPNAME)
>                                 });
>            
>   parser = new QueryParser(G2KEYFIELDS, analyzer);
>         final Vector ids = new Vector();
>         FSDirectory dir = null;
>         IndexSearcher searcher = null;
>         try {
>             dir = FSDirectory.getDirectory(index);
>             searcher = new IndexSearcher(dir);
>             Query query = parser.parse(sSearchQuery);
>             Hits hits = searcher.search(query, oSort);
>             for (int i = 0; i != hits.length() && i != 250; ++i) {
>                 final Document doc = hits.doc(i);
>                 Integer oiGroupId=new Integer(doc.getField (GroupsConstants.IDENTITY).stringValue());
>                 if(!ids.contains(oiGroupId)){
>                     ids.addElement(oiGroupId);
>                 }
>             }
>             searcher.close();
>             dir.close();
> {color}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Reopened: (LUCENE-1193) OOM exception while sorting on more than 1 field

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Otis Gospodnetic reopened LUCENE-1193:
--------------------------------------


> OOM exception while sorting on more than 1 field
> ------------------------------------------------
>
>                 Key: LUCENE-1193
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1193
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 2.3
>         Environment: LINUX, JRUN4, APACHE, 3GB RAM
>            Reporter: vaidy
>
> Lucene Index contains 2.1 Million records (indexed from 2.1 million records from sqlserver DB).
> Lucene Index file Size 256MB
> Lucene version: 2.3
> Searching works fine when we sort the results on a single field. However, if the search results is sorted on more than one field we get Out of Memory exception.
> We restrict the number of search results to 250
> Out of Memory exception is quite consistent in 2.3.  We recently moved from 2.0 to 2.3.  
> Version 2.0 also used to give the OOM but not this frequent.
> {color:red}
> *Code Snippet:*
> final QueryParser parser;
>         StandardAnalyzer analyzer = new StandardAnalyzer();
>         String sSearchQuery = "MUSIC";
>         Sort oSort = null;
> public static final String G2KEYFIELDS = "G2KEYFIELDS";
>             oSort =  new Sort(new SortField[]{
>                                   new SortField (GroupsConstants.LAST_MODIFIED,SortField.INT,true),
>                                   new SortField(GroupsConstants.GROUPNAME)
>                                 });
>            
>   parser = new QueryParser(G2KEYFIELDS, analyzer);
>         final Vector ids = new Vector();
>         FSDirectory dir = null;
>         IndexSearcher searcher = null;
>         try {
>             dir = FSDirectory.getDirectory(index);
>             searcher = new IndexSearcher(dir);
>             Query query = parser.parse(sSearchQuery);
>             Hits hits = searcher.search(query, oSort);
>             for (int i = 0; i != hits.length() && i != 250; ++i) {
>                 final Document doc = hits.doc(i);
>                 Integer oiGroupId=new Integer(doc.getField (GroupsConstants.IDENTITY).stringValue());
>                 if(!ids.contains(oiGroupId)){
>                     ids.addElement(oiGroupId);
>                 }
>             }
>             searcher.close();
>             dir.close();
> {color}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]