Index files not being deleted

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Index files not being deleted

AgentHubcap
I've got an issue with my index files not being deleted.  I have a cron job set up to kick off the index process every 5 minutes (test box, with little data).  The process is as follows:

- Delete all index files via a delete command
- Loop through all the records in the database and index them
- Commits are triggered every 250 records
- An optimize is done at the end (twice because I read there was an issue with the optimize command)

What is happening is that the number of open files keeps growing every time the cron job runs, and the disk space is eventually used up by deleted files (something isn't closing the handle on the file, and thus the file is never really deleted)

My question is what exactly do I need to do to close these files?  This is for an index only -- no searching is taking place.  I'm also using the embedded version if that makes a difference...
Reply | Threaded
Open this post in threaded view
|

Re: Index files not being deleted

Ryan McKinley
>
> - Delete all index files via a delete command

make sure to optimize after deleting the docs -- optimize has lucene get
rid of deleted files rather then appending them to the end of the index.

what version of solr are you running?  if you are running 1.3-dev
deleting *:* is fast -- if you aren't using 1.3, i don't suggest moving
there just for that though....

ryan
Reply | Threaded
Open this post in threaded view
|

Re: Index files not being deleted

AgentHubcap
I'm running 1.2.

Acutally, i am doing an optimize after I delete the indexes.  (twice, as I read there was an issue with the optimize).  Do I need to close something manually?

Here's my optimize code:

        private void optimize() throws IOException
        {
                UpdateHandler updateHandler = SolrCore.getSolrCore().getUpdateHandler();
                CommitUpdateCommand commitcmd = new CommitUpdateCommand(false);
                commitcmd.optimize = true;
                updateHandler.commit(commitcmd);
                updateHandler.close();
        }



ryantxu wrote
>
> - Delete all index files via a delete command

make sure to optimize after deleting the docs -- optimize has lucene get
rid of deleted files rather then appending them to the end of the index.

what version of solr are you running?  if you are running 1.3-dev
deleting *:* is fast -- if you aren't using 1.3, i don't suggest moving
there just for that though....

ryan
Reply | Threaded
Open this post in threaded view
|

Re: Index files not being deleted

AgentHubcap
As it turns out I was modifying code that wasn't being run.  Running an optimize after deleting did solve my problem.  


AgentHubcap wrote
I'm running 1.2.

Acutally, i am doing an optimize after I delete the indexes.  (twice, as I read there was an issue with the optimize).  Do I need to close something manually?

Here's my optimize code:

        private void optimize() throws IOException
        {
                UpdateHandler updateHandler = SolrCore.getSolrCore().getUpdateHandler();
                CommitUpdateCommand commitcmd = new CommitUpdateCommand(false);
                commitcmd.optimize = true;
                updateHandler.commit(commitcmd);
                updateHandler.close();
        }



ryantxu wrote
>
> - Delete all index files via a delete command

make sure to optimize after deleting the docs -- optimize has lucene get
rid of deleted files rather then appending them to the end of the index.

what version of solr are you running?  if you are running 1.3-dev
deleting *:* is fast -- if you aren't using 1.3, i don't suggest moving
there just for that though....

ryan
Reply | Threaded
Open this post in threaded view
|

Re: Index files not being deleted

AgentHubcap
So, this problem came up again.  Now it only happens in a linux environment when searches are being conducted while an index is running.

Does anything need to be closed on the searching side?


AgentHubcap wrote
As it turns out I was modifying code that wasn't being run.  Running an optimize after deleting did solve my problem.  


AgentHubcap wrote
I'm running 1.2.

Acutally, i am doing an optimize after I delete the indexes.  (twice, as I read there was an issue with the optimize).  Do I need to close something manually?

Here's my optimize code:

        private void optimize() throws IOException
        {
                UpdateHandler updateHandler = SolrCore.getSolrCore().getUpdateHandler();
                CommitUpdateCommand commitcmd = new CommitUpdateCommand(false);
                commitcmd.optimize = true;
                updateHandler.commit(commitcmd);
                updateHandler.close();
        }



ryantxu wrote
>
> - Delete all index files via a delete command

make sure to optimize after deleting the docs -- optimize has lucene get
rid of deleted files rather then appending them to the end of the index.

what version of solr are you running?  if you are running 1.3-dev
deleting *:* is fast -- if you aren't using 1.3, i don't suggest moving
there just for that though....

ryan