[jira] Created: (SOLR-1953) It may be possible for temporary files to accumulator until the Solr process is shut down

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (SOLR-1953) It may be possible for temporary files to accumulator until the Solr process is shut down

JIRA jira@apache.org
It may be possible for temporary files to accumulator until the Solr process is shut down
-----------------------------------------------------------------------------------------

                 Key: SOLR-1953
                 URL: https://issues.apache.org/jira/browse/SOLR-1953
             Project: Solr
          Issue Type: Bug
          Components: update
    Affects Versions: 1.4.1, 1.5
            Reporter: Karl Wright


While researching SOLR-1951, the behavior of commons-fileupload in handling multipart form posts came into question.  commons-fileupload creates a temporary file for the main content area of such posts, and purportedly has a background thread which cleans up these files.  However, Mark Miller discovered that the javadoc in this matter may be incorrect, and that commons-fileupload may in fact just be adding files to the JVM's list of files needing cleanup on exit.

If so, this will show up in two ways: first, temporary files will accumulate in the java.io.tmpdir area.  Second, non-heap memory for the JVM will slowly increase over time (since the file pointers the JVM tracks in this way are not kept in the java heap).

I will attach a potential fix; however, this ticket should be viewed as a workitem for the need for further research in this area.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-1953) It may be possible for temporary files to accumulator until the Solr process is shut down

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/SOLR-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karl Wright updated SOLR-1953:
------------------------------

    Attachment: SOLR-1953.patch

Patch that Mark Miller proposed, if it turns out that his analysis is correct.


> It may be possible for temporary files to accumulator until the Solr process is shut down
> -----------------------------------------------------------------------------------------
>
>                 Key: SOLR-1953
>                 URL: https://issues.apache.org/jira/browse/SOLR-1953
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.4.1, 1.5
>            Reporter: Karl Wright
>         Attachments: SOLR-1953.patch
>
>
> While researching SOLR-1951, the behavior of commons-fileupload in handling multipart form posts came into question.  commons-fileupload creates a temporary file for the main content area of such posts, and purportedly has a background thread which cleans up these files.  However, Mark Miller discovered that the javadoc in this matter may be incorrect, and that commons-fileupload may in fact just be adding files to the JVM's list of files needing cleanup on exit.
> If so, this will show up in two ways: first, temporary files will accumulate in the java.io.tmpdir area.  Second, non-heap memory for the JVM will slowly increase over time (since the file pointers the JVM tracks in this way are not kept in the java heap).
> I will attach a potential fix; however, this ticket should be viewed as a workitem for the need for further research in this area.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-1953) It may be possible for temporary files to accumulator until the Solr process is shut down

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878950#action_12878950 ]

Mark Miller commented on SOLR-1953:
-----------------------------------

I've got to look at the code - but I wonder if this should be a shared instance that is reused - the cleaner uses a thread that has a special call related to shutting it down. I don't remember all of the specifics, and I have to look at the commons code, but it may not make sense to use a new cleaner on every object creation. Will post back.

> It may be possible for temporary files to accumulator until the Solr process is shut down
> -----------------------------------------------------------------------------------------
>
>                 Key: SOLR-1953
>                 URL: https://issues.apache.org/jira/browse/SOLR-1953
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.4.1, 1.5
>            Reporter: Karl Wright
>         Attachments: SOLR-1953.patch
>
>
> While researching SOLR-1951, the behavior of commons-fileupload in handling multipart form posts came into question.  commons-fileupload creates a temporary file for the main content area of such posts, and purportedly has a background thread which cleans up these files.  However, Mark Miller discovered that the javadoc in this matter may be incorrect, and that commons-fileupload may in fact just be adding files to the JVM's list of files needing cleanup on exit.
> If so, this will show up in two ways: first, temporary files will accumulate in the java.io.tmpdir area.  Second, non-heap memory for the JVM will slowly increase over time (since the file pointers the JVM tracks in this way are not kept in the java heap).
> I will attach a potential fix; however, this ticket should be viewed as a workitem for the need for further research in this area.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-1953) It may be possible for temporary files to accumulator until the Solr process is shut down

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879214#action_12879214 ]

Mark Miller commented on SOLR-1953:
-----------------------------------

Yup - def looks to me like we only want one Tracker instance rather than a new one everytime - and we also want to call exitWhenFinished in some clean up code - perhaps in SolrDispatchFilter#destroy()

> It may be possible for temporary files to accumulator until the Solr process is shut down
> -----------------------------------------------------------------------------------------
>
>                 Key: SOLR-1953
>                 URL: https://issues.apache.org/jira/browse/SOLR-1953
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.4.1, 1.5
>            Reporter: Karl Wright
>         Attachments: SOLR-1953.patch
>
>
> While researching SOLR-1951, the behavior of commons-fileupload in handling multipart form posts came into question.  commons-fileupload creates a temporary file for the main content area of such posts, and purportedly has a background thread which cleans up these files.  However, Mark Miller discovered that the javadoc in this matter may be incorrect, and that commons-fileupload may in fact just be adding files to the JVM's list of files needing cleanup on exit.
> If so, this will show up in two ways: first, temporary files will accumulate in the java.io.tmpdir area.  Second, non-heap memory for the JVM will slowly increase over time (since the file pointers the JVM tracks in this way are not kept in the java heap).
> I will attach a potential fix; however, this ticket should be viewed as a workitem for the need for further research in this area.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-1953) It may be possible for temporary files to accumulator until the Solr process is shut down

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/SOLR-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated SOLR-1953:
------------------------------

    Attachment: SOLR-1953.patch

I think we want something more like this - though it deserves some more thought.

> It may be possible for temporary files to accumulator until the Solr process is shut down
> -----------------------------------------------------------------------------------------
>
>                 Key: SOLR-1953
>                 URL: https://issues.apache.org/jira/browse/SOLR-1953
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.4.1, 1.5
>            Reporter: Karl Wright
>         Attachments: SOLR-1953.patch, SOLR-1953.patch
>
>
> While researching SOLR-1951, the behavior of commons-fileupload in handling multipart form posts came into question.  commons-fileupload creates a temporary file for the main content area of such posts, and purportedly has a background thread which cleans up these files.  However, Mark Miller discovered that the javadoc in this matter may be incorrect, and that commons-fileupload may in fact just be adding files to the JVM's list of files needing cleanup on exit.
> If so, this will show up in two ways: first, temporary files will accumulate in the java.io.tmpdir area.  Second, non-heap memory for the JVM will slowly increase over time (since the file pointers the JVM tracks in this way are not kept in the java heap).
> I will attach a potential fix; however, this ticket should be viewed as a workitem for the need for further research in this area.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-1953) It may be possible for temporary files to accumulator until the Solr process is shut down

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886479#action_12886479 ]

Karl Wright commented on SOLR-1953:
-----------------------------------

Looks good to me...


> It may be possible for temporary files to accumulator until the Solr process is shut down
> -----------------------------------------------------------------------------------------
>
>                 Key: SOLR-1953
>                 URL: https://issues.apache.org/jira/browse/SOLR-1953
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.4.1, 1.5
>            Reporter: Karl Wright
>         Attachments: SOLR-1953.patch, SOLR-1953.patch
>
>
> While researching SOLR-1951, the behavior of commons-fileupload in handling multipart form posts came into question.  commons-fileupload creates a temporary file for the main content area of such posts, and purportedly has a background thread which cleans up these files.  However, Mark Miller discovered that the javadoc in this matter may be incorrect, and that commons-fileupload may in fact just be adding files to the JVM's list of files needing cleanup on exit.
> If so, this will show up in two ways: first, temporary files will accumulate in the java.io.tmpdir area.  Second, non-heap memory for the JVM will slowly increase over time (since the file pointers the JVM tracks in this way are not kept in the java heap).
> I will attach a potential fix; however, this ticket should be viewed as a workitem for the need for further research in this area.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]