[jira] [Commented] (TIKA-3221) /rmeta/text endpoint - allow a "max parse time" parameter where after exceeded, return bytes/metadata mangaed to get up to that point

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (TIKA-3221) /rmeta/text endpoint - allow a "max parse time" parameter where after exceeded, return bytes/metadata mangaed to get up to that point

Steve Loughran (Jira)

    [ https://issues.apache.org/jira/browse/TIKA-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237679#comment-17237679 ]

Tim Allison commented on TIKA-3221:
-----------------------------------

So, short of returning text extracted so far, I think it would still be useful to know which file caused a timeout/oom.  At this point, in a multithreaded environment, I don't think there's a good way to know which file caused the timeout.

We can add a timeout thread per parse, and the overhead is minimal.  For a "heavy hang" parse of 100 ms per file, with 10 threads and 100 calls per thread, the total elapsed time without a timeout thread (now) is 124644 ms, and the time with a timeout thread is 125634 ms (averaged over 5 runs).  The wallclock times are indistinguishable at around 11700 ms.



> /rmeta/text endpoint - allow a "max parse time" parameter where after exceeded, return bytes/metadata mangaed to get up to that point
> -------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: TIKA-3221
>                 URL: https://issues.apache.org/jira/browse/TIKA-3221
>             Project: Tika
>          Issue Type: Bug
>            Reporter: Nicholas DiPiazza
>            Priority: Major
>
> Can we make a change to the
> {code}
> /rmeta/text
> {code}
> endpoint to allow a "max parse time" parameter where after exceeded, return bytes/metadata managed to get up to that point.
> Motivation:
> I have a massive number of documents that I need to fetch through apache tika server.
> Prior to making a switch to tika server, I used a project I created myself https://github.com/nddipiazza/tika-fork that created tika forked VMs and would send work to the VMs through sockets directly.
> This was OK but super complicated so I chose to switch to the Tika jetty server for simplicity's sake.
> Tika Server works great for the most part for this use case... But one feature I had before was that I could say "If I don't get a result within MAX_PARSE_TIMEOUT_MS, then stop parsing at that moment and return the bytes we managed to get up to that point.
> This is because with the massive number of documents I need to parse, I cannot afford to have any parse hang longer than a certain amount of time. But conversely, if I make timeout 20 seconds, then I suffer massive gaps with *no* content at all.
> With the rmeta/text method, we recently added the ability to send a writeLimit where we will stop parsing after we reach that number of bytes.
> I'm hoping we can do the same for the time parsed. Perhaps when checking byte size, periodically check time and quit parser in the same way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)