[jira] [Commented] (TIKA-3221) /rmeta/text endpoint - allow a "max parse time" parameter where after exceeded, return bytes/metadata mangaed to get up to that point

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (TIKA-3221) /rmeta/text endpoint - allow a "max parse time" parameter where after exceeded, return bytes/metadata mangaed to get up to that point

Steve Loughran (Jira)

    [ https://issues.apache.org/jira/browse/TIKA-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237611#comment-17237611 ]

Nicholas DiPiazza commented on TIKA-3221:
-----------------------------------------

[~tallison] your understanding is correct, and i have encountered this same problem with my hacky attempts to fix the problem. You can get the text up to a point, but the metadata will always be blank.
Might not be worth spending more time on this due to the extensive amount of changes it would need.

> /rmeta/text endpoint - allow a "max parse time" parameter where after exceeded, return bytes/metadata mangaed to get up to that point
> -------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: TIKA-3221
>                 URL: https://issues.apache.org/jira/browse/TIKA-3221
>             Project: Tika
>          Issue Type: Bug
>            Reporter: Nicholas DiPiazza
>            Priority: Major
>
> Can we make a change to the
> {code}
> /rmeta/text
> {code}
> endpoint to allow a "max parse time" parameter where after exceeded, return bytes/metadata managed to get up to that point.
> Motivation:
> I have a massive number of documents that I need to fetch through apache tika server.
> Prior to making a switch to tika server, I used a project I created myself https://github.com/nddipiazza/tika-fork that created tika forked VMs and would send work to the VMs through sockets directly.
> This was OK but super complicated so I chose to switch to the Tika jetty server for simplicity's sake.
> Tika Server works great for the most part for this use case... But one feature I had before was that I could say "If I don't get a result within MAX_PARSE_TIMEOUT_MS, then stop parsing at that moment and return the bytes we managed to get up to that point.
> This is because with the massive number of documents I need to parse, I cannot afford to have any parse hang longer than a certain amount of time. But conversely, if I make timeout 20 seconds, then I suffer massive gaps with *no* content at all.
> With the rmeta/text method, we recently added the ability to send a writeLimit where we will stop parsing after we reach that number of bytes.
> I'm hoping we can do the same for the time parsed. Perhaps when checking byte size, periodically check time and quit parser in the same way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)