[jira] [Commented] (TIKA-2750) Update regression corpus

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] [Commented] (TIKA-2750) Update regression corpus

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/TIKA-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677252#comment-16677252 ]

Tim Allison commented on TIKA-2750:

I drafted a write-up of steps taken so far here: https://wiki.apache.org/tika/CommonCrawl3.  I'm still refetching truncated documents, but we have 71GB of new data. :D

Next step is to figure out what to pitch and what to keep from our older collection of CommonCrawl...

> Update regression corpus
> ------------------------
>                 Key: TIKA-2750
>                 URL: https://issues.apache.org/jira/browse/TIKA-2750
>             Project: Tika
>          Issue Type: Task
>            Reporter: Tim Allison
>            Priority: Major
>         Attachments: CC-MAIN-2018-39-charset_lang_by_tld.zip, CC-MAIN-2018-39-mimes-charsets-by-tld.zip, CC-MAIN-2018-39-mimes-v-detected.zip
> I think we've had great success with the current data on our regression corpus.  I'd like to re-fresh some data from common crawl with three primary goals:
> 1) include more interesting documents (e.g. down sample English UTF-8 text/html)
> 2) include more recent documents (perhaps newer features in PDFs? definitely more ooxml)
> 3) identify and re-fetch truncated documents from the original site -- CommonCrawl truncates docs at 1 MB.  I think some truncated documents have been quite useful, similar to fuzzing, for identifying serious problems with some of our parsers.  However, it would be useful to have more complete files, esp. for PDFs.  In short, we should keep some truncated documents, but I'd also like to get more complete docs.

This message was sent by Atlassian JIRA