[jira] [Created] (HADOOP-17273) FSDataInput/Output Streams to automatically collect IOStatistics

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] [Created] (HADOOP-17273) FSDataInput/Output Streams to automatically collect IOStatistics

Ayush Saxena (Jira)
Steve Loughran created HADOOP-17273:

             Summary: FSDataInput/Output Streams to automatically collect IOStatistics
                 Key: HADOOP-17273
                 URL: https://issues.apache.org/jira/browse/HADOOP-17273
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs
    Affects Versions: 3.3.1
            Reporter: Steve Loughran

the fs input/output streams to automatically collect stream IO statistics even if the inner classes don't; count invocations and durations

An issue here becomes "how would you aggregate with the inner stream, *efficiently*. Not sure there, except maybe

* have some interface to access an updatable IOStatisticsStore which, if the stream implements, says "here is something you can update"
* wrapper class updates that with counts, durations
* we extend DynamicIOStatistics for each counter/gauge etc to have a factory which can add new entries on demand (new AtomicLong, etc)
* so wrapper classes just update their stats, which  updates existing stats or triggers off an on-demand creation of a new entry

Not so informative in terms of low level details (HTTP/IPC requests and latency, errors, bytes discarded) but would give callers benefits of using the API for HDFS, Ozone, GCS

This message was sent by Atlassian Jira

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]