[jira] [Created] (HADOOP-17469) IOStatistics Phase II

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] [Created] (HADOOP-17469) IOStatistics Phase II

Steve Loughran (Jira)
Steve Loughran created HADOOP-17469:

             Summary: IOStatistics Phase II
                 Key: HADOOP-17469
                 URL: https://issues.apache.org/jira/browse/HADOOP-17469
             Project: Hadoop Common
          Issue Type: New Feature
          Components: fs, fs/azure, fs/s3
    Affects Versions: 3.3.1
            Reporter: Steve Loughran

Continue IOStatistics development with goals of

* Easy adoption in applications
* better instrumentation in hadoop codebase (distcp?)
* more stats in abfs and s3a connectors

A key has to be a thread level context for statistics so that app code doesn't have to explicitly ask for the stats for each worker thread. Instead

filesystem components update the context stats as well as thread stats (when?) and then apps can pick up.

* need to manage performance by minimising inefficient lookups, lock acquisition etc on what should be memory-only ops (read()), (write()),
* and for duration tracking, cut down on calls to System.currentTime() so that only 1 should be made per operation,
* need to propagate the context into worker threads

This message was sent by Atlassian Jira

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]