[DISCUSS] Feature Branch Merge and Security Audits

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Feature Branch Merge and Security Audits

larry mccay-2
All -

Given the maturity of Hadoop at this point, I would like to propose that we
start doing explicit security audits of features at merge time.

There are a few reasons that I think this is a good place/time to do the
review:

1. It represents a specific snapshot of where the feature stands as a
whole. This means that we can more easily identity the attack surface of a
given feature.
2. We can identify any security gaps that need to be fixed before a release
that carries the feature can be considered ready.
3. We - in extreme cases - can block a feature from merging until some
baseline of security coverage is achieved.
4. The folks that are interested and able to review security aspects can't
scale for every iteration over every JIRA but can review the checklist and
follow pointers for specific areas of interest.

I have provided an impromptu security audit checklist on the DISCUSS thread
for merging Ozone - HDFS-7240 into trunk.

I don't want to pick on it particularly but I think it is a good way to
bootstrap this audit process and figure out how to incorporate it without
being too intrusive.

The questions that I provided below are a mix of general questions that
could be on a standard checklist that you provide along with the merge
thread and some that are specific to what I read about ozone in the
excellent docs provided. So, we should consider some subset of the
following as a proposal for a general checklist.

Perhaps, a shared document can be created to iterate over the list to fine
tune it?

Any thoughts on this, any additional datapoints to collect, etc?

thanks!

--larry

1. UIs
I see there are at least two UIs - Storage Container Manager and Key Space
Manager. There are a number of typical vulnerabilities that we find in UIs

1.1. What sort of validation is being done on any accepted user input?
(pointers to code would be appreciated)
1.2. What explicit protections have been built in for (pointers to code
would be appreciated):
  1.2.1. cross site scripting
  1.2.2. cross site request forgery
  1.2.3. click jacking (X-Frame-Options)
1.3. What sort of authentication is required for access to the UIs?
1.4. What authorization is available for determining who can access what
capabilities of the UIs for either viewing, modifying data or affecting
object stores and related processes?
1.5. Are the UIs built with proxying in mind by leveraging X-Forwarded
headers?
1.6. Is there any input that will ultimately be persisted in configuration
for executing shell commands or processes?
1.7. Do the UIs support the trusted proxy pattern with doas impersonation?
1.8. Is there TLS/SSL support?

2. REST APIs

2.1. Do the REST APIs support the trusted proxy pattern with doas
impersonation capabilities?
2.2. What explicit protections have been built in for:
  2.2.1. cross site scripting (XSS)
  2.2.2. cross site request forgery (CSRF)
  2.2.3. XML External Entity (XXE)
2.3. What is being used for authentication - Hadoop Auth Module?
2.4. Are there separate processes for the HTTP resources (UIs and REST
endpoints) or are the part of existing HDFS processes?
2.5. Is there TLS/SSL support?
2.6. Are there new CLI commands and/or clients for access the REST APIs?
2.7. Bucket Level API allows for setting of ACLs on a bucket - what
authorization is required here - is there a restrictive ACL set on creation?
2.8. Bucket Level API allows for deleting a bucket - I assume this is
dependent on ACLs based access control?
2.9. Bucket Level API to list bucket returns up to 1000 keys - is there
paging available?
2.10. Storage Level APIs indicate “Signed with User Authorization” what
does this refer to exactly?
2.11. Object Level APIs indicate that there is no ACL support and only
bucket owners can read and write - but there are ACL APIs on the Bucket
Level are they meaningless for now?
2.12. How does a REST client know which Ozone Handler to connect to or am I
missing some well known NN type endpoint in the architecture doc somewhere?

3. Encryption

3.1. Is there any support for encryption of persisted data?
3.2. If so, is KMS and the hadoop key command used for key management?

4. Configuration

4.1. Are there any passwords or secrets being added to configuration?
4.2. If so, are they accessed via Configuration.getPassword() to allow for
provisioning in credential providers?
4.3. Are there any settings that are used to launch docker containers or
shell out any commands, etc?

5. HA

5.1. Are there provisions for HA?
5.2. Are we leveraging the existing HA capabilities in HDFS?
5.3. Is Storage Container Manager a SPOF?
5.4. I see HA listed in future work in the architecture doc - is this still
an open issue?
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Feature Branch Merge and Security Audits

larry mccay-2
New Revision...

This revision acknowledges the reality that we often have multiple phases
of feature lifecycle and that we need to account for each phase.
It has also been made more generic.
I have created a Tech Preview Security Audit list and a GA Readiness
Security Audit list.
I've also included suggested items into the GA Readiness list.

It has also been suggested that we publish the information as part of docs
so that the state of such features can be easily determined from these
pages. We can discuss this aspect as well.

Thoughts?

*Tech Preview Security Audit*
For features that are being merged without full security model coverage,
there need to be a base line of assurances that they do not introduce new
attack vectors in deployments that are from actual releases or even just
built from trunk.

*1. UIs*

1.1. Are there new UIs added with this merge?
1.2. Are they enabled/accessible by default?
1.3. Are they hosted in existing processes or as part of a new
process/server?
1.4. If new process/server, is it launched by default?

*2. APIs*

2.1. Are there new REST APIs added with this merge?
2.2. Are they enabled by default?
2.3. Are there RPC based APIs added with this merge?
2.4. Are they enabled by default?

*3. Secure Clusters*

3.1. Is this feature disabled completely in secure deployments?
3.2. If not, is there some justification as to why it should be available?

*4. CVEs*

4.1. Have all dependencies introduced by this merge been checked for known
issues?


--------------------------------------------------------------------------------------------------------------------------------------------------


*GA Readiness Security Audit*
At this point, we are merging full or partial security model
implementations.
Let's inventory what is covered by the model at this point and whether
there are future merges required to be full.

*1. UIs*

1.1. What sort of validation is being done on any accepted user input?
(pointers to code would be appreciated)
1.2. What explicit protections have been built in for (pointers to code
would be appreciated):
  1.2.1. cross site scripting
  1.2.2. cross site request forgery
  1.2.3. click jacking (X-Frame-Options)
1.3. What sort of authentication is required for access to the UIs?
  1.3.1. Kerberos
    1.3.1.1. has TGT renewal been accounted for
    1.3.1.2. SPNEGO support?
    1.3.1.3. Delegation token?
  1.3.2. Proxy User ACL?
1.4. What authorization is available for determining who can access what
capabilities of the UIs for either viewing, modifying data and/or related
processes?
1.5. Is there any input that will ultimately be persisted in configuration
for executing shell commands or processes?
1.6. Do the UIs support the trusted proxy pattern with doas impersonation?
1.7. Is there TLS/SSL support?

*2. REST APIs*

2.1. Do the REST APIs support the trusted proxy pattern with doas
impersonation capabilities?
2.2. What explicit protections have been built in for:
  2.2.1. cross site scripting (XSS)
  2.2.2. cross site request forgery (CSRF)
  2.2.3. XML External Entity (XXE)
2.3. What is being used for authentication - Hadoop Auth Module?
2.4. Are there separate processes for the HTTP resources (UIs and REST
endpoints) or are they part of existing processes?
2.5. Is there TLS/SSL support?
2.6. Are there new CLI commands and/or clients for accessing the REST APIs?
2.7. What authorization enforcement points are there within the REST APIs?

*3. Encryption*

3.1. Is there any support for encryption of persisted data?
3.2. If so, is KMS and the hadoop key command used for key management?
3.3. KMS interaction with Proxy Users?

*4. Configuration*

4.1. Are there any passwords or secrets being added to configuration?
4.2. If so, are they accessed via Configuration.getPassword() to allow for
provisioning to credential providers?
4.3. Are there any settings that are used to launch docker containers or
shell out command execution, etc?

*5. HA*

5.1. Are there provisions for HA?
5.2. Are there any single point of failures?

*6. CVEs*

Dependencies need to have been checked for known issues before we merge.
We don't however want to list any CVEs that have been fixed but not
released yet.

6.1. All dependencies checked for CVEs?




On Sat, Oct 21, 2017 at 10:26 AM, larry mccay <[hidden email]> wrote:

> Hi Marton -
>
> I don't think there is any denying that it would be great to have such
> documentation for all of those reasons.
> If it is a natural extension of getting the checklist information as an
> assertion of security state when merging then we can certainly include it.
>
> I think that backfilling all such information across the project is a
> different topic altogether and wouldn't want to expand the scope of this
> discussion in that direction.
>
> Thanks for the great thoughts on this!
>
> thanks,
>
> --larry
>
>
>
>
>
> On Sat, Oct 21, 2017 at 3:00 AM, Elek, Marton <[hidden email]> wrote:
>
>>
>>
>> On 10/21/2017 02:41 AM, larry mccay wrote:
>>
>>>
>>> "We might want to start a security section for Hadoop wiki for each of
>>>> the
>>>> services and components.
>>>> This helps to track what has been completed."
>>>>
>>>
>>> Do you mean to keep the audit checklist for each service and component
>>> there?
>>> Interesting idea, I wonder what sort of maintenance that implies and
>>> whether we want to take on that burden even though it would be great
>>> information to have for future reviewers.
>>>
>>
>> I think we should care about the maintenance of the documentation anyway.
>> We also need to maintain all the other documentations. I think it could be
>> even part of the generated docs and not the wiki.
>>
>> I also suggest to fill this list about the current trunk/3.0 as a first
>> step.
>>
>> 1. It would be a very usefull documentation for the end-users (some
>> answers could link the existing documentation, it exists, but I am not sure
>> if all the answers are in the current documentation.)
>>
>> 2. It would be a good example who the questions could be answered.
>>
>> 3. It would help to check, if something is missing from the list.
>>
>> 4. There are future branches where some of the components are not
>> touched. For example, no web ui or no REST service. A prefilled list could
>> help to check if the branch doesn't break any old security functionality on
>> trunk.
>>
>> 5. It helps to document the security features in one place. If we have a
>> list for the existing functionality in the same format, it would be easy to
>> merge the new documentation of the new features as they will be reported in
>> the same form. (So it won't be so hard to maintain the list...).
>>
>> Marton
>>
>
>