[jira] [Resolved] (HADOOP-15780) S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Resolved] (HADOOP-15780) S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-15780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gabor Bota resolved HADOOP-15780.
---------------------------------
    Resolution: Fixed

> S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-15780
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15780
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 3.2.0
>            Reporter: Aaron Fabbri
>            Assignee: Gabor Bota
>            Priority: Major
>
> Our general policy for S3Guard is this: All modifiers of a bucket that is configured for use with S3Guard, must use S3Guard. Otherwise, the MetadataStore will not be properly updated as the S3 bucket changes and problems will arise.
> There are limited circumstances in which may be safe to have an external (non-s3guard) process writing data.  There are also scenarios where it definitely breaks things.
> I think we should start by documenting the cases that this works / does not work for. After we've enumerated that, we can suggest enhancements as needed to make this sort of configuration easier to use.
> To get the ball rolling, some things that do not work:
> - Deleting a path *p* with S3Guard, then writing a new file at path *p* without S3guard (will still have delete marker in S3Guard, making the file appear to be deleted but still visible in S3 due to false "eventual consistency") (as [~[hidden email]] and I have discussed)
> - When fs.s3a.metadatastore.authoritative is true, adding files to directories without S3Guard, then listing with S3Guard may exclude externally-written files from listings.
> (Note, there are also S3A interop issues with other non-S3A clients even without S3Guard, due to the unique way S3A interprets empty directory markers).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]