FileSystem Vs ZKStateStore for RM recovery

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

FileSystem Vs ZKStateStore for RM recovery

Suma Shivaprasad
We are planning to deploy Hadoop 2.6.0 with a default configuration to cache 10000 entries in the state store. With a workload of 150-250 concurrent applications at any time , which state store is better to use  and for what reasons ?

Thanks
Suma
Reply | Threaded
Open this post in threaded view
|

Re: FileSystem Vs ZKStateStore for RM recovery

Nikhil-2
>>cache 10000 entries in the state store.
May I know what are 10000 entries you are talking about here? Any details please.

On Tue, Feb 10, 2015 at 9:59 PM, Suma Shivaprasad <[hidden email]> wrote:
We are planning to deploy Hadoop 2.6.0 with a default configuration to cache 10000 entries in the state store. With a workload of 150-250 concurrent applications at any time , which state store is better to use  and for what reasons ?

Thanks
Suma

Reply | Threaded
Open this post in threaded view
|

Re: FileSystem Vs ZKStateStore for RM recovery

Karthik Kambatla-2
In reply to this post by Suma Shivaprasad
We recommend ZK-store, particularly if you plan to deploy multiple ResourceManagers with failover. ZK-store ensures a single RM has write access and thus is better protected against split-brain cases where both RMs think they are active. 

On Tue, Feb 10, 2015 at 9:59 PM, Suma Shivaprasad <[hidden email]> wrote:
We are planning to deploy Hadoop 2.6.0 with a default configuration to
cache 10000 entries in the state store. With a workload of 150-250
concurrent applications at any time , which state store is better to use
and for what reasons ?

Thanks
Suma



--
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------

Reply | Threaded
Open this post in threaded view
|

Re: FileSystem Vs ZKStateStore for RM recovery

Suma Shivaprasad
In reply to this post by Nikhil-2
We have set yarn.resourcemanager.max-completed-applications=10000. I assume this is the no of entries kept in RMStateStore since I see these in logs

2015-02-11 00:00:00,579 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: Max number of completed apps kept in state store met: maxCompletedAppsInStateStore = 10000, removing app application_1422879343468_543109 from state store.

Thanks
Suma


On Wed, Feb 11, 2015 at 1:09 PM, Nikhil <[hidden email]> wrote:
>>cache 10000 entries in the state store.
May I know what are 10000 entries you are talking about here? Any details please.

On Tue, Feb 10, 2015 at 9:59 PM, Suma Shivaprasad <[hidden email]> wrote:
We are planning to deploy Hadoop 2.6.0 with a default configuration to cache 10000 entries in the state store. With a workload of 150-250 concurrent applications at any time , which state store is better to use  and for what reasons ?

Thanks
Suma


Reply | Threaded
Open this post in threaded view
|

Re: FileSystem Vs ZKStateStore for RM recovery

Suma Shivaprasad
In reply to this post by Karthik Kambatla-2
Can ZKStateStore scale for large clusters. Any idea on the number of concurrent jobs that can be supported on top of these ?

Thanks
Suma

On Wed, Feb 11, 2015 at 1:45 PM, Karthik Kambatla <[hidden email]> wrote:
We recommend ZK-store, particularly if you plan to deploy multiple ResourceManagers with failover. ZK-store ensures a single RM has write access and thus is better protected against split-brain cases where both RMs think they are active. 

On Tue, Feb 10, 2015 at 9:59 PM, Suma Shivaprasad <[hidden email]> wrote:
We are planning to deploy Hadoop 2.6.0 with a default configuration to
cache 10000 entries in the state store. With a workload of 150-250
concurrent applications at any time , which state store is better to use
and for what reasons ?

Thanks
Suma



--
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------


Reply | Threaded
Open this post in threaded view
|

Re: FileSystem Vs ZKStateStore for RM recovery

Tsuyoshi Ozawa-3
Hi Suma,

I think ZooKeeper can handle thousands of updates, so thousands of
jobs can be "launched" at the same time.
More jobs can be running at the same time since the number of updates
against ZooKeeper is less than the number of jobs. Please free to ask
us if you face the scalability or performance issue when you test. We
can tackle the issue.

Thanks,
- Tsuyoshi

On Wed, Feb 11, 2015 at 6:08 PM, Suma Shivaprasad
<[hidden email]> wrote:

> Can ZKStateStore scale for large clusters. Any idea on the number of
> concurrent jobs that can be supported on top of these ?
>
> Thanks
> Suma
>
> On Wed, Feb 11, 2015 at 1:45 PM, Karthik Kambatla <[hidden email]>
> wrote:
>>
>> We recommend ZK-store, particularly if you plan to deploy multiple
>> ResourceManagers with failover. ZK-store ensures a single RM has write
>> access and thus is better protected against split-brain cases where both RMs
>> think they are active.
>>
>> On Tue, Feb 10, 2015 at 9:59 PM, Suma Shivaprasad
>> <[hidden email]> wrote:
>>>
>>> We are planning to deploy Hadoop 2.6.0 with a default configuration to
>>> cache 10000 entries in the state store. With a workload of 150-250
>>> concurrent applications at any time , which state store is better to use
>>> and for what reasons ?
>>>
>>> Thanks
>>> Suma
>>
>>
>>
>>
>> --
>> Karthik Kambatla
>> Software Engineer, Cloudera Inc.
>> --------------------------------------------
>> http://five.sentenc.es
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: FileSystem Vs ZKStateStore for RM recovery

Tsuyoshi Ozawa-3
> I think ZooKeeper can handle thousands of updates,

I meant "thousands of updates per second".

Thanks,
- Tsuyoshi

On Fri, Feb 13, 2015 at 3:59 PM, Tsuyoshi Ozawa <[hidden email]> wrote:

> Hi Suma,
>
> I think ZooKeeper can handle thousands of updates, so thousands of
> jobs can be "launched" at the same time.
> More jobs can be running at the same time since the number of updates
> against ZooKeeper is less than the number of jobs. Please free to ask
> us if you face the scalability or performance issue when you test. We
> can tackle the issue.
>
> Thanks,
> - Tsuyoshi
>
> On Wed, Feb 11, 2015 at 6:08 PM, Suma Shivaprasad
> <[hidden email]> wrote:
>> Can ZKStateStore scale for large clusters. Any idea on the number of
>> concurrent jobs that can be supported on top of these ?
>>
>> Thanks
>> Suma
>>
>> On Wed, Feb 11, 2015 at 1:45 PM, Karthik Kambatla <[hidden email]>
>> wrote:
>>>
>>> We recommend ZK-store, particularly if you plan to deploy multiple
>>> ResourceManagers with failover. ZK-store ensures a single RM has write
>>> access and thus is better protected against split-brain cases where both RMs
>>> think they are active.
>>>
>>> On Tue, Feb 10, 2015 at 9:59 PM, Suma Shivaprasad
>>> <[hidden email]> wrote:
>>>>
>>>> We are planning to deploy Hadoop 2.6.0 with a default configuration to
>>>> cache 10000 entries in the state store. With a workload of 150-250
>>>> concurrent applications at any time , which state store is better to use
>>>> and for what reasons ?
>>>>
>>>> Thanks
>>>> Suma
>>>
>>>
>>>
>>>
>>> --
>>> Karthik Kambatla
>>> Software Engineer, Cloudera Inc.
>>> --------------------------------------------
>>> http://five.sentenc.es
>>>
>>