Quantcast

leap second bug

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

leap second bug

Michael Tsadikov
Our solr servers went into GC hell, and became non-responsive on date
change today.

Restarting tomcats did not help.

Rebooting the machine did.

http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: leap second bug

Óscar Marín Miró
Hello Michael, thanks for the note :)

I'm having a similar problem since yesterday, tomcats are wild on CPU [near
100%]. Did your solr servers did not reply to index/query requests?

Thanks :)

On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <[hidden email]>wrote:

> Our solr servers went into GC hell, and became non-responsive on date
> change today.
>
> Restarting tomcats did not help.
>
> Rebooting the machine did.
>
>
> http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
>



--
Whether it's science, technology, personal experience, true love,
astrology, or gut feelings, each of us has confidence in something that we
will never fully comprehend.
 --Roy H. William
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: leap second bug

Michael McCandless-2
Looks like this is a low-level Linux issue ... see Shay's email to the
ElasticSearch list about it:

    https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY

Also see the comments here:

     http://news.ycombinator.com/item?id=4182642

Mike McCandless

http://blog.mikemccandless.com

On Sun, Jul 1, 2012 at 8:08 AM, Óscar Marín Miró
<[hidden email]> wrote:

> Hello Michael, thanks for the note :)
>
> I'm having a similar problem since yesterday, tomcats are wild on CPU [near
> 100%]. Did your solr servers did not reply to index/query requests?
>
> Thanks :)
>
> On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <[hidden email]>wrote:
>
>> Our solr servers went into GC hell, and became non-responsive on date
>> change today.
>>
>> Restarting tomcats did not help.
>>
>> Rebooting the machine did.
>>
>>
>> http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
>>
>
>
>
> --
> Whether it's science, technology, personal experience, true love,
> astrology, or gut feelings, each of us has confidence in something that we
> will never fully comprehend.
>  --Roy H. William
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: leap second bug

Óscar Marín Miró
Thanks Michael, nice information :)

On Sun, Jul 1, 2012 at 5:29 PM, Michael McCandless <
[hidden email]> wrote:

> Looks like this is a low-level Linux issue ... see Shay's email to the
> ElasticSearch list about it:
>
>
> https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY
>
> Also see the comments here:
>
>      http://news.ycombinator.com/item?id=4182642
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sun, Jul 1, 2012 at 8:08 AM, Óscar Marín Miró
> <[hidden email]> wrote:
> > Hello Michael, thanks for the note :)
> >
> > I'm having a similar problem since yesterday, tomcats are wild on CPU
> [near
> > 100%]. Did your solr servers did not reply to index/query requests?
> >
> > Thanks :)
> >
> > On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <[hidden email]
> >wrote:
> >
> >> Our solr servers went into GC hell, and became non-responsive on date
> >> change today.
> >>
> >> Restarting tomcats did not help.
> >>
> >> Rebooting the machine did.
> >>
> >>
> >>
> http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
> >>
> >
> >
> >
> > --
> > Whether it's science, technology, personal experience, true love,
> > astrology, or gut feelings, each of us has confidence in something that
> we
> > will never fully comprehend.
> >  --Roy H. William
>



--
Whether it's science, technology, personal experience, true love,
astrology, or gut feelings, each of us has confidence in something that we
will never fully comprehend.
 --Roy H. William
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: leap second bug

Jack Krupansky-2
Interesting:

"
The sequence of dates of the UTC second markers will be:

2012 June 30, 23h 59m 59s
2012 June 30, 23h 59m 60s
2012 July 1, 0h 0m 0s
"

See:
http://wwp.greenwichmeantime.com/info/leap-second.htm

So, there were two consecutive second " markers" which were literally
distinct, but numerically identical.

What "design pattern" for timing did Linux violate? In other words, what
lesson should we be learning to assure that we don't have a similar problem
at an application level on a future leap second?

-- Jack Krupansky

-----Original Message-----
From: Óscar Marín Miró
Sent: Sunday, July 01, 2012 11:02 AM
To: [hidden email]
Subject: Re: leap second bug

Thanks Michael, nice information :)

On Sun, Jul 1, 2012 at 5:29 PM, Michael McCandless <
[hidden email]> wrote:

> Looks like this is a low-level Linux issue ... see Shay's email to the
> ElasticSearch list about it:
>
>
> https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY
>
> Also see the comments here:
>
>      http://news.ycombinator.com/item?id=4182642
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sun, Jul 1, 2012 at 8:08 AM, Óscar Marín Miró
> <[hidden email]> wrote:
> > Hello Michael, thanks for the note :)
> >
> > I'm having a similar problem since yesterday, tomcats are wild on CPU
> [near
> > 100%]. Did your solr servers did not reply to index/query requests?
> >
> > Thanks :)
> >
> > On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <[hidden email]
> >wrote:
> >
> >> Our solr servers went into GC hell, and became non-responsive on date
> >> change today.
> >>
> >> Restarting tomcats did not help.
> >>
> >> Rebooting the machine did.
> >>
> >>
> >>
> http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
> >>
> >
> >
> >
> > --
> > Whether it's science, technology, personal experience, true love,
> > astrology, or gut feelings, each of us has confidence in something that
> we
> > will never fully comprehend.
> >  --Roy H. William
>



--
Whether it's science, technology, personal experience, true love,
astrology, or gut feelings, each of us has confidence in something that we
will never fully comprehend.
--Roy H. William

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: leap second bug

Óscar Marín Miró
So, this was the solution, sorry to post it so late, just in case it helps
anyone:

/etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date;
/etc/init.d/ntp start

And tomcat magically switched from 100% CPU to 0.5% :)

From:

https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY

[from Michael McCandless help on this thread]

On Sun, Jul 1, 2012 at 6:15 PM, Jack Krupansky <[hidden email]>wrote:

> Interesting:
>
> "
> The sequence of dates of the UTC second markers will be:
>
> 2012 June 30, 23h 59m 59s
> 2012 June 30, 23h 59m 60s
> 2012 July 1, 0h 0m 0s
> "
>
> See:
> http://wwp.greenwichmeantime.**com/info/leap-second.htm<http://wwp.greenwichmeantime.com/info/leap-second.htm>
>
> So, there were two consecutive second " markers" which were literally
> distinct, but numerically identical.
>
> What "design pattern" for timing did Linux violate? In other words, what
> lesson should we be learning to assure that we don't have a similar problem
> at an application level on a future leap second?
>
> -- Jack Krupansky
>
> -----Original Message----- From: Óscar Marín Miró
> Sent: Sunday, July 01, 2012 11:02 AM
> To: [hidden email]
> Subject: Re: leap second bug
>
>
> Thanks Michael, nice information :)
>
> On Sun, Jul 1, 2012 at 5:29 PM, Michael McCandless <
> [hidden email]> wrote:
>
>  Looks like this is a low-level Linux issue ... see Shay's email to the
>> ElasticSearch list about it:
>>
>>
>> https://groups.google.com/**forum/?fromgroups#!topic/**
>> elasticsearch/_I1_OfaL7QY<https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY>
>>
>> Also see the comments here:
>>
>>      http://news.ycombinator.com/**item?id=4182642<http://news.ycombinator.com/item?id=4182642>
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Sun, Jul 1, 2012 at 8:08 AM, Óscar Marín Miró
>> <[hidden email]> wrote:
>> > Hello Michael, thanks for the note :)
>> >
>> > I'm having a similar problem since yesterday, tomcats are wild on CPU
>> [near
>> > 100%]. Did your solr servers did not reply to index/query requests?
>> >
>> > Thanks :)
>> >
>> > On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <
>> [hidden email]
>> >wrote:
>> >
>> >> Our solr servers went into GC hell, and became non-responsive on date
>> >> change today.
>> >>
>> >> Restarting tomcats did not help.
>> >>
>> >> Rebooting the machine did.
>> >>
>> >>
>> >>
>> http://www.wired.com/**wiredenterprise/2012/07/leap-**
>> second-bug-wreaks-havoc-with-**java-linux/<http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/>
>> >>
>> >
>> >
>> >
>> > --
>> > Whether it's science, technology, personal experience, true love,
>> > astrology, or gut feelings, each of us has confidence in something that
>> we
>> > will never fully comprehend.
>> >  --Roy H. William
>>
>>
>
>
> --
> Whether it's science, technology, personal experience, true love,
> astrology, or gut feelings, each of us has confidence in something that we
> will never fully comprehend.
> --Roy H. William
>



--
Whether it's science, technology, personal experience, true love,
astrology, or gut feelings, each of us has confidence in something that we
will never fully comprehend.
 --Roy H. William
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: leap second bug

Michael Tsadikov
explanation of the cause:

https://lkml.org/lkml/2012/7/1/203

On Wed, Jul 4, 2012 at 1:48 AM, Óscar Marín Miró
<[hidden email]>wrote:

> So, this was the solution, sorry to post it so late, just in case it helps
> anyone:
>
> /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date;
> /etc/init.d/ntp start
>
> And tomcat magically switched from 100% CPU to 0.5% :)
>
> From:
>
>
> https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY
>
> [from Michael McCandless help on this thread]
>
> On Sun, Jul 1, 2012 at 6:15 PM, Jack Krupansky <[hidden email]
> >wrote:
>
> > Interesting:
> >
> > "
> > The sequence of dates of the UTC second markers will be:
> >
> > 2012 June 30, 23h 59m 59s
> > 2012 June 30, 23h 59m 60s
> > 2012 July 1, 0h 0m 0s
> > "
> >
> > See:
> > http://wwp.greenwichmeantime.**com/info/leap-second.htm<
> http://wwp.greenwichmeantime.com/info/leap-second.htm>
> >
> > So, there were two consecutive second " markers" which were literally
> > distinct, but numerically identical.
> >
> > What "design pattern" for timing did Linux violate? In other words, what
> > lesson should we be learning to assure that we don't have a similar
> problem
> > at an application level on a future leap second?
> >
> > -- Jack Krupansky
> >
> > -----Original Message----- From: Óscar Marín Miró
> > Sent: Sunday, July 01, 2012 11:02 AM
> > To: [hidden email]
> > Subject: Re: leap second bug
> >
> >
> > Thanks Michael, nice information :)
> >
> > On Sun, Jul 1, 2012 at 5:29 PM, Michael McCandless <
> > [hidden email]> wrote:
> >
> >  Looks like this is a low-level Linux issue ... see Shay's email to the
> >> ElasticSearch list about it:
> >>
> >>
> >> https://groups.google.com/**forum/?fromgroups#!topic/**
> >> elasticsearch/_I1_OfaL7QY<
> https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/_I1_OfaL7QY
> >
> >>
> >> Also see the comments here:
> >>
> >>      http://news.ycombinator.com/**item?id=4182642<
> http://news.ycombinator.com/item?id=4182642>
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >> On Sun, Jul 1, 2012 at 8:08 AM, Óscar Marín Miró
> >> <[hidden email]> wrote:
> >> > Hello Michael, thanks for the note :)
> >> >
> >> > I'm having a similar problem since yesterday, tomcats are wild on CPU
> >> [near
> >> > 100%]. Did your solr servers did not reply to index/query requests?
> >> >
> >> > Thanks :)
> >> >
> >> > On Sun, Jul 1, 2012 at 1:22 PM, Michael Tsadikov <
> >> [hidden email]
> >> >wrote:
> >> >
> >> >> Our solr servers went into GC hell, and became non-responsive on date
> >> >> change today.
> >> >>
> >> >> Restarting tomcats did not help.
> >> >>
> >> >> Rebooting the machine did.
> >> >>
> >> >>
> >> >>
> >> http://www.wired.com/**wiredenterprise/2012/07/leap-**
> >> second-bug-wreaks-havoc-with-**java-linux/<
> http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Whether it's science, technology, personal experience, true love,
> >> > astrology, or gut feelings, each of us has confidence in something
> that
> >> we
> >> > will never fully comprehend.
> >> >  --Roy H. William
> >>
> >>
> >
> >
> > --
> > Whether it's science, technology, personal experience, true love,
> > astrology, or gut feelings, each of us has confidence in something that
> we
> > will never fully comprehend.
> > --Roy H. William
> >
>
>
>
> --
> Whether it's science, technology, personal experience, true love,
> astrology, or gut feelings, each of us has confidence in something that we
> will never fully comprehend.
>  --Roy H. William
>
Loading...