Java应用程序假装死亡。(The java application feign death. Jstack can make application recovery?)

Java应用程序假装死亡。(The java application feign death. Jstack can make application recovery?)

今天，我发现我的应用程序无法访问。然后我登录到我的服务器。我发现应用程序的线程正常，但CPU负载过高。

然后，我想使用jstack命令。但它指出need -F option 。现在我发现GC日志停留在[GC (Allocation Failure) 2015-10-10T10:18:10.564+0800: 71147.518: [ParNew: ，并且没有其他日志。

然后，我使用jstack -F命令。发生事情发生。CPU正常，服务器也正常，GC日志也正常，第一行是[GC (Allocation Failure) 2015-10-10T10:17:50.757+0800: 53501.137: [ParNew: 210022K->245K(235968K), 369.6907808 secs] 400188K->1 90410K(1022400K), 369.6909604 secs] [Times: user=3475.15 sys=11.69, real=369.63 secs] 。 jstack的结果的可运行线程都是GC线程，如"Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f5f10021800 nid=0x6477 runnable

这是我的JAVA_OPTIONS

-XX：+ UsePerNewGC -XX：+ UseConcMarkSweepGC -XX：+ CMSParallelRemarkEnabled -XX：SurvivorRatio = 8 \ -XX：+ UseCMSInitiatingOccupancyOnly -XX：CMSInitiatingOccupancyFraction = UTF-8 -XMS2g -Xmx2g -Xmn512m -XX：+ UseParNewGC -XX： 75 \ -XX：+ PrintGCDateStamps -verbose：gc -XX：+ PrintGCDetails -Xloggc：/data/logs/gc-skynetlog_web.log \ -XX：+ UseGCLogFileRotation -XX：NumberOfGCLogFiles = 5 -XX：GCLogFileSize = 20M \ -XX ：+ HeapDumpOnOutOfMemoryError -XX：HeapDumpPath = / data / logs / dump_skynetlog_web.hprof

Today i found that my application could not be accessed.Then i login in my server.I found that the application's thread was normal,But the CPU load was too high.

Then,i wanted to use jstack command.But it pointed out need -F option.And now i found the GC log stay in [GC (Allocation Failure) 2015-10-10T10:18:10.564+0800: 71147.518: [ParNew:, and there is no other logs.

Then,i used jstack -F command.Strange things happened.The CPU became normal,and my server also became normal,and The GC log also nomal,and the first line was [GC (Allocation Failure) 2015-10-10T10:17:50.757+0800: 53501.137: [ParNew: 210022K->245K(235968K), 369.6907808 secs] 400188K->1 90410K(1022400K), 369.6909604 secs] [Times: user=3475.15 sys=11.69, real=369.63 secs].And the jstack's result's runnable thread were all GC thread,like "Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f5f10021800 nid=0x6477 runnable

Here are my JAVA_OPTIONS

-server -Dfile.encoding=UTF-8 \ -Xms2g -Xmx2g -Xmn512m \ -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 \ -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75 \ -XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xloggc:/data/logs/gc-skynetlog_web.log \ -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M \ -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/logs/dump_skynetlog_web.hprof

最满意答案

有一个新的bug影响Linux内核3.14 - 3.18

症状，线程在同步原语上阻塞时消耗100％。 发送信号（如jstack -F确实）正在解决这种情况。 对于JVM，如果GC线程受此影响，应用程序将永远停留在STW暂停状态。

Gil Tene在这篇文章中可以找到更多细节

There is a fresh bug affecting Linux kernels 3.14 - 3.18

Symptoms, thread is consuming 100% while blocking on synchronization primitive. Sending signal (like jstack -F does) is fixing this situation. In case of JVM, if GC thread is affect by this, application will stay in STW pause forever.

You can find more details in this post by Gil Tene