Pages

Monday, May 13, 2013

Linux Core and Java JMAP


When Generating heap dumps jmap blocks, and with large heaps  this can take a very long time. In these cases it is often much faster to get a core and then run jmap to extract a heap dump from the core. It is typically best to create the heap dump on the same box where the core was created to avoid environment differences.

So what is Core Dump?
When a program terminated abnormally, the status of the program at the point of termination should be saved some where for further analysis.This status is recorded in the form of a Core Dump File.

The core dump file contains details of where the abnormal termination happened, process stack, symbol table etc.

How can I make sure the core dump will be saved?
Each process has a limit for how big this core can be. If the limit is exceeded no core dump will be saved. By default this limit is 0 , that means no core will be dumped by default.

We need to use the “ulimit” command in linux to find the core file limitations. The “Ulimit” command sets various limits for the current process.

Check the core file size limitations
[root@vx111a ~]# ulimit -a | grep core
core file size          (blocks, -c) 0

Since “0” nothing can be saved.

[root@vx111a ~]# ulimit -c unlimited
Change the limitation to unlimited

[root@vx111a ~]# ulimit -a | grep core
core file size          (blocks, -c) unlimited

Once we change the limitation, we generate a core dump using the “gdb” command available in linux.

GDB command is to allow you to see what is going on inside another program while it executes or what another program was doing at the moment it crashed.

So I started a Java Class called “TestOome” like
[root@vx111a ~]# /usr/jdk1.6.0_14/bin/java -Xms1500m -Xmx1500m TestOome &
[1] 4588

Now iam going to attach the gdb to the process 4588 like

[root@vx111a ~]# gdb --pid=4588
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 4588
Reading symbols from /usr/jdk1.6.0_14/bin/java...(no debugging symbols found)...done.
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x51953b90 (LWP 4599)]
[New Thread 0x519a4b90 (LWP 4598)]
[New Thread 0x51a25b90 (LWP 4597)]
[New Thread 0x51aa6b90 (LWP 4596)]
[New Thread 0x51af7b90 (LWP 4595)]
[New Thread 0x51d48b90 (LWP 4594)]
[New Thread 0x51d99b90 (LWP 4593)]
[New Thread 0x51e1ab90 (LWP 4592)]
[New Thread 0x52064b90 (LWP 4591)]
[New Thread 0x520e5b90 (LWP 4590)]
[New Thread 0xb7419b90 (LWP 4589)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /usr/jdk1.6.0_14/bin/../jre/lib/i386/jli/libjli.so...(no debugging symbols found)...done.
Loaded symbols for /usr/jdk1.6.0_14/bin/../jre/lib/i386/jli/libjli.so
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/jdk1.6.0_14/jre/lib/i386/server/libjvm.so...(no debugging symbols found)...done.
Loaded symbols for /usr/jdk1.6.0_14/jre/lib/i386/server/libjvm.so
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/jdk1.6.0_14/jre/lib/i386/libverify.so...(no debugging symbols found)...done.
Loaded symbols for /usr/jdk1.6.0_14/jre/lib/i386/libverify.so
Reading symbols from /usr/jdk1.6.0_14/jre/lib/i386/libjava.so...(no debugging symbols found)...done.
Loaded symbols for /usr/jdk1.6.0_14/jre/lib/i386/libjava.so
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /usr/jdk1.6.0_14/jre/lib/i386/native_threads/libhpi.so...(no debugging symbols found)...done.
Loaded symbols for /usr/jdk1.6.0_14/jre/lib/i386/native_threads/libhpi.so
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/jdk1.6.0_14/jre/lib/i386/libzip.so...(no debugging symbols found)...done.
Loaded symbols for /usr/jdk1.6.0_14/jre/lib/i386/libzip.so
0xb7f72402 in __kernel_vsyscall ()
(gdb) gcore
Saved corefile core.4588
(gdb) detach
Detaching from program: /usr/jdk1.6.0_14/bin/java, process 4588
(gdb) quit

The last 3 commands are basically important ,
Gcore : generated the Core file
Detach : detaches the gdb from process ID 4588
Quit : quit

Now we can see that the core file Is generated like
[root@vx111a ~]# file core.4588
core.4588: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'java'

There is another way to generate the Core file using
[root@vx111a ~]# kill -SIGABRT 4705


Now we need to create a heap dump from this core file usimg “jmap” like

jmap -heap:format=b JAVA_HOME/bin/java COREFILE > heap.hprof 2>&1

Now we can perform many other function on this core file like

To get Thread Dumps information
[root@vx111a ~]# /usr/jdk1.6.0_14/bin/jstack  /usr/jdk1.6.0_14/bin/java core.4588
Attaching to core core.4588 from executable /usr/jdk1.6.0_14/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 14.0-b16
Deadlock Detection:

No deadlocks found.

Thread 4595: (state = BLOCKED)

Thread 4594: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)


Thread 4593: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Interpreted frame)


Thread 4589: (state = BLOCKED)
 - java.util.Arrays.copyOf(java.lang.Object[], int, java.lang.Class) @bci=8, line=2760 (Interpreted frame)
 - java.util.Arrays.copyOf(java.lang.Object[], int) @bci=6, line=2734 (Interpreted frame)
 - java.util.ArrayList.ensureCapacity(int) @bci=51, line=167 (Compiled frame)
 - java.util.ArrayList.add(java.lang.Object) @bci=7, line=351 (Compiled frame)
 - TestOome.main(java.lang.String[]) @bci=23, line=15 (Compiled frame)

To get the Heap Details ,jmap examnes a core file and prints out the shared object memory maps or heap details

[root@vx111a ~]# /usr/jdk1.6.0_14/bin/jmap  /usr/jdk1.6.0_14/bin/java core.4588
Attaching to core core.4588 from executable /usr/jdk1.6.0_14/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 14.0-b16
0x08048000      46K     /usr/jdk1.6.0_14/bin/java
0x00887000      134K    /lib/libpthread.so.0
0xb7f58000      37K     /usr/jdk1.6.0_14/bin/../jre/lib/i386/jli/libjli.so
0x00880000      20K     /lib/libdl.so.2
0x006fa000      1654K   /lib/libc.so.6
0x006db000      126K    /lib/ld-linux.so.2
0xb741a000      8965K   /usr/jdk1.6.0_14/jre/lib/i386/server/libjvm.so
0x00855000      211K    /lib/libm.so.6
0x008b7000      47K     /lib/librt.so.1
0xb7f65000      55K     /usr/jdk1.6.0_14/jre/lib/i386/libverify.so
0xb73a4000      184K    /usr/jdk1.6.0_14/jre/lib/i386/libjava.so
0x00924000      107K    /lib/libnsl.so.1
0xb739d000      37K     /usr/jdk1.6.0_14/jre/lib/i386/native_threads/libhpi.so
0xb7381000      49K     /lib/libnss_files.so.2
0xb7370000      74K     /usr/jdk1.6.0_14/jre/lib/i386/libzip.so

Happy learning J