Core
Files are very important .This post tells you on how to create core
dumps on linux and use java tools to extract information from them.
What
is a Core File
A
core file is created when a program terminates unexpectedly, due to a
bug, or a violation of the operating system's or hardware's
protection mechanisms. The operating system kills the program and
creates a core file that programmers can use to figure out what went
wrong. It contains a detailed description of the state that the
program was in when it died.
Generate
a Core File
[root@vx111a
bin]# gdb --pid=3853
Loaded
symbols for /lib64/libgcc_s.so.1
0x0000003b7d40803d
in pthread_join () from /lib64/libpthread.so.0
Missing
separate debuginfos, use: debuginfo-install
glibc-2.12-1.25.el6.x86_64 libgcc-4.4.5-6.el6.x86_64
(gdb)
gcore tomcat.core
Saved
corefile tomcat.core
(gdb)
detach
Detaching
from program: /soa/jdk1.7.0_25/bin/java, process 3853
(gdb)
quit
The
PID 3853 belong to a Tomcat Server Process running on Linux. the core
file by the name tomcat.core is generated.
Use
Java Tools to Analyze Core Files
Once
the Core file is generated ,we can use tools available in Java JDK to
analyze various information
JMAP
Use
jmap command to generate heap.dump file from the core Dump like
[root@vx111a
bin]# jmap -heap:format=b /soa/jdk1.7.0_25/java tomcat.core
Attaching
to core tomcat.core from executable /soa/jdk1.7.0_25/java, please
wait...
Error
attaching to core file: Can't attach to the core file
If
you see the “Error attaching to core file: Can't attach to the core
file” , we need to find which executable is used for the generation
of the core file
[root@vx111a
bin]# gdb --core=tomcat.core
GNU
gdb (GDB) Red Hat Enterprise Linux (7.2-48.el6)
Copyright
(C) 2010 Free Software Foundation, Inc.
License
GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This
is free software: you are free to change and redistribute it.
There
is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and
"show warranty" for details.
This
GDB was configured as "x86_64-redhat-linux-gnu".
For
bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Missing
separate debuginfo for the main executable file
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/a5/58f547fe0b95fdc6a109cb7d9692d6d7969794
[New
Thread 3854]
[New
Thread 3855]
[New
Thread 3856]
[New
Thread 3857]
[New
Thread 3858]
[New
Thread 3859]
[New
Thread 3860]
[New
Thread 3861]
[New
Thread 3862]
[New
Thread 3863]
[New
Thread 3864]
[New
Thread 3865]
[New
Thread 3866]
[New
Thread 3868]
[New
Thread 3869]
[New
Thread 3870]
[New
Thread 3872]
[New
Thread 3873]
[New
Thread 3874]
[New
Thread 3875]
[New
Thread 3876]
[New
Thread 3877]
[New
Thread 3878]
[New
Thread 3853]
Core
was generated by `/soa/jdk1.7.0_25//bin/java
-Djava.util.logging.config.file=/soa/apache-tomcat-7.'.
#0
0x0000003f3f6dd1e3 in ?? ()
We
can see that the core was generated using /soa/jdk1.7.0_25//bin/java
Use
jmap with the same executable as
[root@vx111a
bin]# jmap -heap:format=b /soa/jdk1.7.0_25//bin/java tomcat.core
Attaching
to core tomcat.core from executable /soa/jdk1.7.0_25//bin/java,
please wait...
Debugger
attached successfully.
Server
compiler detected.
JVM
version is 23.25-b01
Dumping
heap to heap.bin ...
none
null_check
null_assert
range_check
class_check
array_check
intrinsic
bimorphic
unloaded
uninitialized
unreached
unhandled
constraint
div0_check
age
predicate
loop_limit_check
Finding
object size using Printezis bits and skipping over...
Heap
dump file created
Attach
a Debugger
[root@vx111a
bin]# jmap -J-d64 $JAVA_HOME/bin/java ./tomcat.core
Attaching
to core ./tomcat.core from executable /soa/jdk1.7.0_25//bin/java,
please wait...
Debugger
attached successfully.
Server
compiler detected.
JVM
version is 23.25-b01
0x0000000000400000
7K /soa/jdk1.7.0_25//bin/java
0x0000003b7d400000
142K /lib64/libpthread.so.0
0x00007fcaf1844000
103K /soa/jdk1.7.0_25/bin/../jre/lib/amd64/jli/libjli.so
0x0000003f3f200000
22K /lib64/libdl.so.2
0x0000003f3f600000
1868K /lib64/libc.so.6
0x0000003f3ee00000
152K /lib64/ld-linux-x86-64.so.2
0x00007fcaf0b21000
13190K /soa/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so
0x0000003f3fe00000
584K /lib64/libm.so.6
0x000000363a400000
45K /lib64/librt.so.1
0x00007fcaf0812000
63K /soa/jdk1.7.0_25/jre/lib/amd64/libverify.so
0x00007fcaf05e7000
214K /soa/jdk1.7.0_25/jre/lib/amd64/libjava.so
0x00007fcaf03c8000
60K /lib64/libnss_files.so.2
0x00007fcaf01ad000
120K /soa/jdk1.7.0_25/jre/lib/amd64/libzip.so
0x00007fcadaebf000
89K /soa/jdk1.7.0_25/jre/lib/amd64/libnio.so
0x00007fcadaca9000
107K /soa/jdk1.7.0_25/jre/lib/amd64/libnet.so
0x00007fcadaaa1000
44K /soa/jdk1.7.0_25/jre/lib/amd64/libmanagement.so
0x00007fcada755000
250K /soa/jdk1.7.0_25/jre/lib/amd64/libsunec.so
0x0000003f4aa00000
91K /lib64/libgcc_s.so.1
Obtain
Thread Data
[root@vx111a
bin]# jstack -J-d64 $JAVA_HOME/bin/java ./tomcat.core
Attaching
to core ./tomcat.core from executable /soa/jdk1.7.0_25//bin/java,
please wait...
Debugger
attached successfully.
Server
compiler detected.
JVM
version is 23.25-b01
Deadlock
Detection:
No
deadlocks found.
Thread
3878: (state = BLOCKED)
-
java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
-
org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run() @bci=13,
line=148 (Interpreted frame)
-
java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
Thread
3877: (state = IN_NATIVE)
-
java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0
(Interpreted frame)
-
java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7,
line=398 (Interpreted frame)
-
java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530
(Interpreted frame)
-
java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame)
-
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(java.net.ServerSocket)
@bci=1, line=60 (Interpreted frame)
-
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run() @bci=95,
line=216 (Interpreted frame)
-
java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
Thread
3876: (state = BLOCKED)
-
java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
-
org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run() @bci=13,
line=148 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
Attach
a OS Debugger
[root@vx111a
bin]# gdb --core=tomcat.core
GNU
gdb (GDB) Red Hat Enterprise Linux (7.2-48.el6)
Copyright
(C) 2010 Free Software Foundation, Inc.
License
GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This
is free software: you are free to change and redistribute it.
There
is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and
"show warranty" for details.
This
GDB was configured as "x86_64-redhat-linux-gnu".
For
bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Missing
separate debuginfo for the main executable file
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/a5/58f547fe0b95fdc6a109cb7d9692d6d7969794
[New
Thread 3854]
[New
Thread 3855]
[New
Thread 3856]
[New
Thread 3875]
[New
Thread 3876]
[New
Thread 3877]
[New
Thread 3878]
[New
Thread 3853]
Core
was generated by `/soa/jdk1.7.0_25//bin/java
-Djava.util.logging.config.file=/soa/apache-tomcat-7.'.
#0
0x0000003f3f6dd1e3 in ?? ()
[root@vx111a
bin]# gdb /soa/jdk1.7.0_25//bin/java tomcat.core
GNU
gdb (GDB) Red Hat Enterprise Linux (7.2-48.el6)
Copyright
(C) 2010 Free Software Foundation, Inc.
License
GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This
is free software: you are free to change and redistribute it.
There
is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and
"show warranty" for details.
This
GDB was configured as "x86_64-redhat-linux-gnu".
For
bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading
symbols from /soa/jdk1.7.0_25/bin/java...Missing separate debuginfo
for /soa/jdk1.7.0_25/bin/java
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/a5/58f547fe0b95fdc6a109cb7d9692d6d7969794.debug
(no
debugging symbols found)...done.
[New
Thread 3854]
[New
Thread 3855]
[New
Thread 3856]
[New
Thread 3857]
[New
Thread 3858]
[New
Thread 3859]
[New
Thread 3860]
[New
Thread 3861]
[New
Thread 3862]
[New
Thread 3863]
[New
Thread 3864]
[New
Thread 3865]
[New
Thread 3866]
[New
Thread 3868]
[New
Thread 3869]
[New
Thread 3870]
[New
Thread 3872]
[New
Thread 3873]
[New
Thread 3874]
[New
Thread 3875]
[New
Thread 3876]
[New
Thread 3877]
[New
Thread 3878]
[New
Thread 3853]
Missing
separate debuginfo for
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/5f/59955605983224cffb569cd2a6dcbfdc49fab5
Missing
separate debuginfo for
/soa/jdk1.7.0_25/bin/../jre/lib/amd64/jli/libjli.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/6b/f75f801995a9ae7fd7dd305d6cb1aecb0b7fde
Missing
separate debuginfo for
/soa/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/57/1647724f8fbef4d5a7562a7c58ac712dd266c7
Missing
separate debuginfo for /soa/jdk1.7.0_25/jre/lib/amd64/libverify.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/d8/edd7f1210cf4737ed4824e61722955eba30391
Missing
separate debuginfo for /soa/jdk1.7.0_25/jre/lib/amd64/libjava.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/75/0dc3133d98127f1bfa1431b9ef392101e32bc1
Missing
separate debuginfo for /soa/jdk1.7.0_25/jre/lib/amd64/libzip.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/84/d3b3792145096e1ee056ab0a45cf1ae1555d64
Missing
separate debuginfo for /soa/jdk1.7.0_25/jre/lib/amd64/libnio.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/47/331827d488e43168f261413ef0bd963617a3ad
Missing
separate debuginfo for /soa/jdk1.7.0_25/jre/lib/amd64/libnet.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/58/53fa2e06c1c4d8ddd304202500591393ac6e3e
Missing
separate debuginfo for
/soa/jdk1.7.0_25/jre/lib/amd64/libmanagement.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/cf/493798bafc47fd08e8c4f7efcfa06b77df3643
Missing
separate debuginfo for /soa/jdk1.7.0_25/jre/lib/amd64/libsunec.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/37/5917a2e1167ecd47d97191d9f02ebb9b20bbc7
Reading
symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread
debugging using libthread_db enabled]
Loaded
symbols for /lib64/libpthread.so.0
Reading
symbols from
/soa/jdk1.7.0_25/bin/../jre/lib/amd64/jli/libjli.so...(no debugging
symbols found)...done.
Loaded
symbols for /soa/jdk1.7.0_25/bin/../jre/lib/amd64/jli/libjli.so
Reading
symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded
symbols for /lib64/libdl.so.2
Reading
symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded
symbols for /lib64/libc.so.6
Reading
symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded
symbols for /lib64/ld-linux-x86-64.so.2
Reading
symbols from
/soa/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so...Missing separate
debuginfo for /soa/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/57/1647724f8fbef4d5a7562a7c58ac712dd266c7.debug
(no
debugging symbols found)...done.
Loaded
symbols for /soa/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so
Reading
symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded
symbols for /lib64/libm.so.6
Reading
symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded
symbols for /lib64/librt.so.1
Reading
symbols from /soa/jdk1.7.0_25/jre/lib/amd64/libverify.so...Missing
separate debuginfo for /soa/jdk1.7.0_25/jre/lib/amd64/libverify.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/d8/edd7f1210cf4737ed4824e61722955eba30391.debug
(no
debugging symbols found)...done.
Loaded
symbols for /soa/jdk1.7.0_25/jre/lib/amd64/libverify.so
Reading
symbols from /soa/jdk1.7.0_25/jre/lib/amd64/libjava.so...(no
debugging symbols found)...done.
Loaded
symbols for /soa/jdk1.7.0_25/jre/lib/amd64/libjava.so
Reading
symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded
symbols for /lib64/libnss_files.so.2
Reading
symbols from /soa/jdk1.7.0_25/jre/lib/amd64/libzip.so...(no debugging
symbols found)...done.
Loaded
symbols for /soa/jdk1.7.0_25/jre/lib/amd64/libzip.so
Reading
symbols from /soa/jdk1.7.0_25/jre/lib/amd64/libnio.so...(no debugging
symbols found)...done.
Loaded
symbols for /soa/jdk1.7.0_25/jre/lib/amd64/libnio.so
Reading
symbols from /soa/jdk1.7.0_25/jre/lib/amd64/libnet.so...(no debugging
symbols found)...done.
Loaded
symbols for /soa/jdk1.7.0_25/jre/lib/amd64/libnet.so
Reading
symbols from
/soa/jdk1.7.0_25/jre/lib/amd64/libmanagement.so...Missing separate
debuginfo for /soa/jdk1.7.0_25/jre/lib/amd64/libmanagement.so
Try:
yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/cf/493798bafc47fd08e8c4f7efcfa06b77df3643.debug
(no
debugging symbols found)...done.
Loaded
symbols for /soa/jdk1.7.0_25/jre/lib/amd64/libmanagement.so
Reading
symbols from /soa/jdk1.7.0_25/jre/lib/amd64/libsunec.so...(no
debugging symbols found)...done.
Loaded
symbols for /soa/jdk1.7.0_25/jre/lib/amd64/libsunec.so
Reading
symbols from /lib64/libgcc_s.so.1...(no debugging symbols
found)...done.
Loaded
symbols for /lib64/libgcc_s.so.1
Core
was generated by `/soa/jdk1.7.0_25//bin/java
-Djava.util.logging.config.file=/soa/apache-tomcat-7.'.
#0
0x0000003f3f6dd1e3 in poll () from /lib64/libc.so.6
Missing
separate debuginfos, use: debuginfo-install
glibc-2.12-1.25.el6.x86_64 libgcc-4.4.5-6.el6.x86_64
(gdb)
where
#0
0x0000003f3f6dd1e3 in poll () from /lib64/libc.so.6
#1
0x00007fcadacb5563 in NET_Timeout () from
/soa/jdk1.7.0_25/jre/lib/amd64/libnet.so
#2
0x00007fcadacb68d7 in Java_java_net_PlainSocketImpl_socketAccept ()
from /soa/jdk1.7.0_25/jre/lib/amd64/libnet.so
#3
0x00007fcae9011f90 in ?? ()
#4
0x00000000e2b0b680 in ?? ()
#5
0x00000000e2b0b6c8 in ?? ()
#6
0x0000000000000001 in ?? ()
#7
0x00007fcaf0b1eea8 in ?? ()
#8
0x0000000000000000 in ?? ()
(gdb)
quit
More
to come on this , Happy learning
