Analyzing a Thread dump

As a java developers, we often see our applications working slow , performance decreases. During these times the first step is to take a Thread dump and analyze the dump.

What is a Thread Dump?

A TD is a snap shot of all the threads inside a JVM. A TD Provides us with all the threads that are currently running while the TD is taken. Analyzing the Threads obtained from the TD can give us enough information by which can identify the causes for the performance issues , slow down etc.

What does a TD Contains ?

A TD contains data about each thread including the name of the thread, priority, thread group, state (running/blocked/waiting/parking) as well as the execution stack in form of thread stack trace is included in the thread dump. Threads including VM threads, GC Threads, Applications Threads, Server Threads as well as Blocked Threads and also Dead Lock Threads.

Thread Types

Normally threads in java are divided into

A Daemon thread is a thread that does not prevent the JVM from exiting when the program finishes. An example is the GC thread. So even if the program that we are running finishes the GC thread keeps on running.

The thread running the 'static void main(String[] args)’ method is created as a non-daemon thread, and when this thread stops working, all other daemon threads will stop as well. (The thread running this main method is called the VM thread in HotSpot VM.)

Thread States

A Thread may be in any one of these states,

1.New

2.Runnable

3.Non-Runnable

          Sleep for a time duration

          Wait for a event

          Blocked for a lock

4.Dead

The below gives you more better view,

Most cases we look at the threads that are in Running state or in Non-Runnable(waiting) state inside the TD.

The Runnable says that is threads is actively running and performing a task where as a

Non-Runnable which includes ,

sleep for a time duration ,says that a thread has done some work and went to sleep for a specific duration

Wait of a event ,says that  the thread has been waiting for a Event to happen. Things like waiting for a disk I/O to complete or waiting for a KeyBoard Input like that.

Blocked, implies it cannot proceed with its work till it can obtain a lock which is currently held by someone else

Each of the Thread inside the Thread Dump will specify the State and also the name and its priority.

Thread Special States

interrupted : The user called java.lang.Thread.interrupt() on this thread.

Daemon : This is either JVM internal thread or a thread that was marked as a daemon thread by java.lang.Thread.setDaemon().

in native : This thread is executing native code: either user-supplied JNI code or JVM internal code.

in suspend critical mode : This thread is executing JVM internal code and has marked itself as suspend critical. Garbage collection is stopped for a specified time period.

native_blocked : This thread is executing JVM internal code and has tried to take a JVM internal lock. The thread is blocked because that lock is held by another thread.

native_waiting : This thread is executing JVM internal code and is waiting for notification from another thread about a JVM internal lock.

Consider there thread below which is followed by information about locks and a trace of the thread’s stack at the moment of the thread dump.

"Main Thread" id=1 idx=0x4 tid=8385 prio=5 alive, waiting, native_blocked

    -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0xb7aed058[fat lock]

    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

    at java/lang/Object.wait(J)V(Native Method)

    at java/lang/Object.wait(Object.java:485)

    at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:811)

    ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0xb7aed058[fat lock]

    at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:459)

    at weblogic/Server.main(Server.java:67)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

The “Main Thread” is alive (running ) and and it is currently waiting for an object to be released (waiting).

The native_blocked is a special Thread State inside the JRocket which says “This thread is executing JVM internal code and has tried to take a JVM internal lock. The thread is blocked because that lock is held by another thread”.

How to Take Thread Dumps

Thread dumps can be taken using the jstack command available in java software.

Print Thread Stack For Java Methods

         Jstack <PID>(or jstack -F pid if jstack pid does not respond)

Print lock information for a process

         jstack -l <PID>

Print lock information for a process:

         jstack -l pid

The jstack utility can also be used to print a mixed stack, that is, it can print native stack frames in addition to the Java stack. Native frames are the C/C++ frames associated with VM code and JNI/native code.

         jstack -m pid

We need to make sure that when we use jstack to dump threads to a file ,the file extension should be ".td" which will allows thread analysis tools to process the file.

Analyzing a Thread Stack

Lets analyze how to extract information from a Thread Stack ,consider the below thread stack

"main" prio=10 tid=0x08828800 nid=0x11ec runnable [0xf7432000]

   java.lang.Thread.State: RUNNABLE

        at Struck.main(Struck.java:17)

Each Individual thread gives information like,

Thread Name: The Name of the thread which most of the us are interested , The name in this case is "main"

Thread Type: As we discussed earlier a thread type will be either daemon or non-daemon. In this case the thread type is a non-daemon thread. If we have any daemon threads we will see the type as "daemon" after the thread name

Thread Priority: every thread will have a Priority associated with that. The Thread priority associated here is  "prio=10". Each Java thread is given a numeric priority, between MIN_PRIORITY and MAX_PRIORITY (constants defined in class Thread). At any given time, when multiple threads are ready to be executed, the thread with the highest priority will be chosen for execution. Only when that thread stops, or is suspended for some reason, will a lower priority thread start executing.

Here is the Thread priority Table

Thread.MIN_PRIORITY	The maximum priority of any thread (an int value of 10)
Thread.MAX_PRIORITY	The minimum priority of any thread (an int value of 1)
Thread.NORM_PRIORITY	The normal priority of any thread (an int value of 5)

Thread ID: Every thread will have a Thread ID associated which is obtained via java.lang.Thread.getID() that can be used for thread identification. The thread ID in this case is "tid=0x08828800"

Native ID: Every thread is also associated with a Native ID that indicate threads from the OS perspective by which we can find which threads in the OS level are causing issues. The nid in this case is "nid=0x11ec"

Thread State: As we discussed earlier every thread will have a state, "Runnable" in this case

Object Monitor: Besides the Thread state ,it also shows us a Object monitor by which we can learn out the potential blocking condition. The Thread Condition in this case is

"runnable [0xf7432000]"

Stack Trace: this is by far the most important data that you will find from the Thread Dump. This is also where you will spent most of your analysis time since the Java Stack Trace provides you with 90% of the information that you need in order to pinpoint root cause of many problem pattern types.

Analyzing thread dump Log

Full thread dump Java HotSpot(TM) Server VM (14.2-b01 mixed mode) :

The First line in the thread dump tells us about the JVM type . This also says that this is a Full Thread dump

"VM Periodic Task Thread" prio=10 tid=0x088c5000 nid=0x11f6 waiting on condition

This is normally a Thread managed by the HotSpot VM in order to perform a OS level native Operations. This can be used when we see issues like High CPU in the Os level.

"VM Thread" prio=10 tid=0x088a4800 nid=0x11ef runnable

 The VM thread is a special thread in the HotSpot VM. It performs special tasks in the VM such as garbage collection.

"VM Periodic Task Thread" prio=10 tid=0x088c5000 nid=0x11f6 waiting on condition

this is C++ level thread which performs a number of tasks on systems where there is not a native timer interrupt. It appears to be mostly used for sampling/profiling what the JVM is doing.

"GC task thread#0 (ParallelGC)" prio=10 tid=0x0882f000 nid=0x11ed runnable

Garbage collection thread and the type of the garbage collector we use.

JNI global references: 595

JNI global references are object references from the native code to the java object managed by the Garbage collector. This technically says that object is being hold by the Native code

It is also important to keep an eye on JNI references in order to detect JNI related leaks. This can happen if you program use JNI directly or using third party tools like monitoring tools which are prone to native memory leaks.

Case Studies for analyzing a Thread dump are available too. 

happy learning :-)