Pages

Monday, September 15, 2014

Analyzing a Thread dump

As a java developers, we often see our applications working slow , performance decreases. During these times the first step is to take a Thread dump and analyze the dump.

What is a Thread Dump?
A TD is a snap shot of all the threads inside a JVM. A TD Provides us with all the threads that are currently running while the TD is taken. Analyzing the Threads obtained from the TD can give us enough information by which can identify the causes for the performance issues , slow down etc.

What does a TD Contains ?
A TD contains data about each thread including the name of the thread, priority, thread group, state (running/blocked/waiting/parking) as well as the execution stack in form of thread stack trace is included in the thread dump. Threads including VM threads, GC Threads, Applications Threads, Server Threads as well as Blocked Threads and also Dead Lock Threads.

Thread Types
Normally threads in java are divided into

A Daemon thread is a thread that does not prevent the JVM from exiting when the program finishes. An example is the GC thread. So even if the program that we are running finishes the GC thread keeps on running.

The thread running the 'static void main(String[] args)’ method is created as a non-daemon thread, and when this thread stops working, all other daemon threads will stop as well. (The thread running this main method is called the VM thread in HotSpot VM.)

Thread States
A Thread may be in any one of these states,

1.New
2.Runnable
3.Non-Runnable
          Sleep for a time duration
          Wait for a event
          Blocked for a lock
4.Dead

The below gives you more better view,
Most cases we look at the threads that are in Running state or in Non-Runnable(waiting) state inside the TD.

The Runnable says that is threads is actively running and performing a task where as a
Non-Runnable which includes ,
sleep for a time duration ,says that a thread has done some work and went to sleep for a specific duration
Wait of a event ,says that  the thread has been waiting for a Event to happen. Things like waiting for a disk I/O to complete or waiting for a KeyBoard Input like that.
Blocked, implies it cannot proceed with its work till it can obtain a lock which is currently held by someone else

Each of the Thread inside the Thread Dump will specify the State and also the name and its priority.

Thread Special States

interrupted : The user called java.lang.Thread.interrupt() on this thread.
Daemon : This is either JVM internal thread or a thread that was marked as a daemon thread by java.lang.Thread.setDaemon().
in native : This thread is executing native code: either user-supplied JNI code or JVM internal code.
in suspend critical mode : This thread is executing JVM internal code and has marked itself as suspend critical. Garbage collection is stopped for a specified time period.
native_blocked : This thread is executing JVM internal code and has tried to take a JVM internal lock. The thread is blocked because that lock is held by another thread.
native_waiting : This thread is executing JVM internal code and is waiting for notification from another thread about a JVM internal lock.

Consider there thread below which is followed by information about locks and a trace of the thread’s stack at the moment of the thread dump.

"Main Thread" id=1 idx=0x4 tid=8385 prio=5 alive, waiting, native_blocked
    -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0xb7aed058[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:811)
    ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0xb7aed058[fat lock]
    at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:459)
    at weblogic/Server.main(Server.java:67)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

The “Main Thread” is alive (running ) and and it is currently waiting for an object to be released (waiting).

The native_blocked is a special Thread State inside the JRocket which says “This thread is executing JVM internal code and has tried to take a JVM internal lock. The thread is blocked because that lock is held by another thread”.

How to Take Thread Dumps
Thread dumps can be taken using the jstack command available in java software.
Print Thread Stack For Java Methods
         Jstack <PID>(or jstack -F pid if jstack pid does not respond)

Print lock information for a process
         jstack -l <PID>

Print lock information for a process:
         jstack -l pid

The jstack utility can also be used to print a mixed stack, that is, it can print native stack frames in addition to the Java stack. Native frames are the C/C++ frames associated with VM code and JNI/native code.
         jstack -m pid

We need to make sure that when we use jstack to dump threads to a file ,the file extension should be ".td" which will allows thread analysis tools to process the file.

Analyzing a Thread Stack
Lets analyze how to extract information from a Thread Stack ,consider the below thread stack

"main" prio=10 tid=0x08828800 nid=0x11ec runnable [0xf7432000]
   java.lang.Thread.State: RUNNABLE
        at Struck.main(Struck.java:17)

Each Individual thread gives information like,

Thread Name: The Name of the thread which most of the us are interested , The name in this case is "main"

Thread Type: As we discussed earlier a thread type will be either daemon or non-daemon. In this case the thread type is a non-daemon thread. If we have any daemon threads we will see the type as "daemon" after the thread name

Thread Priority: every thread will have a Priority associated with that. The Thread priority associated here is  "prio=10". Each Java thread is given a numeric priority, between MIN_PRIORITY and MAX_PRIORITY (constants defined in class Thread). At any given time, when multiple threads are ready to be executed, the thread with the highest priority will be chosen for execution. Only when that thread stops, or is suspended for some reason, will a lower priority thread start executing.

Here is the Thread priority Table

 Thread.MIN_PRIORITY
 The maximum priority of any thread (an int value of 10)
 Thread.MAX_PRIORITY
 The minimum priority of any thread (an int value of 1)
 Thread.NORM_PRIORITY
 The normal priority of any thread (an int value of 5)

Thread ID: Every thread will have a Thread ID associated which is obtained via java.lang.Thread.getID() that can be used for thread identification. The thread ID in this case is "tid=0x08828800"

Native ID: Every thread is also associated with a Native ID that indicate threads from the OS perspective by which we can find which threads in the OS level are causing issues. The nid in this case is "nid=0x11ec"

Thread State: As we discussed earlier every thread will have a state, "Runnable" in this case

Object Monitor: Besides the Thread state ,it also shows us a Object monitor by which we can learn out the potential blocking condition. The Thread Condition in this case is
"runnable [0xf7432000]"

Stack Trace: this is by far the most important data that you will find from the Thread Dump. This is also where you will spent most of your analysis time since the Java Stack Trace provides you with 90% of the information that you need in order to pinpoint root cause of many problem pattern types.

Analyzing thread dump Log

Full thread dump Java HotSpot(TM) Server VM (14.2-b01 mixed mode) :
The First line in the thread dump tells us about the JVM type . This also says that this is a Full Thread dump

"VM Periodic Task Thread" prio=10 tid=0x088c5000 nid=0x11f6 waiting on condition
This is normally a Thread managed by the HotSpot VM in order to perform a OS level native Operations. This can be used when we see issues like High CPU in the Os level.

"VM Thread" prio=10 tid=0x088a4800 nid=0x11ef runnable
 The VM thread is a special thread in the HotSpot VM. It performs special tasks in the VM such as garbage collection.

"VM Periodic Task Thread" prio=10 tid=0x088c5000 nid=0x11f6 waiting on condition
this is C++ level thread which performs a number of tasks on systems where there is not a native timer interrupt. It appears to be mostly used for sampling/profiling what the JVM is doing.

"GC task thread#0 (ParallelGC)" prio=10 tid=0x0882f000 nid=0x11ed runnable
Garbage collection thread and the type of the garbage collector we use.

JNI global references: 595
JNI global references are object references from the native code to the java object managed by the Garbage collector. This technically says that object is being hold by the Native code

It is also important to keep an eye on JNI references in order to detect JNI related leaks. This can happen if you program use JNI directly or using third party tools like monitoring tools which are prone to native memory leaks.

Case Studies for analyzing a Thread dump are available too. 

happy learning :-)

5 comments :

  1. I like the helpful information you provide in your articles.
    I'll bookmark your weblog and check again here
    regularly. I'm quite sure I will learn lots of new
    stuff right here! Good luck for the next!

    my web-site: AC Repair Simi Valley

    ReplyDelete
  2. Hey there, You've done a great job. I will
    definitely digg it and personally recommend to my friends.

    I am sure they'll be benefited from this site.

    Feel free to visit my blog post - cialis explication

    ReplyDelete
  3. fastThread parses all formats of thread dumps and its a free online tool thread dump analyzer

    ReplyDelete
  4. does taking continuous threaddump make the process slow ??

    ReplyDelete
  5. Looks Pretty Good and at same time this post is very informative.World-class thread dump analyzer to fire fight any sort of JVM related availability, scalability and performance problems. Applies several intelligence patterns discovers the root cause of the problem instantly.Java Thread Dump Analyzer

    ReplyDelete