Analyzing Slow Response – Case 2 (a High CPU Usage)

Recently we observed a strange issue with one of our Tomcat server in Production. The monitors that we configured on these always showed degraded performance. The HTTP Ping monitor that we configured has a Success status when it responds in less than 2 seconds. But from the last couple of days we were seeing alerts as the HTTP Ping Monitor is taking more than 6 Seconds and to respond causing the alert to fail.

1) We started to analyze the performance of the server by using the basic top command which gave us the highest CPU and Memory Used process like,

localHost $ top -n1 | grep -m1 java

5944  root   20   0 7057m 4.3g 7444 S 312.1 55.9  13871:55 java

We obtained the Process that was actually using the High CPU and amount of the Memory being used. In this case the CPU being used is 312.1

2) Next step is we started to find out the Thread ID that is causing the HIGH CPU usage in this process which gave us this

localHost $ top -n1 -H | grep -m1 java | perl -pe 's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' '

5958

3) In the next process we took the thread dump of the PID 5944 (running Tomcat Process ID ) and obtained the NativeID for the Thread ID using

localHost $ printf "%x" 5958

1746

4) From the Thread Dump we observed that this thread belongs to Garbage Collection

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f306c025800 nid=0x1746 runnable

And some times it pointed us to

"VM Thread" prio=10 tid=0x00007f306c134800 nid=0x1749 runnable

The VM thread is a special thread in the HotSpot VM. It performs special tasks in the VM such as garbage collection. If the VM_Operation suggests that the operation is a garbage collection then it is possible that you have encountered as issue such as heap corruption.

The High CPU usage might be  a Garbage collector issue but it can be some thing else that leave object references in the heap in in-correct state.

The main important thing at this point is to collect as much information as possible about the environment and try out possible workarounds.

Now upon checking the GC status we observed that the memory is totally exhausted and the GC is performing high number of Full GC’s

localHost $ jstat -gcutil  5944

  S0      S1     E      O        P        YGC     YGCT       FGC    FGCT          GCT

  0.00   0.00  70.12 100.00  99.66  57691 4817.991 30178 107006.562 111824.553

In the above case , the FGC element tells us the number of Full GC Caused during the life time.

Since the JVM is busy handling the Full GC of this process, it is actually pausing the other threads to perform the GC events hence causing the slow response to the EMS monitor.

More to come, Hope this helps J

Read More

HTTP, HTTPS and AJP

In a normally web cases, there are certain protocols that we use for communicating with web server from clients. When I said there are certain it means that we have Http,Https and Ajp. In this article we will see the comparison between these and when can be used at what time.

What is HTTP?

The Hypertext Transfer Protocol (HTTP) is an application protocol that allows communication between nodes. Hypertext is structured text that uses logical links between nodes containing text. HTTP is the protocol to exchange or transfer hypertext.

This is the basic protocol we use when working with a Server. This HTTP request contains a series of lines of data with every line terminated with a new line. The lines include request method (get or Post), followed by required resource along with protocol version (HTTP 1.1). The next lines include host being contacted, headers, cookies, browser details like type, preferred language and other details.

Once the request is received and processed a response is sent. The response contains a header block and data block. The first line in the header includes a response code which indicates the status of the request. The response code includes- 3 digit number in the following ranges:
200 and up - success; expect good data to follow
300 and up - good request but only headers (no data). e.g. page has moved
400 and up - error in request. e.g. request was for missing page (404)
500 and up - error in handling request. e.g. program on server has syntax error

The header block also contain details telling about the receiving system, content type  (MIME) which allows the receiving system to know whether to handle the resource in HTML or some other. Then there's a blank line and the actual data.

What is HTTPS?

HTTPS is similar to HTTP with all the above details but it includes extra details about the SSL configuration. What this does is the data encryption. The data is encrypted at the client end and decrypted at the receiving end. This ensures data security. This is all done by an agreement between the client and server before talking to one another.

What is AJP?

As we said the web traffic is carried out in HTTP which is a plain text protocol. The plain text format where actual words make up the message can create a Overhead of traffic. AJP takes care of this by converting these messages into bi nary code thus reducing the amount of space taken by each message.

AJP or Apache JServ Protocol, is an optimized binary version of HTTP that is typically used to allow Tomcat to communicate with an Apache web server.  AJP Connectors are most commonly implemented in Tomcat through the plug-in technology mod_jk, a re-write of the defunct mod_jserv plug-in with extensive optimization.

This is mainly used when we need to integrate server with Apache Web servers and you want Apache to handle the static content contained in the web application, and/or utilize Apache's SSL processing. This connector supports load balancing when used in conjunction with the jvmRoute attribute of the Engine.

Which to Use?

For most requirements, http is enough and if you want to secure things we can use Https. AJP is particularly used for "Load Balancing." This is where one server distributes incoming traffic among several application servers, sharing the load equally.

More to Come, Happy Learning J

Read More