Thursday, January 31, 2013

Here a Few Java Commands that Can Use Full

Hung, Dejava.lang.OutOfMemoryError: nativeGetNewTLAadlocked, or Looping Process

Print thread stack for all Java threads:
kill -QUIT pid
jstack pid (or jstack -F pid if jstack pid does not respond)

Detect deadlocks:
Request deadlock detection: JConsole tool, Threads tab
Print information on deadlocked threads: Control-\
Print list of concurrent locks owned by each thread: -XX:+PrintConcurrentLocks set, then Control-\
Print lock information for a process: jstack -l pid

Get a heap histogram for a process:
Start Java process with -XX:+PrintClassHistogram, then Control-\
jmap -histo pid (with -F option if pid does not respond)

Dump Java heap for a process in binary format to file:
jmap -dump:format=b,file= filename pid (with -F option if pid does not respond)

Print shared object mappings for a process:
jmap pid

Print heap summary for a process:
jmap -heap pid

Print finalization information for a process:
jmap -finalizerinfo pid

Attach the command-line debugger to a process:
jdb -connect sun.jvm.hotspot.jdi.SAPIDAttachingConnector:pid= pid

Post-mortem Diagnostics, Memory Leaks

  • Examine the fatal error log file. Default file name is hs_err_pid pid .log in the working-directory.
  • Create a heap dump:
    • Start the application with HPROF enabled: java -agentlib:hprof=file= file,format=b application; then Control-\
    • Start the application with HPROF enabled: java -agentlib:hprof=heap=dumpapplication
    • JConsole tool, MBeans tab
    • Start VM with -XX:+HeapDumpOnOutOfMemoryError; if OutOfMemoryError is thrown, VM generates a heap dump.
  • Browse Java heap dump:
    • jhat heap-dump-file
  • Dump Java heap from core file in binary format to a file:
    • jmap -dump:format=b,file= filename corefile
  • Get a heap histogram for a process:
    • Start Java process with -XX:+PrintClassHistogram, then Control-\
    • jmap -histo pid (with -F option if pid does not respond)
  • Get a heap histogram from a core file:
    • jmap -histo corefile
  • Print shared object mappings from a core file:
    • jmap corefile
  • Print heap summary from a core file:
    • jmap -heap corefile
  • Print finalization information from a core file:
    • jmap -finalizerinfo corefile
  • Print Java configuration information from a core file:
    • jinfo corefile
  • Print thread trace from a core file:
    • jstack corefile
  • Print lock information from a core file:
    • jstack -l corefile
  • Attach the command-line debugger to a core file on the same machine:
    • jdb -connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:javaExecutable= path,core= corefile
  • Attach the command-line debugger to a core file on a different machine:
    • On the machine with the core file: jsadebugd path corefile
      and on the machine with the debugger: jdb -connect sun.jvm.hotspot.jdi.SADebugServerAttachingConnector:debugServerName=
  • libumem can be used to debug memory leaks.


Note: The vmID argument for the jstat command is the virtual machine identifier.

  • Print statistics on the class loader:
    • jstat -class vmID
  • Print statistics on the compiler:
    • Compiler behavior: jstat -compiler vmID
    • Compilation method statistics: jstat -printcompilation vmID
  • Print statistics on garbage collection:
    • Summary of statistics: jstat -gcutil vmID
    • Summary of statistics, with causes: jstat -gccause vmID
    • Behavior of the gc heap: jstat -gc vmID
    • Capacities of all the generations: jstat -gccapacity vmID
    • Behavior of the new generation: jstat -gcnew vmID
    • Capacity of the new generation: jstat -gcnewcapacity vmID
    • Behavior of the old and permanent generations: jstat -gcold vmID
    • Capacity of the old generation: jstat -gcoldcapacity vmID
    • Capacity of the permanent generation: jstat -gcpermcapacity vmID
  • Monitor objects awaiting finalization:
    • JConsole tool, VM Summary tab
    • jmap -finalizerinfo pid
    • getObjectPendingFinalizationCount method class
  • Monitor memory:
    • Heap allocation profiles via HPROF: java -agentlib:hprof=heap=sites
    • JConsole tool, Memory tab
    • Control-\ prints generation information.
  • Monitor CPU usage:
    • By thread stack: java -agentlib:hprof=cpu=samples application
    • By method: java -agentlib:hprof=cpu=times application
    • JConsole tool, Overview and VM Summary tabs
  • Monitor thread activity:
    • JConsole tool, Threads tab
  • Monitor class activity:
    • JConsole tool, Classes tab

Actions on a Remote Debug Server

First, attach the debug daemon jsadebugd, then execute the command:
  • Dump Java heap in binary format to a file: jmap -dump:format=b,file= filename hostID
  • Print shared object mappings: jmap hostID
  • Print heap summary : jmap -heap hostID
  • Print finalization information : jmap -finalizerinfo hostID
  • Print lock information : jstack -l hostID
  • Print thread trace : jstack hostID
  • Print Java configuration information: jinfo hosted

Other Functions

  • Interface with the instrumented Java virtual machines:
    • Monitor for the creation and termination of instrumented VMs: jstatd daemon
    • List the instrumented VMs: jps
    • Provide interface between remote monitoring tools and local VMs: jstatd daemon
    • Request garbage collection: JConsole tool, Memory tab
  • Print Java configuration information from a running process:
    • jinfo pid
  • Dynamically set, unset, or change the value of certain Java VM flags for a process:
    • jinfo -flag flag
  • Print command-line flags passed to the VM:
    • jinfo -flags
  • Print Java system properties:
    • jinfo -sysprops
  • Pass a Java VM flag to the virtual machine:
    • jconsole -J flag ...
    • jhat -J flag ...
    • jmap -J flag ...
  • Print statistics of permanent generation of Java heap, by class loader:
    • jmap -permstat
  • Report on monitor contention.
    • java -agentlib:hprof=monitor=y application
  • Evaluate or execute a script in interactive or batch mode:
    • jrunscript
  • Interface dynamically with an MBean, via JConsole tool, MBean tab:
    • Show tree structure.
    • Set an attribute value.
    • Invoke an operation.
    • Subscribe to notification.
  • Run interactive command-line debugger:
    • Launch a new VM for the class: jdb class
    • Attach debugger to a running VM: jdb -attach address

From oracle, More to Come

Read More

Tuesday, January 29, 2013

Tunable Linux Kernel Parameters

Parameters that can be used in dealing with Shared memory Segments

localHost:Root-~ $ ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 67108864
max total shared memory (kbytes) = 17179869184
min seg size (bytes) = 1

SHMMAX is used to define the maximum size (in bytes) for a shared memory segment. the larger it is, the better Caché performance will be.

To get the Shmmax value we can use

localHost:Root-~ $ cat /proc/sys/kernel/shmmax

We can change the value in /etc/sysctl.conf file.

SHMALL controls the total amount of shared memory (in pages) that can be used at one time on the system.

To find this we can use,

localHost:Root-~ $ cat /proc/sys/kernel/shmall

SHMMNI : This kernel parameter is used to set the maximum number of shared memory segments system wide. The default value for this parameter is 4096. This value is sufficient and typically does not need to be changed.

localHost:Root-~ $ cat /proc/sys/kernel/shmmni

Page Size: To get the page Size we can use

localHost:Root-~ $ getconf PAGESIZE
If we make any Changes to the /etc/sysctl.conf file , we need to execute the ‘sysctl –p’ command for the changes to update.

Parameters that can be used in dealing with Semaphores

localHost:Root-~ $ ipcs -ls

------ Semaphore Limits --------
max number of arrays = 512
max semaphores per array = 512
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

semmsl – This kernel parameter is used to control the maximum number of semaphores per semaphore set

semmni – This kernel parameter is used to control the maximum number of semaphore sets in the entire Linux system

semmns – This kernel parameter is used to control the maximum number of semaphores (not semaphore sets) in the entire Linux system

File Handles

When configuring the Linux server, it is critical to ensure that the maximum number of file handles is large enough. The setting for file handles denotes the number of open files that you can have on the Linux system

localHost:Root-~ $ cat /proc/sys/fs/file-max

Oracle recommends that the file handles for the entire system be set to at least:
512 * PROCESSES , where PROCESSES is the value specified for the instance(s) running on the server. I typically use a value no less than 65536.

Current usage of file handles can be obtained by using ,

localHost:Root-~ $ cat /proc/sys/fs/file-nr
7650 0 6487024

The file-nr file displays three parameters :
  • Total allocated file handles
  • Currently used file handles
  • Maximum file handles that can be allocated

IP Local Port Range

Oracle strongly recommends to set the local port range ip_local_port_range for outgoing messages to “1024 65000″ which is needed for systems with high-usage. This kernel parameter defines the local port range for TCP and UDP traffic to choose from. The default value for ip_local_port_range is ports 32768 through 61000.

localHost:Root-~ $ cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000

The receive buffers are used by TCP and UDP to hold received data until it is read by the application. The receive buffer cannot overflow because the peer is not allowed to send data beyond the buffer size window. This means that datagrams will be discarded if they don’t fit in the socket receive buffer, potentially causing the sender to overwhelm the receiver.

cat /proc/sys/net/core/rmem_default
cat /proc/sys/net/core/rmem_max
cat /proc/sys/net/core/wmem_default
cat /proc/sys/net/core/wmem_max

Configuring Shell Limits for the oracle User

To improve the performance of the software on Linux systems, Oracle recommends you increase the following shell limits for the oracle user :

Shell Limit
Item in limits.conf
Hard Limit
Maximum number of open file descriptors
Maximum number of processes available to a single user

cat >> /etc/security/limits.conf <<EOF
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

cat >> /etc/pam.d/login <<EOF
# Added for Oracle Shell Limits
session    required     /lib/security/
session    required

More To Come , Happy Learning.

Read More

Wednesday, January 16, 2013

Flush Memory

Flush Memory

[root@vx111a ~]# free -m

                         total   used   free   shared buffers cached
Mem:                  3031   1811  1220   0        201      1266
-/+ buffers/cache:          343    2688
Swap:                 4996       0   4996

[root@vx111a ~]# sync; echo 3>/proc/sys/vm/drop_caches

[root@vx111a ~]# free -m

                           total   used   free  shared buffers cached
Mem:                    3031   1821  1210  0            201 1266
-/+ buffers/cache:             353  2678
Swap:                   4996        0  4996

Forces an immediate write of all updated data from buffers to hard drive (synchronize drive with buffers). While not strictly necessary, a sync assures the Sys admin or user that the data just changed will survive a sudden power failure. 
Read More

Thursday, January 10, 2013

Why Tomcat 7?

Apache Released a New version of Tomcat 7.There is a need for the migration of Applications which are running in Tomcat 6 to Tomcat 7.In this article we will see the advantages of Tomcat 7 and its Features.

1.    Tomcat 7 Requires Java SE 6 Further Versions.
2.    Tomcat 7 Includes Servlet 3.0 API.
3.    Tomcat 7 Includes Expression language 2.2 (mechanism for enabling  the presentation layer (web pages) to communicate with the application logic (managed beans)).
4.    Tomcat 7 Includes JSP 2.2.
5.    Tomcat 7 Contains tomcat-api.jar which contain shared interfaces for Catalina and Jasper. A few other libraries include
  • annotations-api.jar (Annotations package)
  • jasper-el.jar (Jasper 2 EL implementation)
  • jsp-api.jar (JSP 2.2 API)
  • servlet-api.jar (Servlet 3.0 API)
  • tomcat-api.jar (Interfaces shared by Catalina and Jasper)
  • ecj-3.7.1.jar
  • el-api.jar (EL 2.2 API)
  • jasper.jar (Jasper 2 Compiler and Runtime)
  • jasper-el.jar (Jasper 2 EL implementation)
  • jsp-api.jar (JSP 2.2 API)
  • servlet-api.jar (Servlet 3.0 API)
  • tomcat-api.jar (Interfaces shared by Catalina and Jasper)
  • tomcat-coyote.jar (Tomcat connectors and utility classes)
  • tomcat-dbcp.jar (package renamed database connection pool based on Commons DBCP)
6.    Tomcat 7 Includes Generics In Necessary places. This helps programming errors to be detected at compile time rather than run time as well as enabling the writing of cleaner code
7.    Tomcat 7 includes Servlet 3.0's asynchronous support.
8.    Tomcat 7 Includes Annotation Support, which helps the developers in configuring filters, listeners and servlets in a declarative programming style. Also allows defining annotations on servlet
11.  Tomcat 7 provides increased control over session tracking, in particular the ability to use the SSL session ID to track user sessions which provides additional secch allows tomcat basic operations like start, stop. This helps in Integration Testing.
10.  Tomcat 7 provides a better logging feature like asynchronous handler allows Tomcat to write logs to disk in a dedicated thread, so that logging operations do not cause any latency in processing threads.
11.  Tomcat 7 provides increased control over session tracking, in particular the ability to use the SSL session ID to track user sessions which provides additional security.
  Tomcat 7 has File upload support will enable Tomcat users to use file upload functionality within their web applications with the need for additional libraries.

Tomcat 7 includes alias support which allows developers to include external content from a system file location or a war file. An example would be to add shared java script /Images for multiple web applications.
16.  Tomcat 7 provides programmatic login.
17.  Tomcat 7 provides better security for the Manager and Host Manager applications.
16.  Tomcat 7 provides programmatic login.
17.  Tomcat 7 provides better security for the Manager and Host Manager applications.
18.  Tomcat 7 provides a feature where all the connector implementations (http, Ajp) now use Executors to provide the thread pool.
19.  Tomcat 7 manager application has a ‘find Leak’ Button which enables a Full GC and also finds the leaks.
20.  Better Memory Leak protection

Tomcat 7 Roles

With Tomcat 7, there are now 4 roles that allows administrators to delegate access to specific account so that they can do only certain things. The roles are
manager-guimanager-script,manager-jmx, and manager-status.

manager-gui Provides access to all the status pages and tomcat manager web console. The Accounts with this level of access can perform any operation using the manager console.

manager-script provides all the functionality as the manager web console does but using a Text Interface instead of HTML GUI.

manager-jmx provides access to the jmxproxy, which is a monitoring tools and scripts which can be helpful to the developers and administrators. In addition, this role also has access to the status pages. 

manager-status provides the users assigned to that role with access to the statistics that tomcat provides like current threads, max threads, etc. Users belonging to this role will be able to access the Status link on the main tomcat index page but will receive a 403 - Access Denied when attempting to access the Tomcat Manager.

Here are the Links

The Manager roles for the GUI (manager-gui), status (manager-status), scripting (manager-script) and JMX proxy (manager-jmx), defined in "webapps/manager/WEB-INF/web.xml". This allows for fine-grained access control to management tasks.

  • manager-script - Access to the "plain-text" interface, and to the "Server Status" page, via command in the form of: http://{host}:{port}/manager/text/{command}?{parameters}

http://{host}:{port}/manager/text/deploy?path=/sampleApp           // Deploy sampleApp

  • manager-jmx - Access to JMX proxy interface and to the "Server Status" page, via:

The roles manager-gui, manager-script, and manager-jmx are not to be given to a User for Security reasons.

The “admin” role has been broken into “admin-gui” and “admin-script”. These allows for the access control of the Host Manager application. These admin roles provide access to the status pages. 

Happy Learning

Read More

Tuesday, January 8, 2013

Trace Route

Trace route is a computer network diagnostic tool for displaying the route (path) and measuring transit delays of packets across an Internet Protocol (IP) network.

How to use
[root@localhost ~]$ traceroute

traceroute: Warning: has multiple addresses; using
traceroute to (, 30 hops max, 46 byte packets
 1 (  0.353 ms  0.205 ms  0.230 ms
 2 (  0.335 ms  0.226 ms  0.240 ms
 3 (  1.233 ms  1.125 ms  1.088 ms
 4 (  2.181 ms  2.125 ms  1.937 ms
 5 (  2.329 ms  2.123 ms  2.187 ms
 6 (  2.181 ms  2.125 ms  2.437 ms
 7 (  2.981 ms  2.471 ms  2.487 ms
 8 (  12.365 ms  14.499 ms  14.115 ms
 9 (  14.109 ms  12.907 ms  12.467 ms
10 (  12.463 ms  12.607 ms  12.517 ms
11 (  12.913 ms  12.707 ms  12.669 ms
12 (  12.761 ms  12.755 ms  29.290 ms
13 (  12.662 ms  12.502 ms  12.766 ms

traceroute to (, 30 hops max, 46 byte packets
Found the IP address of to be, max is 30 hops to reach, and sending 46 byte size packets.

The numbers at the far left are the number of the hop, followed by the name and/or IP address of the router that hop is going through.

The set of three numbers on the right side of the lines indicate the amount of time, in milliseconds, it took for that hop to complete. Traceroute performs each hop three times.

16 * * *
You can see some times , in the hop we will have nothing but packet loss. The trace route continued for the full 30 hops, each reporting * * * as it went. If your trace to the server has many asterisks like this one, that means that the connection was not able to be completed.  This could be for a variety of reasons including:
A network outage
High amounts of traffic causing network congestion
A firewall dropping traffic from your IP

Many More time Come , Happy learning
Read More