Pages

Showing posts with label Weblogic. Show all posts
Showing posts with label Weblogic. Show all posts

Saturday, July 18, 2015

Weblogic Overload Protection


Weblogic Server has a built-in feature for detecing ,avoding and recovering from overload conditions. So when a Weblogic server is experiencing Out of memory conditions or any of the Server Sub System is in failed state , we can instruct weblogic server to take corrective actions .  We can tune the Weblogic server to take corrective actions during these overload states. We can for example instruct to kill the server process during a failed or a overload state so that the node manager or administrator can take corrective actions.

A managed-server can fail as a result of out-of-memory exceptions or stuck application threads, or if one of its services is running into an error condition. A managed-server instance can monitor its health. If it detects that an unstable state is reached, it declares itself failed.

By usiong the Weblogic Over load protection feature we can prevent the negative consequences like degraded application performance and stability causing the server not to take any requests from users.

Configuring the Over load protection
Go to Server -> Overload tab to configure the over load protection for that server.

In the overload tab we can see 2 actions which can taken when we see a Over load condition

Panic action - When the kernel encounters a panic condition , the below actions can be taken.The following two actions are available.

No-action
Exit the server process

When the server health monitoring encounters a critical situation and flags the server as failed, one of three predefined actions can be taken automatically

No-action
"Force immediate shutdown of this server", meaning that server will shut down completely
"Suspend server for correction action", meaning that server will go into admin state

There are certain other conditions that can be used that exists in the same tab ,

Max Stuck Thread Time - The number of seconds that a thread must be continually working before this server diagnoses the thread as being stuck
Stuck Thread Count - The number of stuck threads after which the server is transitioned into FAILED state. There are options in OverloadProtectionMBean to suspend and shutdown a FAILED server. By default, the server continues to run in FAILED state.


Hope this helps about the over load protection in weblogic.
Read More

Thursday, July 9, 2015

Weblogic - Connect to the Embedded LDAP

In most cases we will not be connecting to the LDAP repository available in Weblogic but in some cases we need to connect to that for obtaining certain details regarding the users and roles. In this article we will see how  we can connect to the Embedded LDAP using an external LDAP explorer.

Before connecting to the Weblogic Embedded LDAP repository , we need to make some changes on the Weblogic side which will allow to browse the repository.

Enable "Anonymous Admin Lookup Enabled" under <Domain Name>/Security and change the credential and  Confirm Credential  with new passwords ( like weblogic) under the <domain Name>/Security/Embedded LDAP tab

Once changes are done, we need to restart the Weblogic Server again. Once started connect to the Weblogic Embedded LDAP using JXplorer with the below configuration

Host: <IP address> | <Domain Name>
Port: Adminserver port
Protocol: LDAP v3
Base DN: dc=weblogicDomainName
Level: User+Password
User DN: cn=Admin
Password: Password Provided Earlier

Now we can browse the embedded browser.
Read More

Work Managers

In the previous versions of Weblogic, the processing of requests is done by multiple execute Queues. Different queues exist to work based on priority and requirements to avoid deadlocks. Weblogic also has some pre-configured Queues that work for the internal traffic like RMI calls etc.

The threads in execute Queue can be managed by the users. Weblogic also provides the ability for configuring custom execute Queue to make sure that certain applications has access to fixed number of threads to handle the load.

Weblogic Server uses a single thread pool, in which all types of work are executed. Weblogic Server prioritizes work based on rules you define, and run-time metrics, including the actual time it takes to execute a request and the rate at which requests are entering and leaving the pool.

From Weblogic Version 9.0, Work managers were introduced. Weblogic Server prioritizes work and allocates threads based on an execution model that consider administrator defined configuration parameters and also run time performance data.

Weblogic uses a single common thread pool for performing work. The size of the thread pool is determined by the kernel and will be resized when required. The Thread pool monitors the overall throughput every two seconds and uses the collected data to determine if thread count needs to change. The current thread pool count , the historical data related to the throughput and past history is taken in to consideration in determining whether the thread pool size needs to be increased or not.

Besides the thread pool, there is a priority Queue which is something like a regular Queue data structure where each element in it has a priority associated with that. In a priority queue, an element with high priority is served before an element with low priority. If two elements have the same priority, they are served according to their order in the queue. Now in Weblogic, the priority of the requests is calculated internally.

So at this point, we no longer have to configure thread counts on a work manager as it will take care of that. However it is possible to affect how server prioritizes work and allocates threads by parameters defined in Work Manager.

User can configure work manager and associate with an application or with a group of application components so that Weblogic server can work based on this configuration and assign pending work and enquired request to execution threads.

Work manager Components

The following are the work manager components that you can associate with the application,

  • Request class
    • Fair-share
    • Response-time goal
    • Context based
  • Constraints
    • Minimum threads constraint
    • Maximum threads constraint
    • Capacity

The request classes affect how requests are prioritized. Request classes help ensure that high priority work is scheduled before less important work, even if the high priority work is submitted after the lower priority work.

A Constraint defines minimum and maximum numbers of threads allocated to execute requests. It also defines the total number of requests that can be queued or executing before weblogic server begins rejecting requests. If the requests exceed or go down with the numbers defined in the constraints, the weblogic server either queues them or rejects them. When the requests are rejected, a HTTP response code of 503 is thrown back.

Work Manager Configuration

Work managers can be configured either in
Domain level -
Config.xml file is used to define global work managers which can be assigned to any application, application component etc

Application level -
Weblogic-application.xml file is used to define for application level which can be used to assign to that specific application or any component in that application

Component Level –
Weblogic-ejb-jar.xml or weblogic.xml files can be used to define work managers in the component level that can be assigned to the component.

Web application Level-
weblogic.xml file in web application are used to define work managers.

Work manager Scope
The scope of the work manager can be any one in the below list

The default Work Manager
This is the default work manager used when none were specified. This will be the default one that will be used by all applications. In many situations, the default Work Manager may be sufficient for most application requirements. Weblogic Server’s thread-handling algorithms assign each application its own fair share by default.

Global Work Managers
When we login to the weblogic console and create a Work manager we are actually creating a global work manager which will be defined in the config.xml file.

The global work manager is used by the application as a template. Each application running will create its own instance which handles the work associated with that application and separates the work from other applications. Handling each application’s work separately, allows an application to be shut down without affecting the thread management of another application. Although each application implements its own Work Manager instance, the underlying components are shared.

Application-scoped Work Managers
Though the default work manager and a global work manager are available but if you can also create Work Managers that are available only to a specific application or module you can configure them in Weblogic-application.xml, Weblogic-ejb-jar.xml or weblogic.xml and finally weblogic.xml file ( as defined in the Work manager configurations )

Does Execute Queue still exist?
As we said the work manager are implemented from Weblogic 9, we can even now configure Execute queues by using

  • Using the command line option: Dweblogic.Use81StyleExecuteQueues=true
  • Setting the Use81StyleExecuteQueues property via the Kernel MBean in config.xml
Once either of this is used to enable Execute Queue, work managers are converted to execute queue based on

  • If the work manager has  minimum or maximum threads constraint, a execute queue is created with same name of work manager and thread count will the value defined in the constraint
  • If the Work Manager does not implement any constraints, then the global default Execute Queue is used
How to Configure
Now let’s see how we can configure a Work manager and test it.

1. Open the admin console, click on the work Managers. In the next screen select the Work manager definition type and click next.


Give the name as “myTestWorkManager” and target it to a server.

Once this is saved. Click on the Created “myTestWorkManager” work Manager. Now we need to Configure the request class and Constraints as shown in the below image

Click on the request class - Select Context Request Class and in the next screen give it a name and deploy to the same target as above

Now click new after the Minimum Threads Constraint, in the next screen give a name and give count as 1 and in the next screen, select the target as same above

Now once again click the new after the Capacity Constraint, in the next screen, give a name and count to 1 and Select the target same as above

Once all the changes are done, we need to restart the servers for the changes to take place

You can see the names that I have used in the above screen.  So in the above work manager configuration we have defined the minimum thread constraint as “1” , this will make sure that only one thread will be executed at any point of time. When we start the application and try to access, we have already started a thread and when we try to access the same application again we will see a HTTP response code of 503.

Create Application
Now we have the work manager created and now we need configure this work manager to an application. For the demo purpose I created a sample war file which will run a jsp page which has code that will display 1 to 60 with 1000ms in gap.

Here is the code for the JSP page,

<%! static int counter=0; %>

<%
session.setAttribute("counter","Counter-"+counter);
counter++;
for(int id=0;id<60;id++)
  {
System.out.println(id+"\tRequest Is Being Processsed for  :"+(String)session.getAttribute("counter"));
try{
Thread.sleep(1000);
}
catch(Exception e)
{
e.printStackTrace();
}
}
out.println("<h2>Request Processing Completed");
%>

Now once we access the application, the first thread will run this code from 1 to 60 giving a gap of 1000ms. Once we try to access the same application again during the first thread still running we see a 503 error.

While creating the application we need to make sure the work manager is associated to the web application we created by adding the work manager details in the weblogic.xml file with

<weblogic-web-app>
<wl-dispatch-policy>myTestWorkManager</wl-dispatch-policy>
</weblogic-web-app>

I am attaching the war file for you testing purpose. Configure the work manager as above and deploy the code to the same server that the work manger was targeted and access the application using http://<IP address>:<Port>/myTestSample/index.jsp. The out put of the application can be seen in the command line. Here is the link for the application.

I will try to explain more about the work manager with more example. More to Come, Happy learning J
Read More

Tuesday, May 26, 2015

Analyzing Slow Response – Case 2 (a High CPU Usage)

Recently we observed a strange issue with one of our Tomcat server in Production. The monitors that we configured on these always showed degraded performance. The HTTP Ping monitor that we configured has a Success status when it responds in less than 2 seconds. But from the last couple of days we were seeing alerts as the HTTP Ping Monitor is taking more than 6 Seconds and to respond causing the alert to fail.

1) We started to analyze the performance of the server by using the basic top command which gave us the highest CPU and Memory Used process like,

localHost $ top -n1 | grep -m1 java
5944  root   20   0 7057m 4.3g 7444 S 312.1 55.9  13871:55 java

We obtained the Process that was actually using the High CPU and amount of the Memory being used. In this case the CPU being used is 312.1

2) Next step is we started to find out the Thread ID that is causing the HIGH CPU usage in this process which gave us this

localHost $ top -n1 -H | grep -m1 java | perl -pe 's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' '
5958

3) In the next process we took the thread dump of the PID 5944 (running Tomcat Process ID ) and obtained the NativeID for the Thread ID using

localHost $ printf "%x" 5958
1746

4) From the Thread Dump we observed that this thread belongs to Garbage Collection
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f306c025800 nid=0x1746 runnable

And some times it pointed us to
"VM Thread" prio=10 tid=0x00007f306c134800 nid=0x1749 runnable

The VM thread is a special thread in the HotSpot VM. It performs special tasks in the VM such as garbage collection. If the VM_Operation suggests that the operation is a garbage collection then it is possible that you have encountered as issue such as heap corruption.
The High CPU usage might be  a Garbage collector issue but it can be some thing else that leave object references in the heap in in-correct state.

The main important thing at this point is to collect as much information as possible about the environment and try out possible workarounds.

Now upon checking the GC status we observed that the memory is totally exhausted and the GC is performing high number of Full GC’s

localHost $ jstat -gcutil  5944
  S0      S1     E      O        P        YGC     YGCT       FGC    FGCT          GCT
  0.00   0.00  70.12 100.00  99.66  57691 4817.991 30178 107006.562 111824.553

In the above case , the FGC element tells us the number of Full GC Caused during the life time.

Since the JVM is busy handling the Full GC of this process, it is actually pausing the other threads to perform the GC events hence causing the slow response to the EMS monitor.


More to come, Hope this helps J

Read More

Friday, May 22, 2015

HTTP, HTTPS and AJP

In a normally web cases, there are certain protocols that we use for communicating with web server from clients. When I said there are certain it means that we have Http,Https and Ajp. In this article we will see the comparison between these and when can be used at what time.

What is HTTP?
The Hypertext Transfer Protocol (HTTP) is an application protocol that allows communication between nodes. Hypertext is structured text that uses logical links between nodes containing text. HTTP is the protocol to exchange or transfer hypertext.

This is the basic protocol we use when working with a Server. This HTTP request contains a series of lines of data with every line terminated with a new line. The lines include request method (get or Post), followed by required resource along with protocol version (HTTP 1.1). The next lines include host being contacted, headers, cookies, browser details like type, preferred language and other details.

Once the request is received and processed a response is sent. The response contains a header block and data block. The first line in the header includes a response code which indicates the status of the request. The response code includes- 3 digit number in the following ranges:
200 and up - success; expect good data to follow
300 and up - good request but only headers (no data). e.g. page has moved
400 and up - error in request. e.g. request was for missing page (404)
500 and up - error in handling request. e.g. program on server has syntax error

The header block also contain details telling about the receiving system, content type  (MIME) which allows the receiving system to know whether to handle the resource in HTML or some other. Then there's a blank line and the actual data.

What is HTTPS?
HTTPS is similar to HTTP with all the above details but it includes extra details about the SSL configuration. What this does is the data encryption. The data is encrypted at the client end and decrypted at the receiving end. This ensures data security. This is all done by an agreement between the client and server before talking to one another.

What is AJP?
As we said the web traffic is carried out in HTTP which is a plain text protocol. The plain text format where actual words make up the message can create a Overhead of traffic. AJP takes care of this by converting these messages into bi nary code thus reducing the amount of space taken by each message.

AJP or Apache JServ Protocol, is an optimized binary version of HTTP that is typically used to allow Tomcat to communicate with an Apache web server.  AJP Connectors are most commonly implemented in Tomcat through the plug-in technology mod_jk, a re-write of the defunct mod_jserv plug-in with extensive optimization.

This is mainly used when we need to integrate server with Apache Web servers and you want Apache to handle the static content contained in the web application, and/or utilize Apache's SSL processing. This connector supports load balancing when used in conjunction with the jvmRoute attribute of the Engine.

Which to Use?
For most requirements, http is enough and if you want to secure things we can use Https. AJP is particularly used for "Load Balancing." This is where one server distributes incoming traffic among several application servers, sharing the load equally.

More to Come, Happy Learning J
Read More

Tuesday, August 5, 2014

Apache Server Integration with Weblogic Cluster

In this article we will see how we can configure Apache server with a weblogic Cluster and access a application deployed in the weblogic cluster.

1. Configure weblogic Cluster with the below nodes as below
MS1:17003
MS2:17004

2. Once the Configuration of the Servers is completed, add them to a cluster and deploy a sample application to the cluster (in this case I deployed a sample application which will print the session ID on the screen)

3. Configure the Apache server as
a. Make sure the mod_wl weblogic .so file is added as
LoadModule weblogic_module modules/mod_wl_22.so
b.Make sure the Location element is added as
<Location />
SetHandler weblogic-handler
</Location>

C. Configure the <IfModule> element with the necessary information as
<IfModule mod_weblogic.c>
WeblogicCluster localhost:17003,localhost:17004
Debug ON
WLLogFile /tmp/wlproxy.log
WLTempDir /tmp
MatchExpression /*
DebugConfigInfo ON
KeepAliveSecs 15
DynamicServerList ON
</IfModule>

We have added the localhost:17003 and localhost:17004 which are the cluster address that we configured.

Now access the application that we deployed in the cluster using apache server URL.

More to Come , Happy learning J

Read More

Monday, August 4, 2014

Apache Server Basic Authentication with Weblogic

In this article we will see how to configure basic Authentication in Apache server which will allow to ask for a user name and password.

1. Create a file in a location like myuser.txt. add the contents
testuser:testuser

This the user name and password combination

2. Now once the file is created we need to encrypt the password using using the htpasswd command like

[root@localhost conf]# htpasswd -b myuser.txt testuser testuser
Updating password for user testuser

Now check the file and we can see the encrypted password as
[root@localhost conf]# cat myuser.txt
testuser:PtNHnFwOxljvM

3. Configure the httpd.conf file with the basic Authentication as

<Location />
SetHandler weblogic-handler
AuthUserFile /etc/httpd/conf/myuser.txt
AuthName "This is a Protected Area"
AuthType Basic
Require valid-user
</Location>

<IfModule mod_weblogic.c>
#WebLogicCluster 172.16.101.241:7003,172.16.101.241:7004
WeblogicCluster localhost:17003,localhost:17004
#WebLogicCluster MS1:17003,MS2:17004
Debug ON
WLLogFile /tmp/wlproxy.log
WLTempDir /tmp
MatchExpression /*
DebugConfigInfo ON
KeepAliveSecs 15
DynamicServerList ON

</IfModule>

The four parameters in the <Location> element are the one that allow apache to ask for a User Name and Password combination. Once the user name and password are provided the request is passed to the backed weblogic but we need to add a parameter in the weblogic config.xml. The parameter needs to be added Otherwise Weblogic Server will try to validate the BASIC authentication Header. The problem is that if there is some request to your application with "Authorization" header the request will be intercepted by Weblogic itself and will not be passed to your application. Weblogic will try to make authentication itself.


Happy learning, More to Come
Read More

Thursday, July 17, 2014

Configuring Apache Http Server with Weblogic

In this article we will see how we can configure Apache to talk with Weblogic server at the backend. We will use the weblogic Apache plugin for connecting the Apache http server to weblogic. For this article I will be using Weblogic 10.3 on a Linux 6 -64 Machine. The apache server running is 2.2.15.

Here is the sequence of steps to configure Apache Http server with weblogic.

1. Copy the so file from
Wlserver_10.2/server/plugin/linux/x86_64/mod_wl_22.so to httpd/modules location

The mod_wl_22.so file depends on the architecture. You need to copy the correct file depending on your architecture.

2. Load the weblogic module using
LoadModule weblogic_module modules/mod_wl_22.so

3. Now add the If Module information which will point to the Weblogic server running at back.

<IfModule mod_weblogic.c>
        WebLogicHost 127.0.0.1
        WebLogicport 7001
        MatchExpression /*
</IfModule>

The above <IfModule> is a basic syntax for providing details about the running weblogic server. We define the weblogic Host and Port and Match Expression as /*.

Now once the configuration is done,restart the apache and access
Http://localhost/console and we will see the login page of the weblogic admin console.


More to learn , Happy learning 
Read More

Thursday, December 26, 2013

Weblogic SNMP Configuration

The Simple Network Management Protocol (SNMP) is an application layer protocol that facilitates the exchange of management information between network devices and it is part of the Transmission Control Protocol/Internet Protocol (TCP/IP) protocol suite. SNMP enables network administrators to manage network performance, solve and find network problems, and plan for network growth.

In weblogic, we can configure SNMP agents that gather information related to the underlying resources and send them to the appropriate managers. These are sent to managers in response to the requests that manager made.

In this article we will see how to create a SNMP agent that collects metrics of a resource.

Create a SNMP Agent

1. Click a SNMP under the Diagnostics in the Weblogic Console











2. On the Right hand side, click on the New under the ‘Server SNMP Agent’.
3. Give a Name to the SNMP agent and save
4. Target the SNMP agent to a managed Server

Now click on the Newly Created Server SNMP agent, under the General tab
Check for the SNMP UDP port which is the port we use to listen.
Make sure we enable the check box “enabled” and Save.

Create a String Monitor

Now click on the ‘String Monitors’ tab, and create a new String Monitor.

A string monitor observes changes in attributes that are expressed as String objects.
1. Create a New String Monitor
2. Provide a Name and in the Monitored Mbean type , select the ‘ServerLifeCycleRuntime’ bean and Click Next












3. In the next screen, select the ‘state’ in the Monitored Attribute Name and click Next.









4. In the Next Screen, give the String to Compare value as ‘RUNNING’. And click Finish.











Now click on the Newly Created String Monitor and specify which runtime beam we want to monitor (in this case ServerA weblogic Instance) and we can also specify the Pooling Interval.





















5. Target the Newly Created String Monitor to the ServerA .

Create a TRAP
Now we will create a Trap,

We can specify that a trap that we create now is generated when there is a match between the value and the string you provide, or you can specify that the trap is generated if the value differs from the string you provide.

So in this case, the Server will generate a string message when the State of the Server changes. So a trap is generated whenever the state message changes.

1. Click on the Trap Destinations tab and select new.
2. Keep the Settings (Host, Port) as they are and save.





















Testing the Traps

When the SNMP agent is created we can see message like this in the Managed Server Instance logs (in this case ServerA logs)

<Dec 24, 2013 8:54:19 PM IST> <Notice> <SNMP> <BEA-320953> <The SNMP Agent has been started on port 161.>
<Dec 24, 2013 8:54:19 PM IST> <Notice> <SNMP> <BEA-320931> <The SNMP trap version is 1>

So the SNMP agent was started on the port 161

In order to test the SNMP agent, start the command and once started it displays a message saying that it is listening on port 162.

Now stop the ServerA weblogic Instance which will make the State value of the ServerA to STOPPED and since that value is different from RUNNING string that we provided it will generatea Trap which is

java -classpath .:/root/Oracle/Middleware/wlserver_10.3/server/lib/weblogic.jar weblogic.diagnostics.snmp.cmdline.Manager SnmpTrapMonitor -p 162

Listening on port:162

--- Snmp Trap Received ---
    Version        : v1
    Source         : UdpEntity:172.16.101.68:161
    Community      : public
    Enterprise     : enterprises.140.625
    AgentAddr      : 172.16.101.68
    TrapOID        : enterprises.140.625.0.70
    RawTrapOID     : 1.3.6.1.4.1.140.625.0.70
    Trap Objects   : {
   { enterprises.140.625.100.5=Tue Dec 24 20:53:01 IST 2013 }
   { enterprises.140.625.100.10=ServerA }
}
    Raw VarBinds   : {
   { enterprises.140.625.100.5=Tue Dec 24 20:53:01 IST 2013 }
   { enterprises.140.625.100.10=ServerA }
}


Many More to Come, Happy learning J
Read More

Friday, December 20, 2013

Weblogic : Extend Your Weblogic Console with WLDF

WLDF was one of the great tool in Weblogic.WLDF was just an extension before 11g but it was included in the admin console which helps in monitoring all the Weblogic Mbeans and can get other details too.

In this article we will see how we can use the WLDF console integrated in the Weblogic admin console

WebLogic uses watches to monitor all kinds of things to see how you system is performing, but not all watches are running. We will see how we can monitor the System Attributes using the WLDF Console.

First we need to create a watch to pick up notifications about the state of the CPU.

1. Click on the Diagnostic Modules in the Weblogic Console
2. In the next Screen, enter the name for the Diagnostic Module. Save
Even though the Diagnostic module was created, it did not have any metrics enabled. Click on the newly Created Diag­nos­tic mod­ule and select the Con­fig­u­ra­tion tab and Col­lected Met­rics sub tab.


3. Click New in the Configuration tab.
4. In the next screen we can choose between “Server run­time” and “Domain run­time” met­rics. (The for­mer means that met­rics will be col­lected at server level; the other option, Domain Run­time, pro­vides access to the Domain Wide met­rics and is avail­able only for the Admin Server)
Select the ‘Domain Runtime’ Metric












5. In the next screen, select the MBean as ‘com.sun.management.UnixOperatingSystem’.

6. In the next screen, select from the Available Attributes to Choosen Attributes.

7. In the next Screen, select the Instances and Finish.

Now Go the Home page and Click On the ‘Monitoring DashBoard’

There will be a new Tab Opened, Inside the new Monitoring DashBoard we see
View List and Metric Browser

Select the server for which you have assigned the Diagnostic Module in this case ‘admin Server’.
Once you select this then move to the Metric Browser, Type ‘Unix’ in the Types which will show you the UnixOperatingSystem under that. Click that

Now we can see ‘java.lang:type=OperatingSystem’ in the Instances Field. Click on this which will show the available metrics in the “Metrics” fields.

Select them and drag them to the New Chart which will give you various chart information regarding the metrics we selected.













Note: Make Sure You Start the Monitoring by click on the green Button on the Top of the View List and Metric Browser tabs.





















More To Come, happy learning
Read More