Thursday, March 26, 2015

Python Module – URLLIB2 – Rest Calls

Scripting languages should provide certain default things that every programmer use every day and fetching content from Internet is one of them

Python provides modules that allow fetching urls. Urllib2 is a python module that helps in fetching URLs (Uniform Resource Locators). It offers a very simple interface, in the form of the urlopen function. This is capable of fetching URLs using a variety of different protocols.

The modules also offer ways for handling basic authentication, cookies, and proxies and so on. These are provided by objects called handlers and openers.

Case 1 – Make a call
In order to make a call to a URL we can use,

import urllib2

response = urllib2.urlopen(url)
print response.read();

This is the simplest way of making a Call to URL and gets the response.

HTTP Requests – As we know HTTP is based on request and response. The client makes a request and server’s sends back the response. Rullib2 has the way to create a request object which will represent the HTTP request and when sent to server, it returns the response object.

The request after creation can be called using the same urlopen() method. This response is a file-like object, which means can be processed using .read() on the response

req = urllib2.Request('http://www.nove.org')
response = urllib2.urlopen(req)
data_page = response.read()

urllib2 makes the use of same request interface to handle ftp too like,
req = urllib2.Request('ftp://example.com/')

Case 2 – Post requests
Urllib2 can be used in posting data too. When using HTML forms, the data needs to be encoded before sending and then passed to the request object as a data argument before using this request object. The encoding can be done using urllib module rather than urllib2.

This can done as, 

import urllib
import urllib2

url = 'http://www.nova.com/SameServlet'
values = {'name' : 'Nova', 'location' : 'Hyderbad', 'language' : 'Python Call' }

data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
data_page = response.read()

Once we have the values we need to use the urllib.urlencode() before appending them to the request object. Once we have the encoded data we can create a Request object passing the URL and data.  The urlopen() is called on the request object to get the response.

Case 3 – Delete Requests
There will be cases where we need to use other methods like PUT, DELETE etc to perform operations. Delete Operations can be done as,

url = "http://nova.com/"+assetID
req = urllib2.Request(url,data='1121')
req.get_method = lambda: 'DELETE'
urllib2.urlopen(req).read()

Case 4 – Headers
Headers play an important role when making calls to external web resource. The user-agent header element is one important piece of information that can identify the source of the hit.
So in order to add the header to the request we can use

request = urllib2.Request('http://localhost:8080/')
request.add_header('User-agent', 'www.nove.com')
response = urllib2.urlopen(request)
data = response.read()

After creating a Request object, use add_header() to set the user agent value before opening the request.

I hope this article on basics of urllib2 will help people to dig more into the library
Read More

Python Module – Minidom - Parsing XML

This is a first of the series of articles on python modules. We will have articles that I will explain the basics of the python module and how to use that.

Parsing XML is one of the important features that every programming language has to provide. We need to parse XML many times when we get response from a Rest call or parsing locally stored XML files.

Xml.dom.minidom is a minimal implementation of the Document Object Model interface. This is much simpler and also smaller. As said in the doc ,users who are not good at Full DOM can use other XML processing module called “xml.tree.ElementTree”.

In this article we will see how we can process a XML file using xml.dom.minidom module.
Consider Sample XML content as

<?xml version="1.0"?>
<company>
          <name>Animal Care Enterprise</name>
          <staff id="1">
                   <nickname>Rats</nickname>
                   <salary>100,000</salary>
          </staff>
          <staff id="2">
                   <nickname>Dogs</nickname>
                   <salary>200,000</salary>
          </staff>
          <staff id="3">
                   <nickname>Cats</nickname>
                   <salary>20,000</salary>
          </staff>
</company>

Case 1 – Printing values
In order to process this we write the code as,

from xml.dom import minidom
from xml.dom.minidom import parse, parseString
from xml.dom.minidom import Document

dot = minidom.parse('dot.xml')
staffs = dot.getElementsByTagName('staff')

for staff in staffs:
       sid = staff.getAttribute("id")
       nickname = staff.getElementsByTagName("nickname")[0]
       salary = staff.getElementsByTagName("salary")[0]
       print("id:%s, nickname:%s, salary:%s" %(sid, nickname.firstChild.data, salary.firstChild.data))

We need to import the necessary Minidom module for processing the XML files. In order to read a file we use,

dot = minidom.parse('dot.xml')
Then we get the staff elements using the getElementsByTagName() method passing the element name.

staffs = dot.getElementsByTagName('staff')

This gives us the array with all the staff element details and we just need to parse them. After executing the code we can see

id:1, nickname:Rats, salary:100,000
id:2, nickname:Dogs, salary:200,000
id:3, nickname:Cats, salary:20,000

If we need to parse xml that is obtained in the Rest response we can use
dom = parseString(assetXML)

This will parse the String as an XML. The parse() and parseString() functions do is connect an XML parser with a “DOM builder” that can accept parse events from any SAX parser and convert them into a DOM tree. 

Case 2 – Adding an Element
Now once we were able to parse the XML doc and get various details of

dot = minidom.parse('dot.xml')
element=dot.createElement("Staff")
dot.childNodes[0].appendChild(element)
print dot.toxml()

We can also see that the <Staff> element was added to the end of the node as below

<?xml version="1.0" ?>
<company>
    <name>Animal Care Enterprise</name>
    <staff id="1">
        <nickname>Rats</nickname>
        <salary>100,000</salary>
    </staff>
    <staff id="2">
        <nickname>Dogs</nickname>
        <salary>200,000</salary>
    </staff>
    <staff id="3">
        <nickname>Cats</nickname>
        <salary>20,000</salary>
    </staff>
<Staff/></company>

Case 3 – Adding an Text Node
As we know the text of an element node is stored in a text node. In order to create a Text node we can use

dot = minidom.parse('dot.xml')
element=dot.createElement("Staff")
txt = dot.createTextNode("hello, world!")
element.appendChild(txt)
dot.childNodes[0].appendChild(element)
print dot.toxml()

and we can see the output as,

<?xml version="1.0" ?>
<company>
    <name>Animal Care Enterprise</name>
    <staff id="1">
        <nickname>Rats</nickname>
        <salary>100,000</salary>
    </staff>
    <staff id="2">
        <nickname>Dogs</nickname>
        <salary>200,000</salary>
    </staff>
    <staff id="3">
        <nickname>Cats</nickname>
        <salary>20,000</salary>
    </staff>
<Staff>hello, world!</Staff></company>

Case 4 - Node Import
Nodes can be imported using Minidom. We can use this import feature to copy nodes between multiple xml files. This can be done as

dom1 = parse("foo.xml")
dom2 = parse("bar.xml")
element = dom1.importNode(dom2.childNodes[1], True) 
#  take 2nd node in "bar.xml" and do  deep copy
dom1.childNodes[1].appendChild(x)  # append to children of 2nd node in "foo.xml"
print dom1.toxml()

Using the above examples we can start working on minidom – a XML processing module available in Python.
Read More

Tuesday, November 18, 2014

Sudo

There are cases sometimes where a non-root user needs to execute a command in Linux. Some of the commands like yum are not allowed to be executed by a non-root user. So whenever a non-root user tries to execute a command , a Permission exceptions is thrown.

There is way in linux which allows executing a command by non-root user if he knows the root password. He can use the sudo command

[oracle@localhost x86_64]$ sudo yum install hello-1-1.el6.x86_64.rpm

there is also another way for allowing non-root users to execute commands by making some changes in sudo file. In this article we will see how we can use the sudo file in providing command execution permissions to some users.

Sudo stands for either "substitute user do" or "super user do". sudo allows a user to run a program as another user (most often the root user). In other cases we use “su” command to change from one user to another user with password even to the root account. Most cases it is a bad idea to log into the root account but it is good to get administrative privileges. Sudo provides us the same thing to a user the administrative privileges.

Lets see a basic example on how to use the sudo command.

Consider installing a rpm from a non-root user like oracle. When we try to execute the command we see the below message,

[oracle@localhost x86_64]$ yum install hello-1-1.el6.x86_64.rpm
Plugin "product-id" can't be imported
Plugin "subscription-manager" can't be imported
Loaded plugins: refresh-packagekit
You need to be root to perform this command.

We can then use the sudo command which will ask for a root password.

[oracle@localhost x86_64]$ sudo yum install hello-1-1.el6.x86_64.rpm
[sudo] password for oracle:

In both the above cases we are not able to install the rpm. Now we will provide the administrative privileges to the oracle user with the help of sudo file. The sudo file is /etc/sudoers. We can only change the file using the root account. Sudo is VERY particular about syntax in the configuration file. So we need to be care full and double our configurations before we save your file.

To make changes to the sudo configuration file you need to use a specific command - visudo. Let's take a look at how to add a user to the sudoers file.

The basic entry for a user looks like this:
user hostlist = (userlist) commandlist

Typically you will find an entry like this:
root  ALL=(ALL) ALL

The above line indicated that the root user on any hosts can run all the commands. Now in our case we need to provide the user oracle with privileges to run the yum command , for that add a line under the above line as

oracle ALL=NOPASSWD:/usr/bin/yum

Once we add the line, save the file. Now login as oracle user and run the yum command as

[oracle@localhost x86_64]$ sudo yum install hello-1-1.el6.x86_64.rpm
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating Red Hat repositories.
Setting up Install Process
Examining hello-1-1.el6.x86_64.rpm: hello-1-1.el6.x86_64
hello-1-1.el6.x86_64.rpm: does not update installed package.
Error: Nothing to do

The yum command installed the hello package. We have provided the administrative privileges to the oracle command for running the yum command. Similarly there are many other options in sudo file which allows us to provide administrative privileges either by groups, IP address and many other ways.

This article has been an introductory for sudo command. I will provide more clear details on using the other options in the sudoers file.
Read More

Monday, November 17, 2014

All You Need to Know about Class-loaders

Class-loaders in java play an important role in Java. They are the pieces that every java developer should be aware of. In this article we will see the working of a Class-loader in java.

So what exactly is a Class-loader?
Every class-loader is an Object. An Instance of the class that extends the abstract Class java.lang.ClassLoader. So every class in java is loaded by one of these instances.

So what does this class-loader do?
The class-loader is the means by which Java Classes and resources are loaded in the JRE (Java Runtime Environment).

What does it load?
As we know a Class file contains the binary representation of a Java Class, which has the executable byte-codes and references to other classes used by this class. This includes references to classes to the java API and other references. The Class-loader locates the byte-codes for the java class that needs to be loaded, read the byte-codes and creates a instance of java.lang.Class class. Then this class is made available to the JVM for execution.

Even statements as simple as String greeting = "hello" or int maxValue = Integer.MAX_VALUE make use of a class loader. They require the String class and the Integer class to be loaded.

Initially only class that needs to be executed as loaded into the class ,once the execution moves forward the Classes being referenced are loaded are loaded thus using the lazy loading.

If classes are loaded by the instances of the class that extends ClassLoader class, then there should be something that makes the started before this that make the instances of the Class-loader class. How does this work?

Class-Loader principles
Every ClassLoader in java Works on 3 Principles,
Delegation: Whenever a Class is to be loaded, the request is forward to the Parent ClassLoader for loading. The Child ClassLoader loads the class if Parent ClassLoader cannot find or load Class.

Visibility: This allows child ClassLoader to see all classes loaded by the parent ClassLoader but the Parent ClassLoader cannot view classes loaded by the Child ClassLoader.

Uniqueness: This allows loading a Class exactly once by taking help of delegation and ensures that the ClassLoader does not re-load the class that is already re-loaded by the Parent.

How Classes are loaded?

Here is an excerpt from the java.lang.ClassLoader

package java.lang;

public abstract class ClassLoader {

  public Class loadClass(String name);
  protected Class defineClass(byte[] b);

  public URL getResource(String name);
  public Enumeration getResources(String name);

  public ClassLoader getParent()
}

The loadClass(String name) is the most important method above, which takes the fully qualified name of the class to be loaded as String and returns an object of class Class.

The next important method is the defineClass(byte b[]) which actually materialize the Class. The byte array parameter is the byte code of the Class which is loaded from Disk.

The getParent() method returns the Parent Class-loader.

getResource and getResources return URLs to actually existing resources when given a name or a path to an expected resource.

So consider if I we check the code,

Public class A {
   Public void createB() {
          B b=new B();
  }
}

In the above snippet, when we say B b=new B(),a couple of things happen here. The creation of B instance is B b = A.class.getClassLoader().loadClass(“B”).newInstance().

So every Object created is associated with its own class (A.Class in this case) and every Class is associated with its own ClassLoader.

Class-Loader Hierarchy
Whenever a JVM starts a bootstrap ClassLoader is started first. This is the parent Class-loader for all Classes. Its responsibility is to load Core Classes from java.lang package and other use full runtime Classes to the memory first. We can say that the bootstrap ClassLoader is the one which does not have a parent Class Loader.

The next one of the extension class-loader which is responsible for loading classes from all the jar file available in the java.ext.dirs path and files available in JRE/lib/ext location.

The next is the Application Class loader which is very important for a developer. This ClassLoader is responsible for loading classes from directories and jar files specified by the CLASSPATH environment variable, java.class.path system property or –classpath command line. This is the child of the extension Class-loader.

Contest Class Loader: In the newer versions of java, there is something called as Context Class Loader which is used with threads. A thread's context class loader is, by default, set to the context class loader of the thread's parent. The Context class loader of a thread is set to the Class-loader that loaded the application. The context class loader can load the classes that the application can load. This is normally used by the java Runtime such as RMI to load classes and resources on behalf of the application.

Endorsed Standard Override Mechanism: The Endorsed Standard Override Mechanism says that if we have a newer version of JAXP class which we want to load but Since this class is already being loaded by the bootstrap ClassLoader from the rt.jar, in order to overcome we can place this new version in a location and access that location by passing an argument as –Djava.endorsed.dir=<Path to Location of Jar>


Here is the Class-Loader hierarchy,













Request for a Class by Client
Lets see how the class-loader thing works when a Client requests to load a Class.

1. A check is performed to see if the requested class is already loaded by the Current Class Loader. If that is loaded it is returned back and used. The JVM will cache all the classes that are loaded by the Class-loaded so a Class which was loaded once will not be loaded again.

2. If the class is not already loaded, the request is delegated to the Parent Class-Loader before the check to the Current Class-loader fails. The delegation can move to the top level (bootstrap Class loader) until no further delegation can be done.

3. So if the parent Class-loader also cannot load the class, the Current Class-loader will then try to search for the requested class in some defined locations such as the bootstrap searches in the location specified in the sun.boot.class.path system property. And Application class-loader will search in the classpath location.

4. If the class is still not found, a java.lang.ClassNotFouncException is thrown.

Does Classes in Class-Loader see?
Classes loaded by a Class-loader can see other Classes loaded by the same Class-loader. Classes inside a Class-loader cannot see Classes loaded by other Classes.

JMV keepts track of the classes loaded by a Class-loader. This is achieved by name-space technique by which every Class-loader in a Java application will have its own Name-space( java.lang.*) and classes loaded in one name –space cannot see classes loaded by other Name-spaces.

There are certain restrictions where we can load multiple class-loaders in the java application and load the same class from multiple class-loaders. At this point , the class will know that there is another version available.

What If I create a custom class in a name-space?
Since we said that classes are loaded by the name-space and classes loaded in one class-loader cannot view classes loaded by other Class-loader. What if I create a class in java.util Package?

Even if we create a custom class with a same package, java only grants access to a Class in the same package which gets loaded by the same Class-loader. In the above case even if we create a Custom class with a same package name, the class will be loaded by the user-class loader.

Container Class-Loader
This class loader hierarchy happens when a JVM starts but the hierarchy may differ when we use J2EE containers. In containers, every ear application deployed will be having its own Class-loader and more over ever war file inside the ear will have its class-loader. The web class-loader will load the classes in the local class-loader and the task is delegated to the Parent Class loader only when Classes cannot be loaded (as Specified in Servlet Specification).















Some containers allows to web modules to load classes from the location specified instead of loading classes from the Parent Class Loader. In Weblogic, we can use <Prefer-web-inf-class> element in the weblogic.xml file which allows jar to be loaded from the WEB-INF location even though a version of the jar is loaded by the parent Class-Loader. These cases arise with the logging libraries where the container holds a version and application ships with a different version.

This is normally done by overriding the loadClass() method of the java.lang.ClassLoader Class. 
We will see some Class Loader case Studies like Tomcat, Weblogic and JBoss in next articles.


Read More

Eclipse Memory Analyzer: Shallow heap and Retained heap

Developers are more concerned about the size of objects when they are created. Eclipse memory analyzer gives us information about the size of the object in 2 calculations.

Shallow Heap
Retained Heap

Shallow heap is the memory consumed by one object. An object needs 32 or 64 bits (depending on the OS architecture) per reference, 4 bytes per Integer, 8 bytes per Long, etc. Depending on the heap dump format the size may be adjusted (e.g. aligned to 8, etc...) to model better the real consumption of the VM.

Shallow size of an object is the amount of allocated memory to store the object itself, not taking into account the referenced objects

Retained size of an object is its shallow size plus the shallow sizes of the objects that are accessible, directly or indirectly, only from this object. In other words, the retained size represents the amount of memory that will be freed by the garbage collector when this object is collected

Dead objects are shown only with shallow size, as they do not actually retain any other objects.

Generally speaking, shallow size of an object is its "flat" size in the heap whereas the retained size of the same object is the amount of heap memory that will be freed when the object is garbage collected.

For example the shallow size of an instance of java.lang.String will not include the memory needed for the underlying char[] where as retained size includes.
Read More

Eclipse Memory Analyzer: Finding Strings

Eclipse Memory analyzer provides various ways of analyzing String from the heap dump. We can get the details of the String using

1. Open the Histogram -> Right Click on “java.lang.String” and select the List Objects. We can get the size by sorting the retained Size ( the memory kept alive by Each String )

2. If we want to get the value that is holding by the String, we can get by selecting the String, right click and “Copy -> Value”. The full value is then copied to the clip Board.

3. We can also use a Query to get the details. Select OOL from the Menu as







Enter “SELECT s.count, toString(s) FROM java.lang.String s” to extract the count property of the strings.
Read More