Pages

Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Wednesday, June 3, 2020

Python Variables : Local, Global and Non-Local

Variables are nothing but memory locations reserved by python to store values. A variable in python can be created simply as var = 10. Python Provides us the facility not to define the type of the variables being used. That is in many other high level language we have to define

Int var =10 #define a int variable
Python has the capability to identify the type of the data being saved and reserves memory based on that. Hence by assigning different data types to variables, you can store integers, decimals or characters in these variables.

Global Variables 
A variable declared outside of the function or in a global scope is called a Global variable. This variable can be accessed inside or outside of the function.

A simple example include, 
jagadishm@: cat variables.py
#!/usr/bin/python
x = "global"

def fun():
    print "x inside function",x

fun()
print "x Outside Function",x

When we execute the function, we can see the same value from inside the function and also outside as below,
jagadishm@: python variables.py
x inside function global
x Outside Function global

The global variable x is defined outside of the function and in global scope and hence it can be accessed by everything in that code.

Changing a Global variable -If we try to change the global variable inside a function like below,
#!/usr/bin/python

x = "global"
print x * 2

def fun():
    print "x inside function",x
    x = x * 2
fun()
print "x Outside Function",x

The first variable change ( x * 2) which display “globalglobal” but the same value change inside a function will result in below error, 
UnboundLocalError: local variable 'x' referenced before assignment

Local Variable - A Local variable is declared inside a function body or within local scope as below,
#!/usr/bin/python
def foo():
    y = "local"

foo()
print(y)

If we run the code, we see an error as below
NameError: name 'y' is not defined

Since the variable y is of local scope and defined inside a function, we cant access that outside the function.

Global and Local variable in same Code - Global and local variable can both be defined inside the code as below,

#!/usr/bin/python
x = "global "

def fun():
    global x
    y = "local"
    x = x * 2
    print(x)
    print(y)

fun()
The global variable can be used inside a function by defining the same variable “x” with a global keyword. 

Non-Local variable - Python 3 provides a nonlocal keyword. Non local variables are used in nested functions whose scope is not defined. This means the variable can be neither in the local or global scope.
Read More

Python Anonymous Functions

Most of the time we write functions to reuse the functionality again and again. But there can be some cases where we don't need to define a function if we need to use it only once. Rather than defining a function we can define an anonymous function. If the function is only used once, or a limited number of times, an anonymous function may be syntactically lighter than using a named function

An Anonymous function or lambda expression in python is a function definition that is not bound to an identifier. In this article, we will see how lambda expressions work in python.

A Lambda function is a small anonymous function that can take any number of arguments, but can only have one expression. This function is generally defined without a name. In python a function is defined with the keyword “def” but anonymous functions are defined using a lambda function.

The basic syntax for a lambda looks as, lambda arguments : expression

A double function in python is written as,
def double(x):
      return x * 2

The above function can be simply written as 
# double = lambda x: x * 2 

We can call the lambda function as double(2) which returns the value. Similarly another simple lambda functions includes as
# x = lambda a : a + 10 , Call the function as print(x(10))
# y = lambda a,b : a * b, call the function as print y(1,2)
# z = lambda a,b,c : a * b + c , Call the function as print z(1,2,3)

Lambda inside a function - A lambda function can be defined inside a function as below,

# def myFun(n):
         return lambda a : a * n

Now the function can be called as,
# doubler=myFun(3)
# print doubler(2)

The function myFun(3) is defined and assigned to doubler, then the doubler(2) is called.

Higher Order Functions - A function that takes in other functions as arguments are called higher order functions. Python has 2 higher order functions map() and filter(). 

Filter() : The filter function in python takes a function and a list of arguments. The function is then called with all the items in the list by comparing with an expression and finally returns a modified list. The modified list contains elements which are returned by evaluating the expression on each of the elements and once the evaluation is returned true.

An Example includes,
# my_list = [ 2,4,6,1,76,34,56,77]
# my_filter_list = list(filter(lambda x: (x%2 == 0), my_list ))

My_filter_list contains all the elements returned by checking each element in my_list with the expressions (x%2 == 0). Once the return of this expression is true, it is then added to the new list.

Map() : Map function also takes a list and an expression. The function is called with all the items in the list and a new list is returned which contains items returned by that function for each item. An example include,

# my_list = [1, 5, 4, 6, 8, 11, 3, 12]
# new_list = list(map(lambda x: x * 2 , my_list))

In the above function, each element of the my_list is mapped with expressions like (1*2) and then value is passed to x which in turn passed to the new_list.

Hope this helps in understanding Anonymous and higher order functions.
Read More

Monday, April 8, 2019

Complete Build Automation for Python Application Using Jenkins Pipeline

In this example pipeline, we will be deploying a python application using jenkins Pipeline code. The pipeline code includes checkout source code from Github, performs a quality scan using the pylint tool, does a unit test using the pytest tool and finally sends an email to the users.

1. Install the necessary tools python, pylint and pytest.

Installing Python  - Python will be by default available in all linux machines.

Installing Pylint - Pylint is a Python static code analysis tool which looks for programming errors, helps enforcing a coding standard, sniffs for code smells and offers simple refactoring suggestions.

Open a Command Prompt or Terminal. On Windows, navigate to the Python root directory (install location) and run the following command: python -m pip install pylint


Installing Pytest - Installing pytest is same as pylint. Open a Command Prompt or Terminal. On Windows, navigate to the Python root directory (install location) and run the following command: pip install -U pytest

Plugins to Install - There are couple of Jenkins plugin to be installed for the below pipeline to work. The plugins are warnings and publishHTML.

2. Create a Jenkins pipeline job, with scm pointing to the below github location. Download the source from here and create your own github repository.

3. Understanding the Jenkinsfile. The pipeline code has 3 different stages. The pipeline looks like below,

pipeline {
  agent any

 stages {
     stage('Checkout') {
            steps {
                git credentialsId: 'github-jenkins', url: 'https://github.com/jagadish12/funniest.git'
                echo 'CheckOut Success'
            }
        }

stage('lint'){
   steps {
        sh "virtualenv --python=/usr/bin/python venv"
        sh "export TERM='linux'"  
        sh 'pylint --rcfile=pylint.cfg funniest/ $(find . -maxdepth 1 -name "*.py" -print) --msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}" > pylint.log || echo "pylint exited with $?"'
        sh "rm -r venv/"
       
        echo "linting Success, Generating Report"
         
        warnings canComputeNew: false, canResolveRelativePaths: false, defaultEncoding: '', excludePattern: '', healthy: '', includePattern: '', messagesPattern: '', parserConfigurations: [[parserName: 'PyLint', pattern: '*']], unHealthy: ''
       
       }   
     }
      
stage('test'){
   steps {
         
        sh "pytest --cov ./ --cov-report html --verbose"
        publishHTML(target:
            [allowMissing: false,
              alwaysLinkToLastBuild: false,
            keepAll: false,
            reportDir: 'htmlcov',
            reportFiles: 'index.html',
            reportName: 'Test Report',
            reportTitles: ''])  
           
        echo "Testing Success"   
          }  
        }
       
stage('mail'){
        steps{
            emailext attachLog: true, body: 'Jenkins Build - Status Report', subject: 'Build Report', to: 'jagadesh.manchala@gmail.com'
        }
    }   

       }
}

The Pipeline contains 4 stages, checkout , lint , test and email.

Checkout - Create a Credential with the name “github-jenkins”. The same credential will be used to check out source code form the github.
   
stage('Checkout') {
            steps {
                git credentialsId: 'github-jenkins', url: 'https://github.com/jagadish12/funniest.git'
                echo 'CheckOut Success'
            }
        }

Lint - the seconds stage is the lint stage. In this stage we will run linting ( quality scan ) using the pylint tool. We will create a python virtualenv and perform the pylint testing inside the virtual environment.  Once the linting of the python code is done, the results will be displayed with the warning plugin in Jenkins.

stage('lint'){
       steps {
        sh "virtualenv --python=/usr/bin/python venv"
        sh "export TERM='linux'"  
        sh 'pylint --rcfile=pylint.cfg funniest/ $(find . -maxdepth 1 -name "*.py" -print) --msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}" > pylint.log || echo "pylint exited with $?"'
        sh "rm -r venv/"
       
        echo "linting Success, Generating Report"
         
        warnings canComputeNew: false, canResolveRelativePaths: false, defaultEncoding: '', excludePattern: '', healthy: '', includePattern: '', messagesPattern: '', parserConfigurations: [[parserName: 'PyLint', pattern: '*']], unHealthy: ''
       
       }   
     }

Test - The third stage is testing stage in which we will use the pytest tool to do the unit testing for the code that we have written.

stage('test'){
      steps {
         
        sh "pytest --cov ./ --cov-report html --verbose"
        publishHTML(target:
            [allowMissing: false,
              alwaysLinkToLastBuild: false,
            keepAll: false,
            reportDir: 'htmlcov',
            reportFiles: 'index.html',
            reportName: 'Test Report',
            reportTitles: ''])  
           
        echo "Testing Success"   
          }  
        }

Email - the Final stage is sending the email to the user defined in this stage. I have given mine , you can change that.

The Complete source code is available here. More to Come, Happy learning :-)
Read More

Saturday, January 9, 2016

Python Keywords

Keywords play an essential role in every programming language. Keywords in every language provide language implemented logic that can be directly used in the user programs.

This article will explain the keywords available in Python with examples.

To find the available keywords in the Python language, we can use the snippet

import keyword
print keyword.kwlist

Which will print all the available keywords.

Here are the list of Keywords available in Python.
['and', 'as', 'assert', 'break', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'exec', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'not', 'or', 'pass', 'print', 'raise', 'return', 'try', 'while', 'with', 'yield']

We will see one by one

True & False - True and False are mainly used when evaluating things. They are the results of the Comparison Operations or logical ( Boolean) operations in Python. True and False in python is same as 1 and 0.

>>> 1 == 1
True
>>> 5 > 3
True
>>> 5 < 3
False

>>> True + True
2
>>> False + False
0

None - None in python is just a value that is commonly used to say either "Empty" or "No value there". None in Python is a signal object which means Nothing. There will be only one copy of the None Object in a Python interpreter session.

The same None is available in other languages as Null or Undefined. None in Python is very different as it is not a primitive type rather it is a Object for the Class NoneType.

Variables names are something like a sticker in Python. So you just need to change the Sticker if you assign a new value. When you write
F = "fork"
you put the sticker "F" on a string object "fork". If you then write
F = None
you move the sticker to the None object.

Normally in Python we didn't write the sticker "F", there was already an F sticker on the None, and all you did was move it, from None to "fork". So when you type F = None, you're "reset[ting] it to its original, empty state

if we decided to treat None as meaning empty state`.
  1. Let's confirm the type of None first
  2. print type(None)

print None.__class__
Output
<type 'NoneType'>
<type 'NoneType'>
Basically, NoneType is a data type just like int, float, etc.

None is a singleton object (meaning there is only one None), used in many places in the language and library to represent the absence of some other value.

For example:
if d is a dictionary, d.get(k) will return d[k] if it exists, but None if d has no key k.

apple = "apple"
print(apple)
>>> apple
apple = None
print(apple)
>>> None
None means nothing, it has no value.
None evaluates to False.

Void functions that do not return anything will return a None object automatically. None is also returned by functions in which the program flow does not encounter a return statement.

>>> def void_function():
...     a=1
...     b=2
...     c = a + b
...
>>> print void_function()
None

import - import keyword is used to import packages into our name spaces. That can be used as

>>> import keyword
>>> print keyword.iskeyword("in")
True

As – is a keyword used with import keyword . this is used when creating a alias for a module. It gives a different name to the module while importing

>>> import math as hai
>>> hai.cos(hai.pi)
-1.0

Assert -  is a keyword used in debugging Purpose. While programming we may need to check if the statements are True or not.

Assert helps us to do this . assert is followed by a condition. If the condition is true nothing happens and else throws a Assertion Error

>>> a = 4
>>> assert a < 5
>>> assert a> 5, "oops an Error"
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
AssertionError: oops an Error

Break and continue
break and continue are used inside for and while loops to alter their normal behavior. break will end the smallest loop it is in and control flows to the statement immediately below the loop. continue causes to end the current iteration of the loop, but not the whole loop

>>> for i in range(1,11):
...     if i == 5:
...          break
...     print i
...
1
2
3
4

>>> for i in range(1,11):
...     if i == 5:
...          continue
...     print i
...
1
2
3
4
6
7
8
9
10

def - defining a Function
In order to define a function in python, we need to follow some syntax
1) Function blocks begin with  a keyword def followed by function name and Parentheses(:)
2) All Parameters or arguments needs to be placed in side the Parentheses.
3) The first line in the function should be a Docstring, which tells us what this function do
4) Code blocks starts after the (:)
5) A statement return [expression] exits a function, optionally passing back an expression to the caller

Example –

>>> def myFucntion():
...     """ This is My Sample Function Doc String"""
...     print "this is Sample Function"
...

>>> myFucntion()
this is Sample Function

del - del is used to delete the reference to an object. Everything is object in Python

>>> a =5
>>> print a
5
>>> del a
>>> print a
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
NameError: name 'a' is not defined

>>> a = [ 1,2,3]
>>> del a[2]
>>> print a
[1, 2]

IN :Member ship operators – The Membership operators are the ones that check whether a String or Number is available in a Data Type or any other Type. This “IN” operator is one of the most used one as

>>> str1 ="i have a Lazy Dog"
>>> str1
'i have a Lazy Dog'
>>> "dog" in str1
False
>>> "Dog" in str1
True
>>> list = [1,2,3,4,5]
>>> '1' in list
False
>>> 2 in list
True

As seen above the “IN” Operator can be used to find if an element exists in a Type

IS - Identity Operators – The Identity operators can be used to check the memory location and tell if both are equal. The “IS” Operator is used to perform the operation of checking. This checks whether the values are same as well are the data types.

>>> i = 10
>>> b = 20
>>> if i is b:
...     print "hai"
... else:
...     print "bye"
...
bye
>>> i = 10
>>> b = 10
>>> if i is b:
...     print "hai"
... else:
...     print "Bye"
...
Hai

Not - There is another Variables called “Not” which can be used to check whether they are not available or not

If not a :
Print “not available”

For - A simple for construct would like this,

>>> fruits = ["apple","mango","grape"]
>>> for fruit in fruits:
...     print fruit
...
apple
mango
grape

That reads, for every element that we assign the variable fruit, in the list fruits  print out the variable fruit

Other example looks as,

>>> numbers = [1,10,20,30,40]
>>> sum = 0
>>> for number in numbers:
...     sum = sum + number
...     print sum
...
1
11
…..

While - The While loop tells to perform an Operation as long as the condition defined is met. An basic example would look like this,

>>> fruits =  ["apple","mango","banana","grape"," raspberry"]
>>> i = 0
>>> while i < len(fruits):
...     print fruits[i]
...     i = i+1
...
apple
mango
banana
grape
raspberry

If - An if Statement consists of a Boolean expression followed by one or more statements. Below is an example of the standard if statement

>>> var = 100
>>> if (var == 100 ) : print "value us 100"
...
value us 100

If-else : An If-else statement can be followed by an optional else Statement which executes when the Boolean Expression is False. A snippet of how the If-else works is shown below.

>>> age = input("enter the age?")
enter the age?20
>>> if age<0:
...     print "age is Less than 0"
... elif age==10:
...     print "age is equal to 10"
... else:
...     print "age is Greater than 10"
...
age is Greater than 10

elif - An Extension to the if-else statement which will check for the condition when if case is bypassed. An example would be

>>> age = input("enter the age?")
enter the age?10
>>> if age<0:
...     print "age is Less than 0"
... elif age==10:
...     print "age is equal to 10"
... else:
...     print "age is Greater than 10"
...
age is equal to 10


del - Deletion - Python provides a feature which is not available in many high level languages called “del”. Using this we can delete a reference to a number object.

The syntax of the del statement is − del var1[,var2[,var3[....,varN]]]]

You can delete a single object or multiple objects by using the del statement. For example −
del var

Anonymous Functions - lambda - Python supports the creation of anonymous functions (i.e. functions that are not bound to a name) at runtime, using a construct called "lambda".

One important thing to remember is that Lambda does not include a return statement since it’s an expression that is retuned.

The general format for lambda form is:
lambda parameter(s): expression using the parameter(s)

An example will be,

>>> k= lambda y: y + y
>>> k(30)
60
>>> k(40)
80

except, raise, try - Exception Handling

#!/software/python/2.7

def run(num):
    try:
        r = 1/num
    except:
        print "Exception Raised"
        return
    return r

print run(10)
print run(0)

[djas999@vx181d testing]$ python test1.py
0
Exception Raised
None

Finally - finally is used with try…except block to close up resources or file streams. Using finally ensures that the block of code inside it gets executed even if there is an unhandled exception. 

finally is for defining "clean up actions". The finally clause is executed in any event before leaving the try statement, whether an exception (even if you do not handle it) has occurred or not.

Global - global is used to declare that a variable inside the function is global (outside the function). If we need to read the value of a global variable, it is not necessary to define it as global.

Pass - pass is a null statement in Python. Nothing happens when it is executed. It is used as a placeholder. Suppose we have a function that is not implemented yet, but we want to implement it in the future. Simply writing,

def function(args):
    pass

We can do the same thing in an empty class as well.

Return - return statement is used inside a function to exit it and return a value. If we do not return a value explicitly, None is returned automatically.

Class - A Class is a user-defined prototype for an object that defines a set of attributes that characterize any object of the class. The attributes are data members ie Class variables and instance variables, methods accessed via a dot notation.

Creating Class
In order to create a Class in Python we use the Class Keyword like,

class Emp:
    """ Common Base Class for EMP"""
    empCount = 0

    def __init__(self,name,age):
        self.name=name
        self.age=age
        empCount +=1

    def disEmp(self):
        print "Total Emp are",Emp.empCount

    def disDT(self):
        print "Emp Details are",self.name,self.age


Logical Operators - and, or , not

Not - Negation of a Condition ( not )

>>> a =[ 1,2,3,4]
>>> if 1 not in a:
...     print "hai"
...
>>> if 1 in a:
...     print "hai"
...
Hai

>>> not True
False
>>> not False
True

and - Or  : Logical Operators

var = "hello"
var1 = "mello"

if
var =="hello" and var1 =="mello":
    print "This is Implementation of and"
elif
var == "hello" or var1=="mello":
    print "This is a Implementation of Or"
else
:
    pass

Yield - yield statement pauses the function saving all its states and later continues from there on successive calls. yield is just like return. It returns whatever you tell it to. The only difference is that the next time you call the function, execution starts from the last call to the yield statement.

As an scenario, consider a case where you have a function which will return from 1 to 10. When you use yield we can get the numbers from 1 to 10 , yield will save the state of the function that gave the number. So if that gave a number 5 now , yield statement will save the state of function that it has to return 6 when the same function was used at any time in the program.

With With keyword allows us to cleanly close the resources that are being used. For example a classic example of opening a file , zipping the file can be done using the file Operations but using With keyword we can write the same thing as

with open(file_path, 'r+') as f_in, gzip.open(file_path+".gz", 'wb') as f_out:
        f_out.writelines(f_in)
        os.remove(file_path)

The above with statement will automatically close the file after the nested block of code. The advantage of using a with statement is that it is guaranteed to close the file no matter how the nested block exits. If an exception occurs before the end of the block, it will close the file before the exception is caught by an outer exception handler. If the nested block were to contain a return statement, continue or break statement, the with statement would automatically close the file in those cases, too
Read More