My Technical Works: 2015

Wednesday, December 30, 2015

Jenkins with Github

In this article we will see how we can configure Jenkins with Github. Our goal is to make Jenkins pull code from the Github repository and build that code locally.

1) The basic steps are to configure the Github with a repository and push some code. Use the articles from the Git.

2) Once the code is pushed to the Github, we need to then make configuration changes on the Jenkins side so that jenkins can pull the code from the Github repository and build the code locally

3) Git hub provides some thing called as the hooks which allows the communication. The post-receive hook of Github are actually only "WebHooks", to communicate with a web server whenever the repository is pushed to.

First lets see the changes that need to be done from the GitHub side

1) Click on the Settings of the repository in Github. Then go to the Webhooks and Services.

2) In the service , search for Jenkins for Github service and choose

Now once you choose the service ,you need to enter the locally run jenkins details in the jenkins hook url as above.

Make sure you get you IP address ( using ifconfig command in linux) and configure the URL as

http://<IP address>:<Port>/github-webhook/

the next step is click on the "Test Service" which will return a payload. If the payload was success, then the configuration test is fine.

2) Configurations need to be done on Jenkins. Once the Configuration on the Github side is done, we need to make some configuration changes on the Jenkins side to connect to the Github and get the source code for building

Choose a New Item from Jenkins for building the Code. Go for the project type if that is maven and click Ok . in the next screen configure the GitHub repository details as shown in the screen shot,

Provide the Github repository details in the Github project URL for the Build Item. Also provide the Github credentials if needed to connect.

In the below part of the same screen , choose the "Poll SCM" check box and add " H/5 * * * *" which will poll the source code for every 5 minutes and build the code.

Jenkins does provide a options to build when a Push to the Github from the local Git is done but Since your jenkins is not accessible from a public IP address, GitHub can not trigger the build via WebHook

So we use the Poll mechanism to poll code for every 5 minutes from Github and build the code. Once the Poll is done we see some thing like this

More to Come,Happy Learning

As we already discussed that GitHub is a web based Git repository hosting service. It offers all of the distributed version control and source code management functionality of Git as well as adding it own features. This article will provide you the basic of the Git hub configuration.

1) Go to the Github.com and create a account

2) The next step is to create a repository as follows,

3) Once the repository is created, we can see the repository as follows,

4) Once the repository is created, we need to make sure to get the Repository HTTPS and SSH URLs from the repository it self

5) Once the Repository is created and code is pushed to the Repository, we can see the code as follows. Git Hub also allows us to merge , compare the branches.

6) Git Hub provides us with various other options such as below to connect local jenkins with the remote Git Hub code for building purpose. This is done by using the webHooks and Services.

I hope this gives you the basic idea of the Git Hub Configuration.

Most people now are days are not using the remote repository a local one. Instead they are using the online repositories for storing their code. If the application is an Open source one , keeping the code available online can make developers download the code and make the changes freely. Git Hub is one such thing.

GitHub is a Web-based Git repository hosting service. It offers all of the distributed revision control and source code management (SCM) functionality of Git as well as adding its own features. As of now GitHub is largest online storage space of collaborative works that exists in the world.

Github provides access control and several collaboration features such as wikis , task management and bug tracking and features requested by projects

In general terms,

Git = local (on your computer ) , Github – remote (web and online)

Some of the other advantages of GitHub are

Share your repositories with others.
Access other user's repositories.
Store remote copies of your repositories (github servers) as backup of your local copies.

Some of the terms that we need to understand while using GitHub are,

Forking - “Forking” is when you create a new project based off of another project that already exists. This allows other people and developers to use the existing code and extend the features. If you find a project on GitHub that you’d like to contribute to, you can fork the repo, make the changes you’d like, and release the revised project as a new repo. If the original repository that you forked to create your new project gets updated, you can easily add those updates to your current fork.

Pull – So you made a fork of the project and now want to make a revision of the project. You can do this by using the pull requests from the original one.

Social networking - The social networking aspect of GitHub is probably its most powerful feature, and is what allows projects to grow more than anything else. Each user on GitHub has their own profile, which can act like a resume of sorts, showing your past work and contributions to other projects via pull requests.

Changelogs - When multiple people are collaborating on a project, it’s really hard to keep track of who changed what, and to keep track of the revisions that took place. GitHub takes care of this problem by keeping track of all the changes that have been pushed to the repository.

These are the basics of Git-Hub. More to Come . Happy learning

Python Functional Programming

Python is classified as a "hybrid" language. It supports the functional programming paradigm, but equally supports imperative paradigms (both procedural and object-oriented).

Functions as Objects - Functions are first-class objects in Python, meaning they have attributes and can be referenced and assigned to variables.

High Order functions - Python also supports higher-order functions, meaning that functions can accept other functions as arguments and return functions to the caller.

List comprehension is nothing but a way of generating new lists from existing list(or from any other iterable data type).

Map - Most of the times when performing basic Operations on each element of a list or set ,we use the for loop to traverse over them and apply a operation on each element like,

>>> items = [1,2,3,4,5]

>>> sq = []

>>> for x in items:

... sq.append(x ** 2)

...

>>> print sq

[1, 4, 9, 16, 25]

But Python provides better ways for doing these common operations day to day, one such built-in feature is map. The map(aFunction, aSequence) function applies a passed-in function to each item in an iterable object and returns a list containing all the function call results.

>>> items = [1,2,3,4,5]

>>> sq = []

>>> def sqr(x): return x**2

...

>>> list(map(sqr,items))

[1, 4, 9, 16, 25]

If we see the above snippet ,we can see that a list of items is passed to the map function along with a sqr function that is already written. The sqr function will square the element passed in the items list.

Since map expects a function to be passed , it is the most common place that is to be used with the lambda routine

>>> list(map((lambda x:x**2),items))

[1, 4, 9, 16, 25]

Map has some performance benefits when compared with the normal for loop that we write. More over map has some more additional benefits like taking arguments in sequence in parallel from a multiple sequence arguments like,

>>> list(map(pow,[2,3,4],[2,3,4]))

[4, 27, 256]

If the function passed is none, an identity function is used on them. if there are multiple arguments, map()returns a list consisting of tuples containing the corresponding items from all iterables.

>>> map(None,[1,2,3])

[1, 2, 3]

Filter –

The filter filters out items based on a test function which is a filter and apply functions to pairs of item and running result.

>>> for x in range(-5,5):

... if x<0:

... print x

...

-5,-4,-3,-2,-1

The Similar can be implemented using the filter as,

>>> list( filter( (lambda x:x<0),range(-5,5) ))

[-5, -4, -3, -2, -1]

Reduce

The reduce routine works in a different way. It applies the function to all the elements passed and then returns one result.

Consider we need to find a highest number in the series [1,3,5,2,6] , now if we have a function written to compare 2 elements as

f = lambda a,b: a if (a > b) else b

and pass this to the reduce routine as

>>> reduce(f,[1,3,5,2,4])

The reduce will take the first 2 arguments and use the function f on them , then take the obtained result with the 3 number and perform the function f on those both. This happens until last and a single value will be the result

The built-in reduce also allows an optional third argument placed before the items in the sequence to serve as a default result when the sequence is empty.

Agile , a process which require a disciplined approach to ensure that customer feedback, continuous testing, and iterative development actually lead to frequent deliveries of working, valuable software.

Similarly DevOps is a software development method that stresses communications, collaboration, integration, automation and measurement of cooperation between software developers and other IT professionals

In Simple words, a Way for developers and IT Operations people to co-operate with each other in releasing of the Software while automating the process of software delivery and infrastructure changes.

The Problem – So what made the DevOps?

Software development can be considered as a community of developers and testers writing code, testing, re-writing etc. Operations teams are all about looking after the systems that run that code. These are the People who work out how much processing power the software need to run, how to make that secure, running that efficiently and how to keep that running.

The main problem in this is that the developers are not aware of the infra-structure process, not knowing what is the runtime being used for their application code to run. Operations Teams on the other hand are not aware of the run time area where the code was developed.

There are cases where the code is developed on Windows 2000 workstations, and tested with Tomcat on Windows, and then deployed on *nix machines, with software load balancers, different Java versions and different Tomcat versions. This is not to mention the entirely different properties files on developer workstations compared to live machines.

The issues start to arise because those two groups of people look after the same systems but otherwise exist in two relatively separate worlds. The way developers tend to look at it, they are the ones who create the software which the operations staff then goes on to break. And then they shout at each other.

Here comes the DevOps. Seeing both developers and IT Operations as two different entities, the DevOps movement tells us about the communication, collaboration and integration between developers and IT Operations.

DevOps movement makes developers and Operations people to work together in understanding the infrastructure being used, High level languages being used in developing the code, understanding the delivery process , deployment model, Runtime being used thus making both parties to stay in similar production like environment during development.

The Other Problem - Traditional deployment process requires weeks and months in moving a new release to production. DevOps comes to help the organization to produce Software and IT Services Rapidly with frequent iterations. This is done by integrating developers and Operations teams to improve collaboration and productivity by automating infrastructure, automating work flows and continuously measuring application performance.

Developers would write software in small chunks that are integrated, tested and monitored and deployed usually in hours versus the traditional way of writing large chunks of software over weeks and do weeks of testing. This allows them to increase the frequency of deployment and improve time to deploy new code.

Infrastructure as Code

Every System admin has the experience of getting waked up early in the morning saying that a machine is not reachable. Admin's generally follow the usual Quick fixes only to find out the machine is actually crashed.

We then go back to our memories to find what we did during the last crash. What scripts were ran , what packages and components were installed and in what sequence then It strikes you that you cannot even recall the order in which everything was installed.

Every developer or support person dreads a moment like this when they have to reconfigure the servers, because of the different aspects involved in the tedious process. Though there are documentation available that outline every steps in dealing with situations like these but consider the time you need to invest in order to accomplish your tasks manually.

Infrastructure as Code talks about writing code to manage configuration and automate provisioning of infrastructure in addition to deployments. This does not talk simply about writing scripts but involves using tested and proven software development process are used in this like using version control, testing, small deployments, use of design patterns etc. In short, this means you write code to provision and manage your server, in addition to automating processes.

How does the developers work in this, the knowledge of server provisioning, configuration management and deployment is no longer only with the systems Admin's. Even developers can easily engage in the activities, because they can easily write infrastructure code in the languages that they are familiar with

The Devops movement is characterized by people with a multidisciplinary skill set - people who are comfortable with infrastructure and configuration, but also happy to roll up their sleeves, write tests, debug, and ship features.

More to Come :-)

Git with Git-hub

In this article we will see how we can configure our local repository to connect to the Git hub remote repository and push our code to the remote repository.

1) Initialize a Git repository using "git init"

2) Now create a couple of files. add and commit to the local repository

[root@vx111a Mytest]# git add ReadMe

[root@vx111a Mytest]# git add Hai.java

[root@vx111a Mytest]# cat hai.java

public class hai {

public static void main(String st[]) {

System.out.println("This Is Git Sample");

}

[root@vx111a Mytest]# git commit -m "testing GIT"

[master (root-commit) bad4374] testing GIT

2 files changed, 6 insertions(+)

create mode 100644 Hai.java

create mode 100644 ReadMe

3) Once the files are added to the local repository we need to add these files to the remote repository but before that we need to add the remote repository details to the Git configuration.

check the remote repository configured for the local repository using,

[root@vx111a Mytest]# git remote show origin

* remote origin

Fetch URL: https://github.com/jagadish12/MyTest.git

Push URL: https://github.com/jagadish12/MyTest.git

HEAD branch: master

Remote branch:

master tracked

Local ref configured for 'git push':

master pushes to master (up to date)

or we can also use,

[root@vx111a Mytest]# git config --get remote.origin.url

https://github.com/jagadish12/MyTest.git

The remote origin is already configured over here , but if that is not configured we dont see the above details.

For adding the remote configuration details, we can use

git remote add origin https://github.com/jagadish12/SampleTest.git

This will add the remote configuration information. Once the remote configuration is added, we can then push our local code committed to the remote repository using,

[root@vx111a Mytest]# git push origin master

Username for 'https://github.com': jagadish12

Password for 'https://jagadish12@github.com':

Counting objects: 3, done.

Delta compression using up to 8 threads.

Compressing objects: 100% (2/2), done.

Writing objects: 100% (2/2), 274 bytes | 0 bytes/s, done.

Total 2 (delta 0), reused 0 (delta 0)

To https://github.com/jagadish12/MyTest.git

bad4374..089a50e master -> master

The git Push asks for the Github User name and password for pushing the data to the remote location ( aka remote repository )

Now we can login to the Git Hub and check the updated file changes in the repository that we created.

Now if we want to remote the remote repository configured for a Git, we can use

[root@vx111a test]# git remote remove origin

[root@vx111a test]# git remote add origin https://github.com/jagadish12/SampleTest.git

More to Come

Github provides access control and several collaboration features such as wikis , task management and bug tracking and features requested by projects

In general terms,

Git = local (on your computer ) , Github – remote (web and online)

Some of the other advantages of GitHub are

Share your repositories with others.
Access other user's repositories.
Store remote copies of your repositories (github servers) as backup of your local copies.

Some of the terms that we need to understand while using GitHub are,

Pull – So you made a fork of the project and now want to make a revision of the project. You can do this by using the pull requests from the original one.

These are some of the basics of Git-Hub. More to Come

In this article we will see how we can configure git with Maven. For this we need to create a Git Repository and then create a maven based application on this. The same thing can be further used in Jenkins with Git Configuration.

The important thing to remember here is when you run the maven command make sure you create a directory with the application name and in side initialize the git and run the maven command in there. Create a directory sampleTest and inside that directory follow the steps,

1) run the maven sample web application command as

mvn archetype:generate -DgroupId=com.git.testing -DartifactId=SampleTest -DarchetypeArtifactId=maven-archetype-webapp -DinteractiveMode=false

2) Check the Status and we will see that we have a CounterWebApp application created.

[root@vx111a testing]# git status

# On branch master

# Initial commit

# Untracked files:

# (use "git add <file>..." to include in what will be committed)

# CounterWebApp/

nothing added to commit but untracked files present (use "git add" to track)

[root@vx111a testing]# ll

total 0

drwxr-xr-x. 3 root root 30 Dec 3 14:47 CounterWebApp

Add and Commit

[root@vx111a testing]# git add .

[root@vx111a testing]# git status

# On branch master

# Initial commit

# Changes to be committed:

# (use "git rm --cached <file>..." to unstage)

# new file: CounterWebApp/pom.xml

# new file: CounterWebApp/src/main/webapp/WEB-INF/web.xml

# new file: CounterWebApp/src/main/webapp/index.jsp

[root@vx111a testing]# git commit -m "Jenkins Testing with Local Repo"

[master (root-commit) 551c28c] Jenkins Testing with Local Repo

3 files changed, 33 insertions(+)

create mode 100644 CounterWebApp/pom.xml

create mode 100644 CounterWebApp/src/main/webapp/WEB-INF/web.xml

create mode 100644 CounterWebApp/src/main/webapp/index.jsp

Check the Status again which is clean

[root@vx111a testing]# git status

# On branch master

nothing to commit, working directory clean

Note – Make sure you add the <plugins> information in the pom.xml file if you want the web app to be packaged.

More to Come.

We have seen the basics of Git and how to create a basic repository and commit files. In this next article we will see the advanced topics of git.

Git Configuration – In order to communicate with the remote repository from our local repository we need to configure user name and email so that git repository will identify who has done operations on the repository. In order to configure those use the below commands as,

git config --global user.name <user Name>

git config --global user.email <Email ID>

Remote Repository Details – Though we have the local repository we need to have a remote repository where we can push our changes so that other can access them. In order to do this, we need to first add the remote Repository URL. This can be done by using

git config remote.origin.url https://github.com/abc/abc.git

Once the remote repository is set, we can get the details using

[root@vx111a master]# git config remote.origin.url
https://github.com/abc/abc.git

Configuration Details

In order to get the configuration details like user.name and email etc, we can get those details using

[root@vx111a master]# git config -l
user.name=jagadish12
user.email=jagadesh.manchala@gmail.com
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=https://github.com/abc/abc.git

Remote Location Details

In order to view the remote location details we can use,

[root@vx111a Mytest]# git remote -v

origin https://github.com/jagadish12/MyTest.git (fetch)

origin https://github.com/jagadish12/MyTest.git (push)

[root@vx111a Mytest]# git ls-remote --get-url origin

https://github.com/jagadish12/MyTest.git

Clone a Repository

Cloning is nothing but a way to access a repository. Checking-out in SVN is similar to Cloning in GIT. Running git clone <repository URL> will pull in a complete copy of the remote repository to your local system. Now you can work away on it, making changes, staging them, committing them, and pushing the changes back. We can clone using

[root@vx111a test]# git clone https://github.com/jagadish12/SampleTest.git

Cloning into 'SampleTest'...

remote: Counting objects: 24, done.

remote: Compressing objects: 100% (16/16), done.

remote: Total 24 (delta 1), reused 24 (delta 1), pack-reused 0

Unpacking objects: 100% (24/24), done.

Branching

We need to make sure that our master repository is always clean. Clean in the sense with out any changes. For the changes that we are doing before moving them to the master we need to make sure that we create a branch of the code and make the changes to the branch. Creating branches in git is done using

[root@vx111a]# git checkout -b test

Switched to a new branch 'test'

Check the Branch Status -

[root@vx111a]# git status

# On branch test

nothing to commit, working directory clean

Check the available branches -

[root@vx111a]# git branch

master

* test

Delete the whole branch using

[root@vx111a]# git checkout master

[root@vx111a]# git branch -D test

For deleting we need to first move out of branch and then we need to delete that branch

When you do a pull request on a branch, you can continue to work on another branch and make another pull request on this other branch.

Merging

Once your feature is finished, you’ll want to merge your branch back to the master branch. This is way easier than it would be in Subversion. You just execute:

git checkout master

git merge <feature_branch>

If there are conflicts, you will be prompted to fix them. Add the conflict fixes to the changeset with git add, and then commit.

Git add and Commit

Once the changes are done, we need to first make sure the changes are added to the local repository and then once added need to commit to the remote repository

For adding the file use, git add <file Name> or git add . ( all files)

For committing the file, we need to use the command,
git commit –m “Message for commit”

If we want to commit only specific files we can use

Git commit file1 file2

Undo Git Commit

There are cases where we need to revert back the commits done. Git provides us the ways using

git reset <file> - which will remove it from the current index (the "about to be committed" list) without changing anything else.

Or we can use “git reset” without any file name to unstage all due changes. This can come in handy when there are too many files to be listed one by one in a reasonable amount of time.

Reset

Consider that we called git add -all, now all files are added. I did not make a commit and push. How can i undo my action?

Git provides us a way for undoing the action by using “git reset HEAD”

Push to origin

Once the changes are committed to the local repository, we need to make sure the changes are pushed to the remote for other developers to access. This can be done using

[root@vx111a SampleTest]# git push origin master

Username for 'https://github.com': jagadish12

Password for 'https://jagadish12@github.com':

Total 0 (delta 0), reused 0 (delta 0)

To https://github.com/jagadish12/SampleTest.git

* [new branch] test -> test

This will push the local repository changes to the remote repository. master is the local repository.

Git Logging

As other SVN tools, Git supports logging. We can use commands for getting the basic details like,

[root@vx111a SampleTest]# git log

commit 3b65aa9e103740a323ee157b93c6f5a4785e4f25

Author: jagadish12 <jagadesh.manchala@gmail.com>

Date: Fri Dec 4 13:44:51 2015 +0530

Hello World Index Changed

commit c384865e37e6848a847859b94a1d7e50d61f2c31

Author: jagadish12 <jagadesh.manchala@gmail.com>

Date: Thu Dec 3 19:54:53 2015 +0530

second Check

commit b96af507d8e407e6c99b8a1e6c6ef5f61152657c

Author: jagadish12 <jagadesh.manchala@gmail.com>

Date: Thu Dec 3 19:52:39 2015 +0530

Sample Test Checking

The “git log” command gives us the basic details of the recent commits and who has done that.

The logging can be minimized using ,

[root@vx111a SampleTest]# git log --oneline

3b65aa9 Hello World Index Changed

c384865 second Check

b96af50 Sample Test Checking

Pretty logging is also available as,

[root@vx111a SampleTest]# git log --pretty

commit 3b65aa9e103740a323ee157b93c6f5a4785e4f25

Author: jagadish12 <jagadesh.manchala@gmail.com>

Date: Fri Dec 4 13:44:51 2015 +0530

Hello World Index Changed

commit c384865e37e6848a847859b94a1d7e50d61f2c31

Author: jagadish12 <jagadesh.manchala@gmail.com>

Date: Thu Dec 3 19:54:53 2015 +0530

second Check

commit b96af507d8e407e6c99b8a1e6c6ef5f61152657c

Author: jagadish12 <jagadesh.manchala@gmail.com>

Date: Thu Dec 3 19:52:39 2015 +0530

Sample Test Checking

Some more logging command include,

git log --graph --oneline --decorate --all
git log --name-status

Remove Remote Branches

Git also provides us ways to remove Branches in the Remote repository. This can be done using

git push origin --delete <branchName>

Git History

Git provides us various commands in working with the History. We can use the basic command like

[root@vx111a git-test]# git log --author=jagadish12
commit c0a6f2e99ae634af6f8d13d6036fa751f9e558c7
Author: jagadish12 <jagadesh.manchala@gmail.com>
Date:   Thu Dec 3 19:41:37 2015 +0530

    checking

commit 1419b2dce7162f8ef8bda8f9cace0e60beb70c4a
Author: jagadish12 <jagadesh.manchala@gmail.com>
Date:   Thu Dec 3 18:48:46 2015 +0530

    hai Comitted

commit 00904bbce573da14764b28c9e0f527e4d00833d1
Author: jagadish12 <jagadesh.manchala@gmail.com>
Date:   Thu Dec 3 17:48:52 2015 +0530

    first Check

You can also easily match on multiple authors as regex is the underlying mechanism for this filter. So to list commits by Jonathan or Adam, you can do this:

git log --author="\(Adam\)\|\(Jon\)"

Differences

Git also provides ways for identifying the differences between multiple branches using

git diff <source_branch> <target_branch>

Tag

it's recommended to create tags for software releases.This is a known concept, which also exists in SVN. You can create a new tag named 1.0.0by executing
git tag 1.0.0 1b2e1d63ff
the 1b2e1d63ff stands for the first 10 characters of the commit id you want to reference with your tag.

By this we had discussed the advanced topics of Git. More to Come

Subscribe to: Posts ( Atom )

My Technical Works

Pages

Wednesday, December 30, 2015

Jenkins with Github

Basic Git-Hub Configuration

Basics of Git-Hub

Monday, December 28, 2015

Python Functional Programming

Basics of DevOps

Git with Git-hub

Basics of GIT-HUB

Maven with GIT

Advanced Git