One liners for Every Day Usage

sed -i '/^$/d' filename : Remove Duplicate Lines.

sed -i '/hello/d' filename: Remove the Line that has hello and modify the file.

cat filename | tr "," "\n" : Arrange comma (,) separated Strings one by one 

cat filename | tr -s '\n' : Remove the Blank Lines.

find . -type f -newermt '2010-01-01' : Compare to Date.

find . -type f -newermt '2010-01-01' ! -newermt '2010-06-01’: Between Dates.

awk -F':' '{printf "%-16s %-16s\n",$1,$2}' filename: Pretty printing.

sed -n 's:.*<id>\(.*\)</id>.*:\1:p' filename :Extract the content between the <id> elements.

sed -i '/^\(report\|-t\(h\|o\)\)/!d' filename :Remove all lines that dont start with string report.

awk -F"-" '{$NF = ""; print}' OFS="" filename : Remove the last character after ‘-‘ .

awk '{print FNR "\t" $0}' filename : Line Number Much like nl command.

awk '{print (NF? ++a ":" :"") $0}' filename : Line Number Much like nl command with no blank lines

awk 'END{print NR}' : Count Lines.

awk '/hello/{n++}; END {print n+0}' filename: Count lines with hello string.

awk '{ print NF ":" $0 } ' filename : Number of Fields in Each Line

awk '{ print $NF }'  : Print the last field of each line

awk '{ field = $NF }; END{ print field }' :  Print the last field of the last line

awk 'NF > 4' : print every line with more than 4 fields

awk '$NF > 4' : print every line where the value of the last field is > 4

awk '{sub(/^[ \t]+/, "")};1' filename : delete leading whitespace (spaces, tabs) from front of each line

awk '{$1=$1};1' filename

awk '{ print NR, $0 }' filename: add number to the lines in file

list -i RIM -e dev | awk '{print $1}' | awk -F"-" '{$NF = ""; print}' OFS="" : remove the first and last lines in output

cat filename | awk -F, '{print NF}' : When Comma is the Separator

cat /etc/hosts | awk '/localhost/ { print $1;}'

echo "test1:test2:test3" | awk 'BEGIN { FS = ":"} { print $2 }': with Field Seperator

awk '{print NR, $0}' test.txt : content of File with line number

awk '/PATTERN/{num++} END{ print num }' file.txt : count the number of lines that contain a pattern

nmap -p <Port No> <Remote Machine>: Hit a port on remote machine

egrep '(string1|string2|stringN)' file.txt : Print lines which contain string1 or string2

sed 's/.$//' or sed 's/^M$//' or sed 's/\x0D$//' : converts a dos file into unix mode.
sed "s/$/`echo -e \\\r`/" or sed 's/$/\r/' or sed "s/$//": converts a unix newline into a DOS newline.

sed 's/^[ \t]*//;s/[ \t]*$//' : Delete both leading and trailing white space and tab in a file.

awk '/CP-A/ { n++ }; END { print n+0 }' fileName : print the total number of lines containing the word pattern.

seq ‘/pattern1/,/pattern2/d’ < inputfile > outfile : will delete all the lines between pattern1 and pattern2.
sed ‘/20,30/d’ < inputfile > outfile : will delete all lines between 20 and 30.   OR sed ‘/20,30/d’ < input > output will delete lines between 20 and 30.

awk '/baz/ { gsub(/foo/, "bar") }; { print }' : Substitute foo with bar in lines that contains ‘baz’.

awk '!/baz/ { gsub(/foo/, "bar") }; { print }' : Substitute foo with bar in lines that does not contain ‘baz’.

awk '$5 == "abc123"' : print each line where the 5th field is equal to ‘abc123’.

awk '$5 != "abc123"' : print each line where 5th field is NOT equal to abc123. 

Compare to file:

find . -newer foo.txt

find . ! -newer foo.txt 

awk String comparison:

ll | awk '$9 == "two" {print $3"="$9}'

ll | awk '$7 >20 {print $9}' 

Find Jar files for class:

find . -name '*.jar' -print0 |  xargs -0 -I '{}' sh -c 'jar tf {} | grep hello.class &&  echo {}' 

find $PWD -type f -name "*Jul-*.ESS-A1.log" -exec rm -f {} \; : Find and Remove Files Matching a pattern 

find $PWD –type f –name “controllermessages.log.” | xargs tar zcvf one.tar : Find and Zip files matching a Pattern 

find . -name "*.jar" | xargs -tn1 jar tvf | grep --color "log4j.xml" : Search The File From Multiple Jar Files 

Count Threads OF a JVM Process : ps uH p | wc –l

Find how many files and application is using : lsof +c 0 | cut -d' ' -f1 | sort | uniq –c

Find and Kill a Process: Ps ux | grep | grep –v grep | awk ‘{print $2}’ | xargs –r kill -9

Kill a Process on the Port : kill -9 `lsof -t -i :port_number`

Find The Command Line Of the Process Using Specific Port : cat /proc/$(lsof -ti:631)/cmdline

Find out what is listening on a series of ports : /sbin/fuser -n tcp {7000..8000}

Get the 10 Biggest Files : du -sh * | sort -n | tail

Convert Of Bytes to Mbs : units --terse "3415014314 bytes" "MB"

SSh and Execute a Command: ssh root@<Remote Machine> -q 'echo $MYDIR’

Find out which process is using up your memory using ps, awk, sort : ps aux | awk '{if ($5 != 0 ) print $2,$5,$6,$11}' | sort -k2n

Find The Command Line Of the Process Using Specific Port : cat /proc/$(lsof -ti:631)/cmdline

Ping a Host Using Specific Interface ( Or can check Internet Connection for an Interface) : ping -I eth0 www.yahoo.com 

Scp To transfer a File:

Scp /root/file1 root@10.5.11.193:.

‘.’ Says to the root folder. You can use /tmp or any thing to copy to a specific location 

Highest CPU: ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10 | awk "{ print $2 }"

Memory Usage: sar -q 1 | tail -1 | awk '{ print "" $3}' | sed 's/%//g'

Finding the CPU Threshold : top -b -n 1 | awk -F'[:,]' '/^Cpu/{sub("\\..*","",$2); print $2}'

Top 10 Process : ps -efF "%x %p %P %U %u %y %a" | sort -r | head

List Process By memory usage :ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS

Who Started this process: ps -o comm= -p $(ps -o ppid= -p 28453)

How Much Ram Is Being Used : ps -o rss -C java | tail -n +2 | (sed 's/^/x+=/'; echo x) | bc

Find When a Process Was Started : ps -o lstart <PID>

Find the Class File in jars: find . -name "*.jar" | while read line; do unzip -l $line; done | grep <Name>

Using Find to Get Directories: find .  -path "*<DIR Name>" –print

Find Using Regex to find class and java files available :find . -regex ".*\(\.class\|\.java\)$"

Top 20 File handling process:

for x in `ps -eF| awk '{ print $2 }'`;do echo `ls /proc/$x/fd 2> /dev/null | wc -l` $x `cat /proc/$x/cmdline 2> /dev/null`;done | sort -n -r | head -n 20 

lsof -n -p <PID> | awk '$4 != "mem" {print}' : memory mapped File x

grep multiple Strings: grep 'ListenAddress\|port\|clusters\|server_cluster'

query by version and release : rpm -q --queryformat '\n%{NAME} %{VERSION} %{RELEASE} %{ARCH}\n' <package Name>

CPU / Memory Metrics : sar -urWqR 1

Multi CPU Metrics : sar -x SELF -I SUM -P ALL -wu 1

Disk Metrics : iostat -d -x 1 2

Network Metrics : netstat –s or sar -n DEV -n EDEV 1

Processes Metrics : ps -eo

Find and grep : find . -type f -exec grep -l java.lang.OutOfMemoryError {} \;

Who is using Port :

/sbin/fuser 10011/tcp

or

lsof -i tcp:10012

Hit a URL : curl -sL -w "%{http_code}\\n" "http://vx111a:10011/wls_monitor/" -o /dev/null

Show connections based on the host and the port using @host:port :

lsof -i@192.168.1.5:22

lsof -i@eth0.omhq111a.nova.com

Threads in a Process : ps -eLo pid,ppid,tid,pcpu,comm | grep PID

Differences between 2 files in remote hosts:

diff <(ssh alice cat /etc/apt/sources.list) <(ssh bob cat /etc/apt/sources.list)

Find all files Older than 2 days and gzip them:

find $PWD -type f -mtime +2 | xargs gzip

Find and Gzip:

find $PWD –type f –name “*.log” | xargs tar zcvf one.tar

How to list all unique ip address currently connected to a specific port:

ss -o state established '( dport = :10012 )'|awk -F"[\t :]+" 'NR!=1{ ip[$5]+=1 } END{ for (i in ip){n++};print n }'

All process Running as a Specific User:

pgrep -l -f -x -u root

List all the Jars loaded by a Process:

lsof -p PID | grep jar

Return Number of kernel Threads Owned by a process:

ps -o thcount –p PID

Show Ports that belong to this PID:

netstat --all --program | grep PID

Drop all the Connections available now for a Port:

iptables -I INPUT -p tcp -dport 80 -j DROP

SSh and Execute a Command:

ssh root@ -q 'echo $MYDIR’

Find out what is listening on a series of ports:

/sbin/fuser -n tcp {7000..8000}

View details of network activity between:

lsof -i :7000-8000

List all files opened by a particular command:

lsof -c java

List Threads By PID along with Thread Start Time:

ps -o lwp,lstart --pid PID

Extract files from war/Ear:

jar xf abc.war log4j-test.xml WEB-INF/

All Connections from a Specific process:

lsof -p PID -a –i

show the number of connections active to a port and also the number of connections from that ip in order:

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort –n

determine which application is utilizing a certain port? :

lsof -w -n -i tcp:80 ( or any Othe Port)

Owner of the File:

/sbin/fuser admin.lok

List of Files That Are Open For Writing to Disk:

lsof | grep -e "[[:digit:]]\+w"

See which Process is Holding the File:

lsof -r1 /common/jboss.log

List the files accessed by a program:

strace -f -o foo.trace su user -c 'mycommand'?

Environment Variables belong to a Process:

ps ewwo command PID | tr ' ' '\n' | grep \=

List Threads by Pid along with Thread Start Time:

ps -o pid,lwp,lstart --pid PID -L

CPU usage for EACH cores:

ps ax -L -o pid,tid,psr,pcpu,args | sort -nr -k4| head -15 | cut -c 1-90

Largest File Or Directory:du -sk /var/log/* | sort -r -n | head -10

sed -e '1d;n;d' file : Remove every Second Line

sed '$d' : Remove the last Line  

Remove Even Number lines (2,4..) : sed -n '1~2p' file

Remove Odd Number Lines (1,3..) : sed -n '2~2p' file

Remove lines which contain more than 3 char in the second column : awk -F, 'length($2)<=3'

Delete line containing specific number of char (ex:5) : awk 'length<=5' file

List all lines which contain more than 1 Capitalized Character : grep -o '\b[[:alpha:]]*[[:upper:]][[:alpha:]]*[[:upper:]][[:alpha:]]*' file

Remove Lines which start with Capital Words : sed -r 's/\b[A-Z](\w*)\b//g' file

Print Words that Start with Vowels :  sed -r 's/\b[^AEIOUaeiou]\w*//g'  file

Files not being Accessed : find  -mtime +7 -type f   -exec sh -c '/sbin/fuser {} >/dev/null 2>&1 || echo {}' ';'

Run a Command and Display yes or No : status=`jinfo -sysprops $id 2>&1 | grep nodeName 2>&1 >/dev/null  && echo "yes" || echo "no"`

Clean the PID returned by Pgrep Command : id=`echo "$Pid" | grep -v  "^$$"`

/usr/bin/ssh -o "StrictHostKeyChecking no" -n  userName@Virtuals 'cat | bash /dev/stdin' $hello < ./appVal.sh : Run a Script on the Remove Machine by passing an argument $hello

tail Multiple Log Files same Time : tail -f /var/log/syslog -f /var/log/auth.log

Remove "" from first and last in Variable : echo $VAR | sed "s/\"//g"

Env to Sed : env | sed -n /"$HOSTNAME"/p

Filter in Vi : 1,$!awk '{print $1}' | sort | tr [:lower:] [:upper:]

Save from 6 - 9 to a File : 6,9w >> /tmp/newfile

Comment from Line 5-10: 5,10s/^/#/g

Comment all Lines : %s/^/#/g

External Connections made : lsof -n -i  | grep ESTABLISHED

External Connections : lsof -iUDP -P -n | egrep -v '(127|::1)'

10Mb of File : dd if=/dev/zero of=C:/temp/testfiles/10MB.testfile bs=1024 count=12400

Sub String with awk : echo "12345678" | awk '{print substr($0, 3, 2);}'

Which User is using the Port : /sbin/fuser -nu tcp 10001

list and kill any processes currently using /mount : fuser -vmk /mount

How do I monitor opened files of a process in realtime?: strace -e trace=open,close,read,write,connect,accept start.sh AS-A2

All Ports : lsof -iUDP -P -n | egrep -v '(127|::1)'

Command Line arg of the Process : tr '\0' '\n' </proc/1234/cmdline

cut -d: -f1,2,7 passwd : print 1,2and 7 Fields

cut -d: -f5-8 passwd  : print Fields between 5 and 8

cut --output-delimiter=, -d: -f1- passwd : Output with a Delimited “,”

cut -c1-25 /etc/motd : print Char 1-25

cut --complement -c10-25 :print every thing leaving 10-25

Break Into Commas: command | paste –sd,

Case-Insensitive Find File Search :find . -print –iname “pattern”

Find and Unzip : find $PWD -name 'one.zip' -print -exec unzip -l {} \;

Exclude svn data : zip -9 -r --exclude=*.svn*  foo.zip [directory-to-compress]

Hit a URL : wget --inet4-only http://vx112:11011/hell.jsp

Remove the first 2 char from  a file : sed 's/^.\{,2\}//' mental1

Push Data to a port on IP address : nc 67.206.9.118 59312 <<< "<dasj>Sending Data By Jagadish."

sort -k  1 -s file : Sort 1 column in a  file with out touching the second column

rpm -q --all –last : Show the rpm installed along with dates

awk '{ print $14,$15 }' /proc/P12363/stat : percentages of CPU utime and stime of each process?

Last reboot : display last reboot time

sed 's/.$//' file : remove last char in the line of a File

Linux Directory Sizes : find . -type d | grep / | du –h

ps auwwx|gawk '!/%MEM/ {print $4,$11}'|sort -nr|head -n20 : Top 20 process by Memory

ps auwwx|gawk '{count[$NF]++}END{for(j in count) print ""count[j]":",j}'|sort -nr|head -n20 : Process with High Thread Count

sed -n '201,300p' file : display lines from 200 to 300

sed -i '/\<jvm\>/ s/$/sun/' file : add sun at the end of the line which has jvm 
find process that are being opened by a user and process ID :
lsof -u <User Name> -ap <PID> : use -a Flag for add
discard ping output :
ping -c 1 $host >/dev/null || { echo " The remote Host is unavailable" ; exit; }  
how many sockets are in each connection state:
netstat -t -n | cut -c 68- | sort | uniq -c | sort -n
      1
      1          State
      4          ESTABLISHED
Getting current TCP connection count on a system:

wc -l /proc/net/tcp

21 /proc/net/tcp

Kill Sockets
By Foreign address
netstat -anp | grep 192.168.0.100 | grep CLOSE_WAIT | awk '{print $7}' | cut -d \/ -f1 | grep -oE "[[:digit:]]{1,}" | xargs kill 
By Port
netstat -anp | grep ':80 ' | grep CLOSE_WAIT | awk '{print $7}' | cut -d \/ -f1 | grep -oE "[[:digit:]]{1,}" | xargs kill

By IP and Port

netstat -anp | grep 192.168.0.100 | grep ':80 ' | grep CLOSE_WAIT | awk '{print $7}' | cut -d \/ -f1 | grep -oE "[[:digit:]]{1,}" | xargs kill

Connect to Process using strace

strace -o pid18354_strace.log -p <PID>

Hit a URL and Get Response

wget --spider -nv -T 5 -t 1 http://host:10011/context

Tar and unzip at same time

tar jxf firefox-bin.tar.bz2

find Parent process ID

ps -l <PID> ( check for the ppid)

View Config Files with out Comments

grep ^[^#] somefile.conf

More to Come , Use these