Pages

Friday, January 17, 2014

One liners for Every Day Usage

sed -i '/^$/d' filename : Remove Duplicate Lines.
sed -i '/hello/d' filename: Remove the Line that has hello and modify the file.
cat filename | tr "," "\n" : Arrange comma (,) separated Strings one by one 
cat filename | tr -s '\n' : Remove the Blank Lines.
find . -type f -newermt '2010-01-01' : Compare to Date.
find . -type f -newermt '2010-01-01' ! -newermt '2010-06-01’: Between Dates.
awk -F':' '{printf "%-16s %-16s\n",$1,$2}' filename: Pretty printing.
sed -n 's:.*<id>\(.*\)</id>.*:\1:p' filename :Extract the content between the <id> elements.
sed -i '/^\(report\|-t\(h\|o\)\)/!d' filename :Remove all lines that dont start with string report.
awk -F"-" '{$NF = ""; print}' OFS="" filename : Remove the last character after ‘-‘ .
awk '{print FNR "\t" $0}' filename : Line Number Much like nl command.
awk '{print (NF? ++a ":" :"") $0}' filename : Line Number Much like nl command with no blank lines
awk 'END{print NR}' : Count Lines.
awk '/hello/{n++}; END {print n+0}' filename: Count lines with hello string.
awk '{ print NF ":" $0 } ' filename : Number of Fields in Each Line
awk '{ print $NF }'  : Print the last field of each line
awk '{ field = $NF }; END{ print field }' :  Print the last field of the last line
awk 'NF > 4' : print every line with more than 4 fields
awk '$NF > 4' : print every line where the value of the last field is > 4
awk '{sub(/^[ \t]+/, "")};1' filename : delete leading whitespace (spaces, tabs) from front of each line
awk '{$1=$1};1' filename
awk '{ print NR, $0 }' filename: add number to the lines in file
list -i RIM -e dev | awk '{print $1}' | awk -F"-" '{$NF = ""; print}' OFS="" : remove the first and last lines in output
cat filename | awk -F, '{print NF}' : When Comma is the Separator
cat /etc/hosts | awk '/localhost/ { print $1;}'
echo "test1:test2:test3" | awk 'BEGIN { FS = ":"} { print $2 }': with Field Seperator
awk '{print NR, $0}' test.txt : content of File with line number
awk '/PATTERN/{num++} END{ print num }' file.txt : count the number of lines that contain a pattern
nmap -p <Port No> <Remote Machine>: Hit a port on remote machine
egrep '(string1|string2|stringN)' file.txt : Print lines which contain string1 or string2
sed 's/.$//' or sed 's/^M$//' or sed 's/\x0D$//' : converts a dos file into unix mode.
sed "s/$/`echo -e \\\r`/" or sed 's/$/\r/' or sed "s/$//": converts a unix newline into a DOS newline.
sed 's/^[ \t]*//;s/[ \t]*$//' : Delete both leading and trailing white space and tab in a file.
awk '/CP-A/ { n++ }; END { print n+0 }' fileName : print the total number of lines containing the word pattern.
seq ‘/pattern1/,/pattern2/d’ < inputfile > outfile : will delete all the lines between pattern1 and pattern2.
sed ‘/20,30/d’ < inputfile > outfile : will delete all lines between 20 and 30.   OR sed ‘/20,30/d’ < input > output will delete lines between 20 and 30.
awk '/baz/ { gsub(/foo/, "bar") }; { print }' : Substitute foo with bar in lines that contains ‘baz’.
awk '!/baz/ { gsub(/foo/, "bar") }; { print }' : Substitute foo with bar in lines that does not contain ‘baz’.
awk '$5 == "abc123"' : print each line where the 5th field is equal to ‘abc123’.
awk '$5 != "abc123"' : print each line where 5th field is NOT equal to abc123. 
Compare to file:
find . -newer foo.txt
find . ! -newer foo.txt 
awk String comparison:
ll | awk '$9 == "two" {print $3"="$9}'
ll | awk '$7 >20 {print $9}' 
Find Jar files for class:
find . -name '*.jar' -print0 |  xargs -0 -I '{}' sh -c 'jar tf {} | grep hello.class &&  echo {}' 
find $PWD -type f -name "*Jul-*.ESS-A1.log" -exec rm -f {} \; : Find and Remove Files Matching a pattern 
find $PWD –type f –name “controllermessages.log.” | xargs tar zcvf one.tar : Find and Zip files matching a Pattern 
find . -name "*.jar" | xargs -tn1 jar tvf | grep --color "log4j.xml" : Search The File From Multiple Jar Files 
Count Threads OF a JVM Process : ps uH p | wc –l
Find how many files and application is using : lsof +c 0 | cut -d' ' -f1 | sort | uniq –c
Find and Kill a Process: Ps ux | grep | grep –v grep | awk ‘{print $2}’ | xargs –r kill -9
Kill a Process on the Port : kill -9 `lsof -t -i :port_number`
Find The Command Line Of the Process Using Specific Port : cat /proc/$(lsof -ti:631)/cmdline
Find out what is listening on a series of ports : /sbin/fuser -n tcp {7000..8000}
Get the 10 Biggest Files : du -sh * | sort -n | tail
Convert Of Bytes to Mbs : units --terse "3415014314 bytes" "MB"
SSh and Execute a Command: ssh root@<Remote Machine> -q 'echo $MYDIR’
Find out which process is using up your memory using ps, awk, sort : ps aux | awk '{if ($5 != 0 ) print $2,$5,$6,$11}' | sort -k2n
Find The Command Line Of the Process Using Specific Port : cat /proc/$(lsof -ti:631)/cmdline
Ping a Host Using Specific Interface ( Or can check Internet Connection for an Interface) : ping -I eth0 www.yahoo.com 
Scp To transfer a File:
Scp /root/file1 root@10.5.11.193:.
‘.’ Says to the root folder. You can use /tmp or any thing to copy to a specific location 
Highest CPU: ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10 | awk "{ print $2 }"
Memory Usage: sar -q 1 | tail -1 | awk '{ print "" $3}' | sed 's/%//g'
Finding the CPU Threshold : top -b -n 1 | awk -F'[:,]' '/^Cpu/{sub("\\..*","",$2); print $2}'
Top 10 Process : ps -efF "%x %p %P %U %u %y %a" | sort -r | head
List Process By memory usage :ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS
Who Started this process: ps -o comm= -p $(ps -o ppid= -p 28453)
How Much Ram Is Being Used : ps -o rss -C java | tail -n +2 | (sed 's/^/x+=/'; echo x) | bc
Find When a Process Was Started : ps -o lstart <PID>
Find the Class File in jars: find . -name "*.jar" | while read line; do unzip -l $line; done | grep <Name>
Using Find to Get Directories: find .  -path "*<DIR Name>" –print
Find Using Regex to find class and java files available :find . -regex ".*\(\.class\|\.java\)$"
Top 20 File handling process:
for x in `ps -eF| awk '{ print $2 }'`;do echo `ls /proc/$x/fd 2> /dev/null | wc -l` $x `cat /proc/$x/cmdline 2> /dev/null`;done | sort -n -r | head -n 20 
lsof -n -p <PID> | awk '$4 != "mem" {print}' : memory mapped File x
grep multiple Strings: grep 'ListenAddress\|port\|clusters\|server_cluster'
query by version and release : rpm -q --queryformat '\n%{NAME} %{VERSION} %{RELEASE} %{ARCH}\n' <package Name>
CPU / Memory Metrics : sar -urWqR 1
Multi CPU Metrics : sar -x SELF -I SUM -P ALL -wu 1
Disk Metrics : iostat -d -x 1 2
Network Metrics : netstat –s or sar -n DEV -n EDEV 1
Processes Metrics : ps -eo
Find and grep : find . -type f -exec grep -l java.lang.OutOfMemoryError {} \;
Who is using Port :
/sbin/fuser 10011/tcp
or
lsof -i tcp:10012
Hit a URL : curl -sL -w "%{http_code}\\n" "http://vx111a:10011/wls_monitor/" -o /dev/null
Show connections based on the host and the port using @host:port :
lsof -i@192.168.1.5:22
lsof -i@eth0.omhq111a.nova.com
Threads in a Process : ps -eLo pid,ppid,tid,pcpu,comm | grep PID
Differences between 2 files in remote hosts:
diff <(ssh alice cat /etc/apt/sources.list) <(ssh bob cat /etc/apt/sources.list)
Find all files Older than 2 days and gzip them:
find $PWD -type f -mtime +2 | xargs gzip
Find and Gzip:
find $PWD –type f –name “*.log” | xargs tar zcvf one.tar
How to list all unique ip address currently connected to a specific port:
ss -o state established '( dport = :10012 )'|awk -F"[\t :]+" 'NR!=1{ ip[$5]+=1 } END{ for (i in ip){n++};print n }'
All process Running as a Specific User:
pgrep -l -f -x -u root
List all the Jars loaded by a Process:
lsof -p PID | grep jar
Return Number of kernel Threads Owned by a process:
ps -o thcount –p PID
Show Ports that belong to this PID:
netstat --all --program | grep PID
Drop all the Connections available now for a Port:
iptables -I INPUT -p tcp -dport 80 -j DROP
SSh and Execute a Command:
ssh root@ -q 'echo $MYDIR’
Find out what is listening on a series of ports:
/sbin/fuser -n tcp {7000..8000}
View details of network activity between:
lsof -i :7000-8000
List all files opened by a particular command:
lsof -c java
List Threads By PID along with Thread Start Time:
ps -o lwp,lstart --pid PID
Extract files from war/Ear:
jar xf abc.war log4j-test.xml WEB-INF/
All Connections from a Specific process:
lsof -p PID -a –i
show the number of connections active to a port and also the number of connections from that ip in order:
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort –n
determine which application is utilizing a certain port? :
lsof -w -n -i tcp:80 ( or any Othe Port)
Owner of the File:
/sbin/fuser admin.lok
List of Files That Are Open For Writing to Disk:
lsof | grep -e "[[:digit:]]\+w"
See which Process is Holding the File:
lsof -r1 /common/jboss.log
List the files accessed by a program:
strace -f -o foo.trace su user -c 'mycommand'?
Environment Variables belong to a Process:
ps ewwo command PID | tr ' ' '\n' | grep \=
List Threads by Pid along with Thread Start Time:
ps -o pid,lwp,lstart --pid PID -L
CPU usage for EACH cores:
ps ax -L -o pid,tid,psr,pcpu,args | sort -nr -k4| head -15 | cut -c 1-90
Largest File Or Directory:du -sk /var/log/* | sort -r -n | head -10
sed -e '1d;n;d' file : Remove every Second Line
sed '$d' : Remove the last Line  
Remove Even Number lines (2,4..) : sed -n '1~2p' file
Remove Odd Number Lines (1,3..) : sed -n '2~2p' file
Remove lines which contain more than 3 char in the second column : awk -F, 'length($2)<=3'
Delete line containing specific number of char (ex:5) : awk 'length<=5' file
List all lines which contain more than 1 Capitalized Character : grep -o '\b[[:alpha:]]*[[:upper:]][[:alpha:]]*[[:upper:]][[:alpha:]]*' file
Remove Lines which start with Capital Words : sed -r 's/\b[A-Z](\w*)\b//g' file
Print Words that Start with Vowels :  sed -r 's/\b[^AEIOUaeiou]\w*//g'  file
Files not being Accessed : find  -mtime +7 -type f   -exec sh -c '/sbin/fuser {} >/dev/null 2>&1 || echo {}' ';'
Run a Command and Display yes or No : status=`jinfo -sysprops $id 2>&1 | grep nodeName 2>&1 >/dev/null  && echo "yes" || echo "no"`
Clean the PID returned by Pgrep Command : id=`echo "$Pid" | grep -v  "^$$"`
/usr/bin/ssh -o "StrictHostKeyChecking no" -n  userName@Virtuals 'cat | bash /dev/stdin' $hello < ./appVal.sh : Run a Script on the Remove Machine by passing an argument $hello
tail Multiple Log Files same Time : tail -f /var/log/syslog -f /var/log/auth.log
Remove "" from first and last in Variable : echo $VAR | sed "s/\"//g"
Env to Sed : env | sed -n /"$HOSTNAME"/p
Filter in Vi : 1,$!awk '{print $1}' | sort | tr [:lower:] [:upper:]
Save from 6 - 9 to a File : 6,9w >> /tmp/newfile
Comment from Line 5-10: 5,10s/^/#/g
Comment all Lines : %s/^/#/g
External Connections made : lsof -n -i  | grep ESTABLISHED
External Connections : lsof -iUDP -P -n | egrep -v '(127|::1)'
10Mb of File : dd if=/dev/zero of=C:/temp/testfiles/10MB.testfile bs=1024 count=12400
Sub String with awk : echo "12345678" | awk '{print substr($0, 3, 2);}'
Which User is using the Port : /sbin/fuser -nu tcp 10001
list and kill any processes currently using /mount : fuser -vmk /mount
How do I monitor opened files of a process in realtime?: strace -e trace=open,close,read,write,connect,accept start.sh AS-A2
All Ports : lsof -iUDP -P -n | egrep -v '(127|::1)'
Command Line arg of the Process : tr '\0' '\n' </proc/1234/cmdline
cut -d: -f1,2,7 passwd : print 1,2and 7 Fields
cut -d: -f5-8 passwd  : print Fields between 5 and 8
cut --output-delimiter=, -d: -f1- passwd : Output with a Delimited “,”
cut -c1-25 /etc/motd : print Char 1-25
cut --complement -c10-25 :print every thing leaving 10-25
Break Into Commas: command | paste –sd,
Case-Insensitive Find File Search :find . -print –iname “pattern”
Find and Unzip : find $PWD -name 'one.zip' -print -exec unzip -l {} \;
Exclude svn data : zip -9 -r --exclude=*.svn*  foo.zip [directory-to-compress]
Hit a URL : wget --inet4-only http://vx112:11011/hell.jsp
Remove the first 2 char from  a file : sed 's/^.\{,2\}//' mental1
Push Data to a port on IP address : nc 67.206.9.118 59312 <<< "<dasj>Sending Data By Jagadish."
sort -k  1 -s file : Sort 1 column in a  file with out touching the second column
rpm -q --all –last : Show the rpm installed along with dates
awk '{ print $14,$15 }' /proc/P12363/stat : percentages of CPU utime and stime of each process?
Last reboot : display last reboot time
sed 's/.$//' file : remove last char in the line of a File
Linux Directory Sizes : find . -type d | grep / | du –h
ps auwwx|gawk '!/%MEM/ {print $4,$11}'|sort -nr|head -n20 : Top 20 process by Memory
ps auwwx|gawk '{count[$NF]++}END{for(j in count) print ""count[j]":",j}'|sort -nr|head -n20 : Process with High Thread Count
sed -n '201,300p' file : display lines from 200 to 300
sed -i '/\<jvm\>/ s/$/sun/' file : add sun at the end of the line which has jvm 
find process that are being opened by a user and process ID :
lsof -u <User Name> -ap <PID> : use -a Flag for add
discard ping output :
ping -c 1 $host >/dev/null || { echo " The remote Host is unavailable" ; exit; }  
how many sockets are in each connection state:
netstat -t -n | cut -c 68- | sort | uniq -c | sort -n
      1
      1          State
      4          ESTABLISHED
Getting current TCP connection count on a system:
wc -l /proc/net/tcp
21 /proc/net/tcp

Kill Sockets
By Foreign address
netstat -anp | grep 192.168.0.100 | grep CLOSE_WAIT | awk '{print $7}' | cut -d \/ -f1 | grep -oE "[[:digit:]]{1,}" | xargs kill 
By Port
netstat -anp | grep ':80 ' | grep CLOSE_WAIT | awk '{print $7}' | cut -d \/ -f1 | grep -oE "[[:digit:]]{1,}" | xargs kill
By IP and Port
netstat -anp | grep 192.168.0.100 | grep ':80 ' | grep CLOSE_WAIT | awk '{print $7}' | cut -d \/ -f1 | grep -oE "[[:digit:]]{1,}" | xargs kill
Connect to Process using strace
strace -o pid18354_strace.log -p <PID>
Hit a URL and Get Response
wget --spider -nv -T 5 -t 1 http://host:10011/context
Tar and unzip at same time
tar jxf firefox-bin.tar.bz2
find Parent process ID
ps -l <PID> ( check for the ppid)
View Config Files with out Comments
grep ^[^#] somefile.conf

More to Come , Use these