Contents

Filter unique values from third column, separated by space
Filter unique values from third column, separated by :
Search for text in files recursively
Copy files to other destination (Win32)
Gzip files that where last modified in 30 or more days (Win32)
Run task every day at 01:00 (Win32)
Block responses to pings
Zip SVN modified Java files
Deploy EAR script
1. List JAR contents to file
Size of each directory inside current directory
Backup folders in HOME
Get installed updates on Windows
Find files modified less than an hour ago
Lock file to allow only one instance of a script to run
awk , load file into array
Shell script sample
Change TIME_WAIT (time wait) tcp/ip
Show root folders occupied space
Add swapfile 512MB
Add swapfile 4GB
Alias current date
1. Alias history without line number
Calculator using bc
Get slackbuild
Prompt PS1 without current folder
Prompt PS1 with folder
PowerShell commands
Search using GNU grep
Grep to ignore lines
Show full user name with ps command
watch command
screen command
calculate_space.py
Bell warning sound console
Archive files modified in the last 15 days

Filter unique values from third column, separated by space

   1 cat datax.log |  awk -F ' ' '/IMEI/{print $3}' | sort | uniq | wc -l

Filter unique values from third column, separated by :

   1 cat datax.log |  awk -F ':' '/IMEI/{print $3}' | sort | uniq | wc -l

Search for text in files recursively

   1 cd startFolder
   2 grep -H "textToSearch" * -R

Copy files to other destination (Win32)

@echo off
REM Mount drive
echo Mounting remote drive
net use z: \\192.168.1.1\backs password /USER:userx

REM copy current day files
echo Copying current day files
forfiles -p "I:" -s -m *FULL*.sqb -d +0 -c "cmd /c copy @path Z:\BACKUPS\@file"

echo Deleting files older than 7 days
REM delete files older than 7 days
forfiles -p "Z:\BACKUPS" -m *.sqb -d 7 -c "cmd /c del @path"

Gzip files that where last modified in 30 or more days (Win32)

forfiles /p "c:\logs" /d -30 /m *.log /c "cmd /c gzip @path "

Run task every day at 01:00 (Win32)

at 01:00 /every:M,T,W,Th,F,S,Su c:\task.bat

Block responses to pings

   1 #block responses to pings
   2 iptables -A INPUT -p icmp -m icmp --icmp-type echo-request -j DROP
   3 iptables -L
   4 #restore responses to pings
   5 iptables -F
   6 iptables -L

Zip SVN modified Java files

   1 #!/bin/sh
   2 # chmod 755 /usr/local/bin/modifiedJavaSvn.sh # on CygWin
   3 svn status | awk '/.java/{ print( gensub( /\\/,"/","g",$2) ); }' | xargs -i zip modifiedJavaFiles.zip {}
   4 unzip -v modifiedJavaFiles.zip

Deploy EAR script

   1 #!/usr/bin/bash
   2 # deploy EAR in remote JBoss . 
   3 # Must have pub SSH keys (authorized_keys) deployed on remote server
   4 # Can be used with cygwin
   5 FILEX=sample.ear
   6 DEST_DIR=/opt/jboss/server/default/deploy/
   7 ECLIPSE_PROJECT=/home/userx/workspace/sample-ear/
   8 
   9 deploy()
  10 {
  11   SERVER=$1
  12   USER=$2
  13   echo "Deploying in $SERVER ..."
  14 
  15   cd $PROJECT
  16   scp target/$FILEX $USER@$SERVER:/tmp
  17   ssh $USER@$SERVER -C "mv /tmp/$FILEX $DEST_DIR/$FILEX"
  18   ssh $USER@$SERVER -C "ls -lah $DEST_DIR"
  19   ssh $USER@$SERVER -C "chmod 666  $DEST_DIR/$FILEX"
  20   ssh $USER@$SERVER -C "/sbin/service jboss abort"
  21   ssh $USER@$SERVER -C "sudo /sbin/service jboss start"
  22 }
  23 
  24 deploy "example-server" "userx"

List JAR contents to file

cd ~/.m2/repository
find . -name "*.jar" | xargs -i jar -tf {} > /tmp/explodedJars.txt
find . -name "*.jar" | xargs -i sh -c "echo {} ; jar -tf {}" > /tmp/explodedJars.txt

Size of each directory inside current directory

find . -maxdepth 1 -type d | xargs -i du {} -hs

Backup folders in HOME

   1 #!/bin/bash
   2 # cd ~
   3 # ln -s scripts/backup.sh backup.sh
   4 # sh backup.sh
   5 DATE=`date '+%Y%m%d%H%M%S'`
   6 FILENAME=~/backups/backup_$DATE.tar.gz
   7 cd ~
   8 mkdir -p ~/backups
   9 rm $FILENAME
  10 tar cvzf $FILENAME Documents GNUstep ThunderbirdMail Desktop .Skype scripts moin-1.9.7
  11 # tar tvzf backup_20150108103242.tar.gz
  12

Get installed updates on Windows

wmic qfe list brief /format:texttablewsys > updates.txt

Find files modified less than an hour ago

find . -mtime -1h -name "*.cfg"

Lock file to allow only one instance of a script to run

   1 #!/bin/sh
   2 LOCK_FILE=$(echo "/var/tmp/lockTest.lck")
   3 PID=$$
   4 LOCKED=0
   5 COUNT=0
   6 echo "Trying to get lock to $PID in $LOCK_FILE"
   7 
   8 while [ $LOCKED -eq "0" ] && [ $COUNT -le "3" ]; do
   9     if ( set -o noclobber; echo "$PID" > "$LOCK_FILE" ) 2> /dev/null; then
  10         LOCKED=1
  11     else        
  12         sleep 1
  13         COUNT=$(($COUNT+1))
  14         echo "Trying to get lock to $PID in $LOCK_FILE, retry $COUNT"
  15     fi    
  16 done
  17 
  18 if [ $LOCKED -eq "0" ]; then
  19     echo "Unable to get lock to $PID with $COUNT retries"
  20     exit 1
  21 else
  22     echo "Lock acquired to $PID"    
  23 fi   
  24 #####################################
  25 sleep 10
  26 #####################################
  27 rm -f "$LOCK_FILE"
  28 LOCKED=0
  29 echo "Released lock to $PID"
  30 exit 0

awk , load file into array

https://magvar.wordpress.com/2011/05/18/awking-it-how-to-load-a-file-into-an-array-in-awk/

   1 #! /bin/bash
   2 INPUTFILE="inputfile.csv"
   3 DATAFILE="data.csv"
   4 OUTFILE="output.csv"
   5 
   6 awk 'BEGIN {
   7 while (getline < "'"$INPUTFILE"'")
   8 {
   9 split($0,ft,",");
  10 id=ft[1];
  11 name=ft[2];
  12 
  13 key=id;
  14 data=name;
  15 nameArr[key]=data;
  16 }
  17 close("'"$INPUTFILE"'");
  18 
  19 while (getline < "'"$DATAFILE"'")
  20 {
  21 split($0,ft,",");
  22 id=ft[1]; # Id is the first column
  23 phone=ft[2]; # Phonenumber is the second
  24 name=nameArr[id]; # Get the name from the nameArr, using "id" as key
  25 print id","name","phone > "'"$OUTFILE"'"; # Print directly to the outputfile
  26 }
  27 }'

Shell script sample

   1 #!/bin/sh
   2 function replaceString {
   3   RES=$(echo $1 | sed "s/$2/$3/g")
   4   echo $RES
   5 }
   6 
   7 function add {
   8   echo $( echo "$1 + $2" | bc )    
   9 }
  10 
  11 function inc {
  12   echo $( expr $1 + 1 )  
  13 }
  14 ####################################
  15 echo $(add "1.5" "3.2" )
  16 
  17 TEXT1="AAA BBB CCC"
  18 
  19 for WORD in $TEXT1
  20 do
  21   echo ${WORD}
  22 done
  23 
  24 WITHOUTSPACES=$(replaceString "AAA BBB XXX" " " "" )
  25 echo $WITHOUTSPACES
  26 
  27 if [ $WITHOUTSPACES == "AAABBBXXX" ]
  28 then
  29   echo "Equal"
  30 fi
  31 
  32 if [ $WITHOUTSPACES != "AAA BBB XXX" ]
  33 then
  34   echo "Diff"
  35 fi
  36 
  37 A=$(ls / | wc -l)
  38 A=$(expr $A + 1)
  39 if [ $A -eq 22 ]
  40 then
  41   echo $A
  42 fi
  43 
  44 B="222"
  45 B=$(expr $B)
  46 if [ $B -eq 222 ]
  47 then
  48   echo $B
  49 fi
  50 
  51 COUNT="0"
  52 while [ $COUNT -lt 10 ]
  53 do
  54   COUNT=$( inc $COUNT )
  55 done
  56 
  57 $(test 0 -ge 0)
  58 RES=$(echo $?)
  59 echo $RES
  60 
  61 echo "Input data"
  62 read IN1
  63 echo $IN1
  64 
  65 case $IN1 in
  66   hi)
  67     echo "yeahh"
  68     ;;
  69   *)
  70     echo "ups"
  71     ;;
  72 esac

Change TIME_WAIT (time wait) tcp/ip

Linux

   1 echo 1 > /proc/sys/net/ipv4/tcp_fin_timeout
   2 echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
   3 echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

Freebsd

sysctl net.inet.tcp.msl=1500

Show root folders occupied space

   1 cd /tmp
   2 ls --color=never | xargs -i du {} -hs
   3 find . -type d -maxdepth 1 | xargs -i du {} -hs
   4 
   5 #~/.bashrc
   6 alias folderUsedSpace='find . -type d -maxdepth 1 | xargs -i du {} -hs'

Add swapfile 512MB

   1 df -h
   2 dd if=/dev/zero of=/swapfile1 bs=1024 count=524288
   3 chown root:root /swapfile1 
   4 chmod 0600 /swapfile1 
   5 mkswap /swapfile1 
   6 swapon /swapfile1
   7 free -m
   8 vi /etc/fstab
   9 # /swapfile1 none swap sw 0 0
  10 free -m
  11 htop

Add swapfile 4GB

   1 # create_swapfile.sh 
   2 #!/bin/sh
   3 # 4GB swapfile
   4 # bs 1024 bytes 
   5 # calc 2^30*4/1024
   6 # 4194304.000
   7 dd if=/dev/zero of=/swapfile1 bs=1024 count=4194304
   8 chown root:root /swapfile1
   9 chmod 0600 /swapfile1
  10 /sbin/mkswap /swapfile1
  11 /sbin/swapon /swapfile1
  12 free -m
  13 vi /etc/fstab
  14 # /swapfile1 none swap sw 0 0
  15 free -m
  16 htop

Alias current date

   1 alias currdate="date '+%Y%m%dT%H%M%S'"

Alias history without line number

   1 alias hist='history | cut -c 8-'

Calculator using bc

   1 #!/bin/sh
   2 echo "scale=3;$1" | bc

Get slackbuild

getslackbuild.sh

   1 #!/bin/sh
   2 PACKAGENAME=$1
   3 PAGE=/tmp/page.html
   4 HEADERS=/tmp/headers.html
   5 RES=/tmp/slackbuildres.txt
   6 curl https://slackbuilds.org/result/?search=$PACKAGENAME -D $HEADERS > $PAGE 2>/dev/null
   7 LOCATION=$(cat $HEADERS | grep -i Location | sed  's/Location: //g' )
   8 echo "Location: $LOCATION"
   9 POS=$(echo $LOCATION | grep -b -o "?search" | cut -d: -f1 )
  10 LOCATION_FIXED=$( echo $LOCATION | cut -c1-"$POS" | sed  "s/$PACKAGENAME\///g"   )
  11 INFO_FIXED=$( echo $LOCATION | cut -c1-"$POS"  )
  12 PACKAGE=$(echo $LOCATION_FIXED$1.tar.gz | sed 's/http:/https:/g' | sed  's/repository\//slackbuilds\//g')
  13 INFO=$(echo $INFO_FIXED$1.info | sed 's/http:/https:/g'  | sed  's/repository\//slackbuilds\//g'  )
  14 echo "Package: $PACKAGE"
  15 echo "Info: $INFO"
  16 cd /tmp
  17 rm -rf $PACKAGENAME/
  18 rm -f $PACKAGENAME.tar.gz*
  19 echo "Getting package"
  20 wget $PACKAGE
  21 tar xvzf $PACKAGENAME.tar.gz
  22 cd $PACKAGENAME
  23 echo "Getting info"
  24 wget $INFO
  25 DOWNLOAD=$(cat $PACKAGENAME.info | grep "DOWNLOAD=" | sed 's/"//g' | sed 's/DOWNLOAD=//g' )
  26 echo "Download source code"
  27 wget $DOWNLOAD
  28 echo "Run slackbuild"
  29 ./$PACKAGENAME.SlackBuild > $RES 2>&1
  30 SBO=$( cat $RES | grep "Slackware package " | grep " created" | sed 's/Slackware package //g' | sed 's/ created\.//g'  )
  31 echo "SlackBuild package: $SBO"

Prompt PS1 without current folder

   1 # prompt without current folder
   2 #PS1="$(date +'%Y-%m-%dT%H:%M:%S') $(whoami)@${HOST}\$ "
   3 #PS1="$(date +'%Y-%m-%dT%H:%M:%S') $(whoami)@$(hostname -a)\$ "
   4 
   5 PS1="\D{%Y-%m-%dT%H:%M:%S} [\u@\h:\l]\n$"

Prompt PS1 with folder

   1 # PS1="$(date +'%Y-%m-%dT%H:%M:%S') $(whoami)@$(hostname -a)\n${PWD}\n\$ "
   2 PS1="\D{%Y-%m-%d}T\A \u@\H\n\w\r\n\$ "
   3 $PS1="$(date +'%Y-%m-%dT%H:%M:%S') [\u@\h:\l \w]\\n\\$"
   4 PS1="\D{%Y-%m-%dT%H:%M:%S} [\u@\h:\l \w]\n$"

PowerShell commands

Get-ChildItem c:\windows\system32\*.txt -Recurse | Select-String -Pattern "Microsoft" -CaseSensitive
sls "searchstring" file

Search using GNU grep

   1 #!/bin/sh
   2 echo "Searching $1 in $(pwd)"
   3 grep -H -n "$1" * -R

Grep to ignore lines

   1 grep -v "^#" file # grep file ignoring lines starting with #
   2

Show full user name with ps command

To see every process with a user-defined format:

ps axo user:32,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,comm
o format User-defined format.

watch command

WATCH(1)                                                                                      User Commands                                                                                      WATCH(1)

NAME
       watch - execute a program periodically, showing output fullscreen

By default, the program is run every 2 seconds.
watch -n 5 sh command.sh # executes the script command.sh each 5 seconds and show its output on the screen

screen command

https://www.thegeekstuff.com/2010/07/screen-command-examples/

Screen multiplexer. The screen command is most used for ssh session because it helps to continue your work after a disconnection without losing the current processes in progress.

# start bash in screen
screen bash
# detach from bash ( it will keep running)
# ctrl+a d
# check running processes with screen
screen -ls
# attach to process
screen -r <id in screen -ls>

calculate_space.py

calculate_space.py

   1 #!/usr/bin/python
   2 import os
   3 
   4 filename='du_out.txt'
   5 os.system('du . -h > %s'%(filename))
   6 
   7 f = open('%s'%(filename))
   8 lines=[]
   9 
  10 for item in f:
  11     lines.append(item)
  12 line_g = [ l for l in lines if "G\t." in l]
  13 
  14 for l in line_g:
  15     print l.replace("\n","")
  16 os.system('rm %s'%(filename))

Bell warning sound console

~/scripts/bell,sh
chmod 755 ~/scripts/bell,sh

   1 #!/bin/sh
   2 TEXT1="1 2 3 4"
   3 
   4 for WORD in $TEXT1
   5 do
   6   aplay /usr/share/sounds/sound-icons/prompt.wav > /dev/null  2>&1
   7 done

Archive files modified in the last 15 days

   1 find . -type f -mtime -15 -print0 | tar -czvf /tmp/backup.tar.gz --null -T -
   2 tar tvzf /tmp/backup.tar.gz