Differences between revisions 47 and 74 (spanning 27 versions)

Contents

Filter unique values from third column, separated by space
Filter unique values from third column, separated by :
Search for text in files recursively
Copy files to other destination (Win32)
Gzip files that where last modified in 30 or more days (Win32)
Run task every day at 01:00 (Win32)
Block responses to pings
Zip SVN modified Java files
Deploy EAR script
List JAR contents to file
Size of each directory inside current directory
Backup folders in HOME
Get installed updates on Windows
Find files modified less than an hour ago
Lock file to allow only one instance of a script to run
awk , load file into array
Shell script sample
Change TIME_WAIT (time wait) tcp/ip
Show root folders occupied space
Add swapfile 512MB
Add swapfile 4GB
Alias current date
Alias history without line number
Calculator using bc
Get slackbuild
Prompt PS1 without current folder
Prompt PS1 with folder
PowerShell commands
Search using GNU grep
Grep to ignore lines
Show full user name with ps command
watch command
screen command
calculate_space.py
Bell warning sound console
Archive files modified in the last 15 days
findtext
findresource
Format Pen drive NTFS

Filter unique values from third column, separated by space

   1 cat datax.log |  awk -F ' ' '/IMEI/{print $3}' | sort | uniq | wc -l

Filter unique values from third column, separated by :

   1 cat datax.log |  awk -F ':' '/IMEI/{print $3}' | sort | uniq | wc -l

Search for text in files recursively

   1 cd startFolder
   2 grep -H "textToSearch" * -R

Copy files to other destination (Win32)

@echo off
REM Mount drive
echo Mounting remote drive
net use z: \\192.168.1.1\backs password /USER:userx

REM copy current day files
echo Copying current day files
forfiles -p "I:" -s -m *FULL*.sqb -d +0 -c "cmd /c copy @path Z:\BACKUPS\@file"

echo Deleting files older than 7 days
REM delete files older than 7 days
forfiles -p "Z:\BACKUPS" -m *.sqb -d 7 -c "cmd /c del @path"

Gzip files that where last modified in 30 or more days (Win32)

forfiles /p "c:\logs" /d -30 /m *.log /c "cmd /c gzip @path "

Run task every day at 01:00 (Win32)

at 01:00 /every:M,T,W,Th,F,S,Su c:\task.bat

Block responses to pings

   1 #block responses to pings
   2 iptables -A INPUT -p icmp -m icmp --icmp-type echo-request -j DROP
   3 iptables -L
   4 #restore responses to pings
   5 iptables -F
   6 iptables -L

Zip SVN modified Java files

   1 #!/bin/sh
   2 # chmod 755 /usr/local/bin/modifiedJavaSvn.sh # on CygWin
   3 svn status | awk '/.java/{ print( gensub( /\\/,"/","g",$2) ); }' | xargs -i zip modifiedJavaFiles.zip {}
   4 unzip -v modifiedJavaFiles.zip

Deploy EAR script

   1 #!/usr/bin/bash
   2 # deploy EAR in remote JBoss . 
   3 # Must have pub SSH keys (authorized_keys) deployed on remote server
   4 # Can be used with cygwin
   5 FILEX=sample.ear
   6 DEST_DIR=/opt/jboss/server/default/deploy/
   7 ECLIPSE_PROJECT=/home/userx/workspace/sample-ear/
   8 
   9 deploy()
  10 {
  11   SERVER=$1
  12   USER=$2
  13   echo "Deploying in $SERVER ..."
  14 
  15   cd $PROJECT
  16   scp target/$FILEX $USER@$SERVER:/tmp
  17   ssh $USER@$SERVER -C "mv /tmp/$FILEX $DEST_DIR/$FILEX"
  18   ssh $USER@$SERVER -C "ls -lah $DEST_DIR"
  19   ssh $USER@$SERVER -C "chmod 666  $DEST_DIR/$FILEX"
  20   ssh $USER@$SERVER -C "/sbin/service jboss abort"
  21   ssh $USER@$SERVER -C "sudo /sbin/service jboss start"
  22 }
  23 
  24 deploy "example-server" "userx"

List JAR contents to file

   1 cd ~/.m2/repository
   2 find . -name "*.jar" | xargs -i jar -tf {} > /tmp/explodedJars.txt
   3 find . -name "*.jar" | xargs -i sh -c "echo {} ; jar -tf {}" > /tmp/explodedJars.txt

Size of each directory inside current directory

find . -maxdepth 1 -type d | xargs -i du {} -hs

Backup folders in HOME

   1 #!/bin/bash
   2 # cd ~
   3 # ln -s scripts/backup.sh backup.sh
   4 # sh backup.sh
   5 DATE=`date '+%Y%m%d%H%M%S'`
   6 FILENAME=~/backups/backup_$DATE.tar.gz
   7 cd ~
   8 mkdir -p ~/backups
   9 rm $FILENAME
  10 tar cvzf $FILENAME Documents GNUstep ThunderbirdMail Desktop .Skype scripts moin-1.9.7
  11 # tar tvzf backup_20150108103242.tar.gz
  12

Get installed updates on Windows

wmic qfe list brief /format:texttablewsys > updates.txt

Find files modified less than an hour ago

find . -mtime -1h -name "*.cfg"

Lock file to allow only one instance of a script to run

   1 #!/bin/sh
   2 LOCK_FILE=$(echo "/var/tmp/lockTest.lck")
   3 PID=$$
   4 LOCKED=0
   5 COUNT=0
   6 echo "Trying to get lock to $PID in $LOCK_FILE"
   7 
   8 while [ $LOCKED -eq "0" ] && [ $COUNT -le "3" ]; do
   9     if ( set -o noclobber; echo "$PID" > "$LOCK_FILE" ) 2> /dev/null; then
  10         LOCKED=1
  11     else        
  12         sleep 1
  13         COUNT=$(($COUNT+1))
  14         echo "Trying to get lock to $PID in $LOCK_FILE, retry $COUNT"
  15     fi    
  16 done
  17 
  18 if [ $LOCKED -eq "0" ]; then
  19     echo "Unable to get lock to $PID with $COUNT retries"
  20     exit 1
  21 else
  22     echo "Lock acquired to $PID"    
  23 fi   
  24 #####################################
  25 sleep 10
  26 #####################################
  27 rm -f "$LOCK_FILE"
  28 LOCKED=0
  29 echo "Released lock to $PID"
  30 exit 0

awk , load file into array

https://magvar.wordpress.com/2011/05/18/awking-it-how-to-load-a-file-into-an-array-in-awk/

   1 #! /bin/bash
   2 INPUTFILE="inputfile.csv"
   3 DATAFILE="data.csv"
   4 OUTFILE="output.csv"
   5 
   6 awk 'BEGIN {
   7 while (getline < "'"$INPUTFILE"'")
   8 {
   9 split($0,ft,",");
  10 id=ft[1];
  11 name=ft[2];
  12 
  13 key=id;
  14 data=name;
  15 nameArr[key]=data;
  16 }
  17 close("'"$INPUTFILE"'");
  18 
  19 while (getline < "'"$DATAFILE"'")
  20 {
  21 split($0,ft,",");
  22 id=ft[1]; # Id is the first column
  23 phone=ft[2]; # Phonenumber is the second
  24 name=nameArr[id]; # Get the name from the nameArr, using "id" as key
  25 print id","name","phone > "'"$OUTFILE"'"; # Print directly to the outputfile
  26 }
  27 }'

Shell script sample

   1 #!/bin/sh
   2 function replaceString {
   3   RES=$(echo $1 | sed "s/$2/$3/g")
   4   echo $RES
   5 }
   6 
   7 function add {
   8   echo $( echo "$1 + $2" | bc )    
   9 }
  10 
  11 function inc {
  12   echo $( expr $1 + 1 )  
  13 }
  14 ####################################
  15 echo $(add "1.5" "3.2" )
  16 
  17 TEXT1="AAA BBB CCC"
  18 
  19 for WORD in $TEXT1
  20 do
  21   echo ${WORD}
  22 done
  23 
  24 WITHOUTSPACES=$(replaceString "AAA BBB XXX" " " "" )
  25 echo $WITHOUTSPACES
  26 
  27 if [ $WITHOUTSPACES == "AAABBBXXX" ]
  28 then
  29   echo "Equal"
  30 fi
  31 
  32 if [ $WITHOUTSPACES != "AAA BBB XXX" ]
  33 then
  34   echo "Diff"
  35 fi
  36 
  37 A=$(ls / | wc -l)
  38 A=$(expr $A + 1)
  39 if [ $A -eq 22 ]
  40 then
  41   echo $A
  42 fi
  43 
  44 B="222"
  45 B=$(expr $B)
  46 if [ $B -eq 222 ]
  47 then
  48   echo $B
  49 fi
  50 
  51 COUNT="0"
  52 while [ $COUNT -lt 10 ]
  53 do
  54   COUNT=$( inc $COUNT )
  55 done
  56 
  57 $(test 0 -ge 0)
  58 RES=$(echo $?)
  59 echo $RES
  60 
  61 echo "Input data"
  62 read IN1
  63 echo $IN1
  64 
  65 case $IN1 in
  66   hi)
  67     echo "yeahh"
  68     ;;
  69   *)
  70     echo "ups"
  71     ;;
  72 esac

Change TIME_WAIT (time wait) tcp/ip

Linux

   1 echo 1 > /proc/sys/net/ipv4/tcp_fin_timeout
   2 echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
   3 echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

Freebsd

sysctl net.inet.tcp.msl=1500

Show root folders occupied space

   1 cd /tmp
   2 ls --color=never | xargs -i du {} -hs
   3 find . -type d -maxdepth 1 | xargs -i du {} -hs
   4 
   5 #~/.bashrc
   6 alias folderUsedSpace='find . -type d -maxdepth 1 | xargs -i du {} -hs'

Add swapfile 512MB

   1 df -h
   2 dd if=/dev/zero of=/swapfile1 bs=1024 count=524288
   3 chown root:root /swapfile1 
   4 chmod 0600 /swapfile1 
   5 mkswap /swapfile1 
   6 swapon /swapfile1
   7 free -m
   8 vi /etc/fstab
   9 # /swapfile1 none swap sw 0 0
  10 free -m
  11 htop

Add swapfile 4GB

   1 # create_swapfile.sh 
   2 #!/bin/sh
   3 # 4GB swapfile
   4 # bs 1024 bytes 
   5 # calc 2^30*4/1024
   6 # 4194304.000
   7 dd if=/dev/zero of=/swapfile1 bs=1024 count=4194304
   8 chown root:root /swapfile1
   9 chmod 0600 /swapfile1
  10 /sbin/mkswap /swapfile1
  11 /sbin/swapon /swapfile1
  12 free -m
  13 vi /etc/fstab
  14 # /swapfile1 none swap sw 0 0
  15 free -m
  16 htop

Alias current date

   1 alias currdate="date '+%Y%m%dT%H%M%S'"

Alias history without line number

   1 alias hist='history | cut -c 8-'

Calculator using bc

   1 cd ~/scripts/
   2 nano calc
   3 chmod 755 calc

   1 #!/bin/sh
   2 echo "scale=3;$1" | bc

Get slackbuild

getslackbuild.sh

   1 #!/bin/sh
   2 PACKAGENAME=$1
   3 PAGE=/tmp/page.html
   4 HEADERS=/tmp/headers.html
   5 RES=/tmp/slackbuildres.txt
   6 curl https://slackbuilds.org/result/?search=$PACKAGENAME -D $HEADERS > $PAGE 2>/dev/null
   7 LOCATION=$(cat $HEADERS | grep -i Location | sed  's/Location: //g' )
   8 echo "Location: $LOCATION"
   9 POS=$(echo $LOCATION | grep -b -o "?search" | cut -d: -f1 )
  10 LOCATION_FIXED=$( echo $LOCATION | cut -c1-"$POS" | sed  "s/$PACKAGENAME\///g"   )
  11 INFO_FIXED=$( echo $LOCATION | cut -c1-"$POS"  )
  12 PACKAGE=$(echo $LOCATION_FIXED$1.tar.gz | sed 's/http:/https:/g' | sed  's/repository\//slackbuilds\//g')
  13 INFO=$(echo $INFO_FIXED$1.info | sed 's/http:/https:/g'  | sed  's/repository\//slackbuilds\//g'  )
  14 echo "Package: $PACKAGE"
  15 echo "Info: $INFO"
  16 cd /tmp
  17 rm -rf $PACKAGENAME/
  18 rm -f $PACKAGENAME.tar.gz*
  19 echo "Getting package"
  20 wget $PACKAGE
  21 tar xvzf $PACKAGENAME.tar.gz
  22 cd $PACKAGENAME
  23 echo "Getting info"
  24 wget $INFO
  25 DOWNLOAD=$(cat $PACKAGENAME.info | grep "DOWNLOAD=" | sed 's/"//g' | sed 's/DOWNLOAD=//g' )
  26 echo "Download source code"
  27 wget $DOWNLOAD
  28 echo "Run slackbuild"
  29 ./$PACKAGENAME.SlackBuild > $RES 2>&1
  30 SBO=$( cat $RES | grep "Slackware package " | grep " created" | sed 's/Slackware package //g' | sed 's/ created\.//g'  )
  31 echo "SlackBuild package: $SBO"

Prompt PS1 without current folder

   1 # prompt without current folder
   2 #PS1="$(date +'%Y-%m-%dT%H:%M:%S') $(whoami)@${HOST}\$ "
   3 #PS1="$(date +'%Y-%m-%dT%H:%M:%S') $(whoami)@$(hostname -a)\$ "
   4 
   5 PS1="\D{%Y-%m-%dT%H:%M:%S} [\u@\h:\l]\n$"

Prompt PS1 with folder

   1 # PS1="$(date +'%Y-%m-%dT%H:%M:%S') $(whoami)@$(hostname -a)\n${PWD}\n\$ "
   2 PS1="\D{%Y-%m-%d}T\A \u@\H\n\w\r\n\$ "
   3 $PS1="$(date +'%Y-%m-%dT%H:%M:%S') [\u@\h:\l \w]\\n\\$"
   4 PS1="\D{%Y-%m-%dT%H:%M:%S} [\u@\h:\l \w]\n$"

PowerShell commands

Get-ChildItem c:\windows\system32\*.txt -Recurse | Select-String -Pattern "Microsoft" -CaseSensitive
sls "searchstring" file

Search using GNU grep

   1 #!/bin/sh
   2 echo "Searching $1 in $(pwd)"
   3 grep -H -n "$1" * -R

Grep to ignore lines

   1 grep -v "^#" file # grep file ignoring lines starting with #
   2

Show full user name with ps command

To see every process with a user-defined format:

ps axo user:32,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,comm
o format User-defined format.

watch command

WATCH(1)                                                                                      User Commands                                                                                      WATCH(1)

NAME
       watch - execute a program periodically, showing output fullscreen

By default, the program is run every 2 seconds.
watch -n 5 sh command.sh # executes the script command.sh each 5 seconds and show its output on the screen

screen command

https://www.thegeekstuff.com/2010/07/screen-command-examples/

Screen multiplexer. The screen command is most used for ssh session because it helps to continue your work after a disconnection without losing the current processes in progress.

# start bash in screen
screen bash
# detach from bash ( it will keep running)
# ctrl+a d
# check running processes with screen
screen -ls
# attach to process
screen -r <id in screen -ls>

calculate_space.py

calculate_space.py

   1 #!/usr/bin/python
   2 import os
   3 
   4 filename='du_out.txt'
   5 os.system('du . -h > %s'%(filename))
   6 
   7 f = open('%s'%(filename))
   8 lines=[]
   9 
  10 for item in f:
  11     lines.append(item)
  12 line_g = [ l for l in lines if "G\t." in l]
  13 
  14 for l in line_g:
  15     print l.replace("\n","")
  16 os.system('rm %s'%(filename))

Bell warning sound console

~/scripts/bell,sh
chmod 755 ~/scripts/bell,sh

   1 #!/bin/sh
   2 TEXT1="1 2 3 4"
   3 
   4 for WORD in $TEXT1
   5 do
   6   aplay /usr/share/sounds/sound-icons/prompt.wav > /dev/null  2>&1
   7 done

Archive files modified in the last 15 days

   1 find . -type f -mtime -15 -print0 | tar -czvf /tmp/backup.tar.gz --null -T -
   2 tar tvzf /tmp/backup.tar.gz

findtext

   1 #!/bin/sh
   2 grep -n -H "$1" -R 2>/dev/null

findresource

   1 #!/bin/sh
   2 find . -name "*$1*" 2>/dev/null

Format Pen drive NTFS

   1 sudo dmesg # find drive
   2 sudo cfdisk /dev/sdb # if sbd is the drive for the identified pen drive
   3 # create new partition NTFS in sdb1
   4 # delete, new, 14.5 GB, primary, type 7 HPFS/NTFS/exFAT, write, yes, quit
   5 sudo mkfs.ntfs /dev/sdb1

-  ⇤ ← Revision 47 as of 2018-10-12 10:36:26 → 
  Size: 8836
  Editor: localhost
  Comment:
+   ← Revision 74 as of 2025-04-05 21:13:39 → ⇥
  Size: 12005
  Editor: vitor
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
+<<TableOfContents(2)>>
-Line 62:
+Line 64:
-== Deploy EAR script ==
+= Deploy EAR script =
-Line 90:
+Line 92:
-== List JAR contents to file ==
 * cd ~/.m2/repository
 * find . -name "*.jar" | xargs -i jar -tf {} > /tmp/explodedJars.txt
 * find . -name "*.jar" | xargs -i sh -c "echo {} ; jar -tf {}" > /tmp/explodedJars.txt

== Size of each directory inside current directory ==
+= List JAR contents to file =
{{{#!highlight sh
cd ~/.m2/repository
find . -name "*.jar" | xargs -i jar -tf {} > /tmp/explodedJars.txt
find . -name "*.jar" | xargs -i sh -c "echo {} ; jar -tf {}" > /tmp/explodedJars.txt
}}}


= Size of each directory inside current directory =
-Line 98:
+Line 103:
-== Backup folders in HOME ==
+= Backup folders in HOME =
-Line 113:
+Line 118:
-== Get installed updates on Windows ==
+= Get installed updates on Windows =
-Line 118:
+Line 123:
-== Find files modified less than an hour ago ==
+= Find files modified less than an hour ago =
-Line 123:
+Line 128:
-== Lock file to allow only one instance of a script to run ==
+= Lock file to allow only one instance of a script to run =
-Line 157:
+Line 162:
-== awk , load file into array ==
+= awk , load file into array =
-Line 191:
+Line 196:
-== Shell script sample ==
+= Shell script sample =
-Line 252:
+Line 257:
-}}}

== Change TIME_WAIT (time wait) tcp/ip ==
+echo "Input data"
read IN1
echo $IN1

case $IN1 in
  hi)
    echo "yeahh"
    ;;
  *)
    echo "ups"
    ;;
esac
}}}

= Change TIME_WAIT (time wait) tcp/ip =
-Line 256:
+Line 274:
-{{{
+{{{#!highlight sh
-Line 265:
+Line 283:
-== Show root folders occupied space ==
+= Show root folders occupied space =
-Line 275:
+Line 293:
-== Add swapfile 512MB ==
+= Add swapfile 512MB =
-Line 290:
+Line 308:
-== Alias current date ==
+= Add swapfile 4GB =
{{{#!highlight sh
# create_swapfile.sh 
#!/bin/sh
# 4GB swapfile
# bs 1024 bytes 
# calc 2^30*4/1024
# 4194304.000
dd if=/dev/zero of=/swapfile1 bs=1024 count=4194304
chown root:root /swapfile1
chmod 0600 /swapfile1
/sbin/mkswap /swapfile1
/sbin/swapon /swapfile1
free -m
vi /etc/fstab
# /swapfile1 none swap sw 0 0
free -m
htop
}}}

= Alias current date =
-Line 295:
+Line 333:
-== Alias history without line number ==
+= Alias history without line number =
-Line 300:
+Line 338:
-== Calculator using bc ==
+= Calculator using bc =
{{{#!highlight sh
cd ~/scripts/
nano calc
chmod 755 calc
}}}
-Line 306:
+Line 350:
-== Get slackbuild ==
+= Get slackbuild =
-Line 342:
+Line 386:
-== Prompt PS1 without current folder ==
+= Prompt PS1 without current folder =
-Line 347:
+Line 391:
-PS1="\D{%Y-%m-%d}T\A \u@\H\r\n\$ "
}}}

== Prompt PS1 with folder ==
+PS1="\D{%Y-%m-%dT%H:%M:%S} [\u@\h:\l]\n$"
}}}

= Prompt PS1 with folder =
-Line 354:
+Line 399:
-Line 356:
+Line 400:
-}}}

== PowerShell commands ==
+PS1="\D{%Y-%m-%dT%H:%M:%S} [\u@\h:\l \w]\n$"
}}}

= PowerShell commands =
-Line 365:
+Line 409:
-== Search using GNU grep ==
+= Search using GNU grep =
-Line 372:
+Line 415:
+= Grep to ignore lines =
{{{#!highlight bash
grep -v "^#" file # grep file ignoring lines starting with #
}}}

= Show full user name with ps command =
To see every process with a user-defined format:
 * ps axo user:32,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,comm
 * o format User-defined format.

= watch command =
{{{
WATCH(1)                                                                                      User Commands                                                                                      WATCH(1)

NAME
       watch - execute a program periodically, showing output fullscreen

}}}
 * By default, the program is run every 2 seconds.
 * watch -n 5 sh command.sh # executes the script command.sh each 5 seconds and show its output on the screen

= screen command =
 * https://www.thegeekstuff.com/2010/07/screen-command-examples/
Screen multiplexer.
The screen command is most used for ssh session because it helps to continue your work after a disconnection without losing the current processes in progress.

{{{
# start bash in screen
screen bash
# detach from bash ( it will keep running)
# ctrl+a d
# check running processes with screen
screen -ls
# attach to process
screen -r <id in screen -ls>
 
}}}

= calculate_space.py  =
 * calculate_space.py 
{{{#!highlight python
#!/usr/bin/python
import os

filename='du_out.txt'
os.system('du . -h > %s'%(filename))

f = open('%s'%(filename))
lines=[]

for item in f:
    lines.append(item)
line_g = [ l for l in lines if "G\t." in l]

for l in line_g:
    print l.replace("\n","")
os.system('rm %s'%(filename))
}}}

= Bell warning sound console =
 * ~/scripts/bell,sh
 * chmod 755 ~/scripts/bell,sh
{{{#!highlight bash
#!/bin/sh
TEXT1="1 2 3 4"

for WORD in $TEXT1
do
  aplay /usr/share/sounds/sound-icons/prompt.wav > /dev/null  2>&1
done
}}}

= Archive files modified in the last 15 days =
{{{#!highlight bash
find . -type f -mtime -15 -print0 | tar -czvf /tmp/backup.tar.gz --null -T -
tar tvzf /tmp/backup.tar.gz
}}}

= findtext =
{{{#!highlight sh
#!/bin/sh
grep -n -H "$1" -R 2>/dev/null
}}}

= findresource =
{{{#!highlight sh
#!/bin/sh
find . -name "*$1*" 2>/dev/null
}}}

= Format Pen drive NTFS =
{{{#!highlight sh
sudo dmesg # find drive
sudo cfdisk /dev/sdb # if sbd is the drive for the identified pen drive
# create new partition NTFS in sdb1
# delete, new, 14.5 GB, primary, type 7 HPFS/NTFS/exFAT, write, yes, quit
sudo mkfs.ntfs /dev/sdb1
}}}