removing duplicate files using shell script

Friends,

One liner script provided below, can be used to identify the duplicate files within the system. If you want to search in a partticular path, just replace it with "/tmp" provided in sample command. It will redirect all the duplicate file names into removal_list.txt which can be used to delete them.

find /tmp "$@" -type f -print0 xargs -0 -n1 md5sum sort --key=1,32 uniq -w 32 -d --all-repeated=separate sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> removal_list.txt

Performance comparision

Checkout the "time" command.

For checking the time taken for execution of any command do the following:

Say you want to execute "ls -l grep sanju.txt" and measure time taken for this.
Now open a bash shell and on the prompt
$time ls -l grep sanju.txt

it shows you the time taken by the command execution

Unaliasing in BASH

Unaliasing in BASH
How can we unalias a command temporarily without using unalias command ?

just put the escape character before the command i.e. "\"e.g. :-

$\ls

It will show the without alias.

How to count a particular word in VI?

If you want to count a particular word, say PROG, in a file, try this trick

:1,$s/zebra/&/g

or

if you want to count all words in a file, try

g [ctrl-g] at command mode

source: http://www.vim.org/tips/tip.php?tip_id=689

Some useful commands

df : report disk free space
gview : load VIM, improved vi editor
history : display a list of previous commands
!! : repeat previous command
ls -p grep "/" : list sub-directories in the current directory
touch file.ext : create a NULL file named 'file.ext'
vmstat : virtual memory status
uname : system information

TRIMMING THE LOG

Various unix processes canproduce fast growing logs thatsometimes need to be trimmedinstead of deleted, for referenceor troubleshooting. And youlikely have no desire to edit thefiles. Here is a handy kshscript that will quickly trimthe log so it keeps recentinformation and lets you keep asmany lines as you think you mightneed.

I call it trimlog:

#! /bin/sh
# trimlog
filesize=`cat $1wc -l`
trim=`expr $filesize - $2`
if [ $trim -gt 0 ]
then
sed "1,$trim d" $1 > /tmp/$1
mv /tmp/$1 $1
echo $1 trimmed by $trim lines
else
echo "Invalid argument"
fi
Use it by feeding in the name of the log you want to trim followedby the number of lines you want to keep:

# trimlog oracle_listener.log 10000

Of course you can't use it in the /tmp directory.
There are probably newer and more efficient ways to do this, but this works.

kill really does not KILL

kill is not really meant only to kill a process. But mainintention is to send a signalto process. And the processhandles these signals to dovarious tasks.

kill -INT 1234

where 1234 is the Process ID is really, this set an interrupt signal, Ctrl-C is an example of interupt signal givento shell.

kill -HUP 1234

some UNIX daemons handle this, to do refreshing business.

kill -KILL 1234

This is the sure kill,which cannot be handled by any process. Also know as,

kill -9 1234

Just, check the in the file /usr/include/sys/signal.h for all signals on the system or you can use:

kill -l

to get the complete list of signals.

Note: More about this can be found onkill and signal man pages.

AUTOMATIC LOGIN TO REMOTE HOSTS w/o PASSWORD

Consider two hosts.host1.domain.comhost2.domain.com

>From hosts (user:deepak), you wish to logon tohost1 (user:paul).

Paul would create a .rhostsfile in his directory

chmod 744 .rhosts

(No write permission forothers)
The first line would contain

host1.domain.com deepak

Now, deepak should be able to logon to host2.domain.comas user paul by using rlogin.
Deepak will type

e.g.
rlogin hosts.domain.com -l paul

And he should be in.

CRYPT AN ASCII FILE

An ascii file can be easily encrypted and decrypted.
To encrypt simply pipe the STDOUT ofthe file to "crypt" and redirect it toa new file name. Enter a passowrd whenprompted with "Enter key".

$ cat foo crypt > foo.e
Enter key:

To unencrypt simply pipe the STDOUT ofthe encrypted file to "crpyt" andredirect it to a new file name. Entera passowrd when prompted with"Enter key".
$ cat foo.e crypt > foo.new
Enter key:

CASE TRANSLATION

If you have occasion whereyou need to translate uppercase to lower and viceversa, and happen to run aksh, enter the following functions into your .bash_profile:

trans ()
{
tr '[:upper:]' '[:lower:]' < $1 > $2
}
TRANS()
{
tr '[:lower:]' '[:upper:]' <$1 > $2
}
The first function trans will translate all uppercaseletters in the first file(argument) to lowercase in the second file.
The second function TRANS does the exact opposite.

Tracking User's activity using "script -a"

Unix System provides a very good method allows us to capture of all commands issued by the user “XYZ”. For this, we can user "script" command. This will print all what appears on the screen (commands and their output) to the filename

But remember, Its not a User History.... It stores all commands entered by all users (.e. its the shell history .

Add the following line to your BASH profile

script -a ~/.termlog`date +%m%d%y`

This enable to capture all the terminal activities to a file ending with current date

Cut and paste text files by columns

How many times have you wanted to edit those results files columnwise?

Here’s a generic UNIX way: (startcol and endcol refer to character columns, including whitespace. Tab counts as 1.)

cut -cstartcol-endcol file {>receiving file} paste sourcefile1 sourcefile2 {>final file}

I like to preview and fine-tune my commands by looking at the result on standard out (the shell window) and then redirecting output to a (temporary) file when things look right. Those feeling really adventurous can combine things into a single command line…

Great Unix Tips & Tricks: Vol 1

Whether you are a newbie to Linux or you are a seasoned guru, you are bound to find some scripts or programs which just make your life on the computer exponentially easier. The following is a list of them and explains what they do.

1. tac

tac is a command similar to cat, except that it reverses the output of the specified file.

tac

2. Multiple X-Sessions

Using multiple X-Sessions allows a user to have open more than just one instance of XFree, so multiple desktop environments can be used at a time.

startx -- :

terminal name is simply which terminal X should start on. By default it starts on

0:0. If you want two open, you may want it open on 0:1, which can be run by:

startx -- :1

The resulting desktops will exist on F7 - F12.

3. links

links is a text based browser which has full support for tables

links

is simply the hostname or IP address of the computer to connect to.

sparse files

Due to the Unix concept of sparse files, you can create seemingly enormous files that in reality take next to no disk space. The following program will create a 305,419,897 byte file called 'core' that may result in you receiving a 'cleanup' email message from a less-than-seasoned Unix sysadmin even though it occupies virtually no real disk space.

    $ cat bigcore.c
    #include 
    #include 

    int main(void) {
        int fd = open("core", O_CREAT|O_WRONLY|O_TRUNC, 0600);
        lseek(fd, 0x12345678, SEEK_SET);
        write(fd, "1", 1);
        close(fd);
        return 0;
    }

    $ cc -o bigcore bigcore.c
    $ ./bigcore
    $ ls -l core
    -rw-------  1 dmr  staff  305419897 May  1 03:50 core
    $ du -k core
    48      core
    $

Limiting File Size in UNIX

C shell
limit filesize 2m

Korn/Bash shell

ulimit -f max-size
ex: Korn/Bash shell limit 2 MB.
ulimit -f 2000

Removing non-consecutive duplicate lines from a file

The uniq command will "Discard all but one of successive identical lines" from a file or input stream.

In order to remove non-consecutive duplicate lines, use awk:

awk '!x[$0]++' FILE

A list of directories

To see a list of subdirectories of the current directory:

printf "%s\n" */

With a Bourne shell:

echo */.

The Easy PATH

To format one's PATH variable for easy viewing, try this function:
==============

path()
{
oldIFS=$IFS
IFS=:
printf "%s\n" $PATH
IFS=$oldIFS
}
==============

A typical run of the function:
$ path
/bin
/usr/bin
/usr/bin/X11
/usr/X11R6/bin
/usr/local/bin
/home/sanju/bin
/home/sanju/scripts

Linux: How to Encrypt and decrypt files with a password

To encrypt and decrypt files with a password, use gpg command. It is an encryption and signing tool for Linux/UNIX like operating system such as FreeBSD/Solaris.

GnuPG stands for GNU Privacy Guard and is GNU's tool for secure communication and data storage. It can be used to encrypt data and to create digital signatures. It includes an advanced key management facility.

Encrypting a file in linux
To encrypt single file, use command gpg as follows:

$ gpg -c filename

To encrypt myfinancial.info file, type the command:$ gpg -c myfinancial.info

Output:

Enter passphrase:
Repeat passphrase:

This will create a myfinancial.info.gpg file.

Option:
-c : Encrypt with symmetric cipher

Caution if you ever forgot your password aka passphrase, you cannot recover the data as it use very strong encryption.

Shell Scripting: Using date within report/log file names

When you write a shell scripts you need to create filename with date in it. For example instead of log file name "traces.log", you can create a filename called "traces-Jun-27-8.log".

The date in file will make it easy to find out all logs or reports. You can display the current date and time in the given FORMAT using date command. If you just type date command it will display in standard FORMAT:

$ date
Output:
Fri Jun 27 11:43:25 IST 2008

To display date in MONTH-DAY-YEAR format you need to use date command as follows:

$ date +"%b-%d-%y"

Jun-27-08

As you can see I have used FORMAT as follows

date +"FORMAT"

Where, FORMAT can be any one of the following:

• %a : Abbreviated weekday name (Sun..Sat)
• %b : Abbreviated month name (Jan..Dec)
• %B : Full month name, variable length (January..December)
• %d : day of month (01..31)
• %e : day of month, blank padded ( 1..31)
• %H : 24 hour format (00..23)
• %I : 12 hour format (01..12)
• %j : day of year (001..366)

First obtained date:
$ NOW=$(date +"%b-%d-%y")

Create a file with date in filename
$ LOGFILE="log-$NOW.log"

Display filename:
$ echo $LOGFILE

You can use first two commands in script.