Linux Command Line tips that every Linux user should know

Below is the collection of Linux command line tips which I’ve found useful for Linux users. To get more information about the command mentioned below just open your terminal and type man <command>.

Things a Linux user must learn

  • Learn bash: No need to refer a lengthy bash guide or something else. Just read the complete man page of bash (man bash).
  • Learn vim: You might be using Emacs or Eclipse for your work all the time but nothing can compete vim.
  • Learn ssh: Learn the basics of passwordless authentication.
  • Learn basics of bash job management: Using &, Ctrl-C, fg, bg, Ctrl-Z, jobs, kill.
  • Learn basic commands for file management: ls and ls -l, less, head, tail and tail -f, ln and ln -s (learn the differences between hard links and soft links), chown, mount, chmod, df, du (du -sk *).
  • Learn basic commands for network management: dig, ifconfig.
  • Learn how to use grep, find and sed.
  • Learn how to use aptitude or yum (depends on the distro) to find and install packages.

For daily use

  • In bash, you may use Ctrl+R to search in command history.
  • In bash, you may use Ctrl+W to delete the last word, and Ctrl+U to delete the complete line.
  • Use cd – command to go back to the previous working directory.
  • Learn how to use xargs.

$ find . -name \*.py | xargs grep some_function

$ cat hosts | xargs -I{} ssh [email protected]{} hostnameX

  • Use pstree -p command to get see the process tree.
  • Learn various signals. For example, to suspend a process, use kill -STOP [pid]. Type man 7 signal in terminal for complete guide.
  • If you want to keep running a background process forever then you can use nohup or disown.
  • Use netstat -lntp command to see what the processes are listening. You should check about lsof also.
  • In your bash script you can use subshells to group commands.

# do something in current dir

(cd /some/other/dir; other-command)

# continue in original dir

  • Trimming of strings: ${var%suffix} and ${var#prefix}. For example if var=foo.pdf, then echo ${var%.pdf}.txt prints “foo.txt”.
  • The output of a command can be treated like a file via <(some command). For example, compare local /etc/hosts with a remote one: diff /etc/hosts <(ssh somehost cat /etc/hosts)
  • Know about “here documents” in bash.
  • Learn how to redirect both standard output and standard error via: some-command >logfile 2>&1.
  • You should know about ASCII table (with hex and decimal values). Type man ascii in terminal.
  • While working remotely via ssh, you should use screen or dtach to save your session.
  • For web deveopers use of curl and curl -I, wget etc is useful.
  • To convert HTML page to text file: lynx -dump -stdin
  • If you must handle XML, xmlstarlet is good.
  • In ssh, learn how to port tunnel with -L or -D (and occasionally -R). Also learn how to access web sites from a remote server.
  • If you were typing a command but then changed your mind, Press Alt+shift+3. It will add # at the beginning and enter it as a comment.

Data processing

  • Learn about sort and uniq.
  • Learn about cut, paste, and join.
  • Learn how to get union, intersection and difference of text files.

cat a b | sort | uniq > c # c is a union b

cat a b | sort | uniq -d > c # c is a intersect b

cat a b b | sort | uniq -u > c # c is set difference a – b

  • Summing all numbers in the second column of a text file, code given below is probably 3X faster and 3X shorter than equivalent Python.

awk ‘{ x += $2 } END { print x }’

  • Learn about strings and grep command.
  • To split files into different parts learn about split (to split by size) and csplit (to split by a pattern).

System debugging

  • To know the status of your disk, cpu or network use iostat, netstat, top (or the better htop), and (especially) dstat.
  • To know your system’s memory status use free and vmstat command.
  • Use mtr which is a network diagnostic tool.
  • To find out which process or socket is using bandwidth, try iftop or nethogs.
  • You may use ab tool which is helpful for quick checking of web server performance.
  • For more serious network debugging take use of wireshark or tshark.
  • Learn how to use strace, and that you can strace a running process (with -p). This is helpful if your program is failing, hanging, or crashing, and you don’t know why.
  • Use the ldd command to check shared libraries.
  • Learn how to connect to a running process with gdb and get its stack traces.
  • Knowledge of /proc is very helpful. Examples: /proc/cpuinfo, /proc/xxx/smaps, /proc/xxx/exe, /proc/xxx/cwd, /proc/xxx/fd/.
  • When debugging why something went wrong in the past? To know about this use the sar command. It collects, reports and saves system activity information.

PS: I think I have missed some tips because they didn’t come in my mind at the moment. If you know some good command line tips then please share them in the comment. Thank you :)

Edit: I found some useful tips from reddit users which they gave after reading this post.

1. ifconfig is deprecated, alternative of it is ip.
2. Use of aliases is also an important thing which I forgot to mention.

Cheat Sheet

Download or bookmark the cheat sheet given below. It is very useful.

linux command line cheat sheet

Noticed “rm -rf / – make computer faster” under file commands in the cheat sheet? Don’t ever do that, that will delete all of your files. Do it on your enemy’s system :P

How to read a CSV file in Perl?

read csv file perl

Perl is used for manipulating text files and in this article I will teach how to read a CSV file in Perl? First of all let’s start with

What is a CSV file?

A comma-separated values (CSV) file stores tabular data (numbers and text) in plain-text form. As a result, such a file is easily human-readable (e.g., in a text editor).

CSV is a simple file format that is widely supported by consumer, business, and scientific applications. Among its most common uses is to move tabular data between programs that naturally operate on a more efficient or complete proprietary format. For example: a CSV file might be used to transfer information from a database program to a spreadsheet. -Wikipedia

Using Perl to read a CSV file

Now suppose we have a CSV file which has 4 rows and each row has fields which are separated by comma as given below:

Linux,Unix,1,Ubuntu
RHEL,Kubuntu,2,Fedora
Mint,Puppy,3,ArchLinux
Mandriva,OpenSUSE,4,CentOS

If you want to extract out the data from 3rd column and calculate the sum of all the numbers then the approach to do this will be as follows:

Read the first line of the CSV file and extract out the 3rd column. Assign the extracted value to a scalar variable and then read the next line, extract out the value of 3rd column, add it to the scalar variable which contains the first value and store result in the same variable. Then read the 3rd line and so on…

Below is the Perl code for this purpose using split function.

  #!/usr/bin/perl
  use strict;
  use warnings;
 
  my $filename = $ARGV[0];
  my $total = 0;
 
  open(my $FH, '<', $filename) or die "Couldn't open $filename $!";
 
  while (my $fetchline = <$FH>){
    chomp $fetchline;
    my @elements = split "," , $fetchline;
    $total += $elements[2];
  }
 
  print "$total\n";

Save the file as text.plx and if the CSV file name is filename.csv then you may run the code as

$ perl text.plx filename.csv

In this case we get 10 (1+2+3+4) as output.

In this article we used split function to extract out the columns from CSV file using comma (,) as separator. But this code is error prone because of 2 reasons.

Reason 1: If the filename.csv contains something like

Linux,Unix,1,Ubuntu
RHEL,Kubuntu,2,Fedora
Mint,"Puppy,Linux",3,ArchLinux
Mandriva,OpenSUSE,4,CentOS

Notice the 3rd row. The 2nd field of it has a comma “Puppy,Linux”. If we use the same approach of splitting fields by comma separator then it will throw an error because split function will split out Puppy and Linux since they are separated by a comma. Then it will do something like 1+2+Linux+4 which is wrong.

Reason 2: If the filename.csv contains something like

Linux,Unix,1,Ubuntu
RHEL,Kubuntu,2,Fedora
Mint,"Puppy,
Linux",3,ArchLinux
Mandriva,OpenSUSE,4,CentOS

Here the 3rd row contains multi-line fields. When the above code will be used it will treat the CSV file as it contains 5 lines instead of 4.

So we can conclude that if the CSV file is in good format AND if it does not contain field separators within quote AND if there are no multi-line fields in CSV file then our code (which we have written above) will work perfectly and will give correct result. Take use of Text::CSV_XS module which is used for reading and writing CSV files, it will deal with those error prone situations very efficiently.

How to find the CPU information from terminal in Linux?

lscpu output

You can use lscpu command or can view the file /proc/cpuinfo to get the CPU information from terminal.

Man page description of lscpu command:

lscpu gathers CPU architecture information like number of CPUs, threads, cores, sockets, NUMA nodes, information about CPU caches, CPU family, model and stepping from sysfs and /proc/cpuinfo, and prints it in human-readable format. Alternatively, it can print out in parsable format including how different caches are shared by different CPUs, which can also be fed to other programs.

$ lscpu

or

$ cat /proc/cpuinfo

Output:

How to delete all files of a directory in Linux?

How to delete all files of a directory in Linux?

To delete all files of a directory in Linux there are 2 methods available.

Suppose you want to delete all files of the directory dirname whose path is /home/usr/dirname then to delete all files of this directory you may use the 2 methods give below:

Method 1: Use rm command

$ rm /home/usr/dirname/*

or

$ cd /home/usr/dirname/
$rm *

Note: The above method will not delete files from sub-directories of dirname. To remove the files from sub-directories use

$ rm -r /home/usr/dirname/*

Use f option to force delete the files.

rm -rf /home/usr/dirname/*

Method 2: Use Find command

$ find /home/usr/dirname -type f -delete

or

$ cd /home/usr/dirname
$ find . -type f -delete

Note: Using the above method will delete all the files from sub-directories of dirname but it will not delete the sub-directories.

Edit: find and delete all files and folders (sub folders) [credit: ubungu]

find path_to_find -name ‘*test*’ -exec rm -rf {} \;

Types of Filesystem, differences and dealing with them using Perl

If you are looking to write code in Perl which will work on multiple platforms then you must have the knowledge of filesystem of each platform. Filesystems are categoriesed in 3 parts.

1. Filesystem of Unix

FFS (Berkeley Fast File System) is the ancestor of all current filesystems of Unix variants. The filesystem has been extended by different vendors according to their needs. Like for better security they have been extended to provide support for POSIX ACL (access control list).

In Unix filesystem the root is denoted by / (forward slash). The path of any directory in Unix filesystem starts from / and goes deeper in the system. The directory names or file names in Unix filesystem are case-sensitive.

Path Example: /root/home/user/Desktop/file.txt

2. Filesystem of Windows

Operating systems which are based on Windows supports 3 filesystems: FAT (File Allocation Table), NTFS (NT FileSystem), FAT32 (Advance version of FAT).

FAT filesystem is not case-sensitive. It uses \\ (backward slash) as path separator. In the FAT systems the direcotries and files contains some flags, these flags are known as attributes like “Read Only”. NTFS supports unicode feature.

Path Example: C:\\home\\user\\Desktop\\file.txt

3. Filesystem of Mac OS

Classic Mac OS used HFS (Hierarchical File System). The version 8.1 of Mac OS uses HFS+ which is an advance version of HFS. The current filesystem looks similar to Unix filesystem. The method of showing paths is same in both OS. The difference between these is that Unix file system is case-sensitive whereas HFS+ is not case-sensitive.

Summary

OS and FileSystem Path Separator Filename length Absolute path format Relative path format Unique features
Unix (Berkeley FFS and others) / OS-dependent number of chars /dir/file dir/file OS-variant dependent additions
Mac OS (HFS+) / 255 Unicode chars /dir/file dir/file Mac OS legacy support, BSD extended attributes
Windows based OS (NTFS) \\ 255 Unicode chars Drive:\\dir\\file dir\\file File encryption and compression
DOS (basic FAT) \\ 8.3 Drive:\\dir\\file dir\\file Attributes

 

Using Perl to deal with different Filesystems

By now we know all the filesystems differ from each other and to deal with these in Perl we use File::Spec module. This module is used to hide the differences between these filesystems. Let us learn how we use this module. We will use the catfile method to get the path of the file.

Example:

use File::Spec;
my $path= File::Spec->catfile(qw{home user Desktop file.txt});

In this way the scalar variable $path get sets to home\user\desktop\file.txt in a Windows operating system. and in Unix based system it gets set to home/user/Desktop/file.txt. The File::Spec module also contains methods like curdir and updir to get the current directory (.) and up-directory (..).

You may also use the moudle Path::Class which is an another good wrapper. You will have to install it before using.

Below is an example of how it works:

use Path::Class;
my $file = file (qw{home user Desktop file.txt});
my $dir = file (qw{home user Desktop});

The $file and $dir are scalar variable which contains the path to the file.txt file and Desktop directory respectively.

Here’s the path

print $file;
print $dir;

The output will depend on operating system. If it is Unix then it will yield

home/user/desktop/file.txt and home//user//desktop

and in Windows based systems it will give

home\\user\\desktop\\file.txt and home\\user\\desktop

Here $file and $dir are objects which have several methods which can be applied on them for some use. Like absoulte method gives the absolute path and slurp method slurps through the file’s contents.

my $abspath = $file->absolute;
my $content = $file->slurp;
$file->remove(); #delete the file

When you want to write a code for a particular system and you want to make this system understand the path of other operating system then you may use foreign_file() and foreign_dir() methods. These will return the path based the argument which you have to specify explicitly.

use Path::Class qw(foreign_file foreign_dir);
my $foreignfile = foreign_file('Win32', qw{home user Desktop file.txt});
my $foreigndir = foreign_dir('Win32', qw{home user});

Now $foreignfile will contain home\\user\\desktop\\file.txt even if the code is run from a Unix based system. This approach is very handy and useful. In the next article we will continue and expand our discussion.

Understanding file permissions and access rights in Linux

In Linux everything is a file. To set access rights on the specific files we use chmod command. To see what are the access rights/ permissions of a specific file we use ls -l command. Below we will learn how to see permission of a file and change it.

Note: To change permissions of a file you must be login as root user.

Suppose you have a file named as linuxstall.txt in a directory named as LinuxStall which is in /tmp. Use ls -l command to see its permissions as:

ls -l /tmp/LinuxStall/linuxstall.txt

The output will be as

-rwxr--r--. 1 root root 0 Jan 5 15:39 /tmp/LinuxStall/linuxstall.txt

Let us first understand what does the above line means by breaking it down in different parts.

The left most bit can be of two types either d or .
If it is d then it means that it is a directory.
If it is then it means that it is a file.

Here in our example it is means it is a file.

Part 1: r w x (Shows owner’s access rights)
Explanation:
First bit is which means it is a file, if it is d then it means it is directory.
First bit is r which means read i.e the owner can open this file.
Second bit is w which means write i.e the owner can edit this file.
Third bit is x which means execute i.e the owner can execute this file.

Part 2: r – – (shows group’s access rights)
Explanation:
First bit is r which means read i.e the group members can open this file.
Second bit is which means group members can not edit this file.
Third bit is which means group members can not execute this file.

Part 3: r – – (shows other users which are neither owner nor group members)
Explanation:
First bit is r which means read i.e the other users can open this file.
Second bit is which means other users can not edit this file.
Third bit is which means other users can not execute this file.

 

Changing file permission using chmod command

chmod stands for change file mode bits

There are two methods of changing file permission using chmod command.

Syntax: chmod permission path_of_file

Method 1: Numerical Method

In this method we use numeric bits which corresponds to r w x. The numeric values for these are

r = 4, w = 2, x = 1

chmod 444 /tmp/LinuxStall/linuxstall.txt

will change the file permission of file linuxstall.txt to

-r – – r – – r – – which means owner, group, others can only read this file.

chmod 555 /tmp/LinuxStall/linuxstall.txt

will change the file permissions of file linuxstall.txt to

-r – x r – x r – x which means owner, group, other can read and execute the file.

chmod 666 /tmp/LinuxStall/linuxstall.txt

will change the file permissions of file linuxstall.txt to

-r w – r w – r w – which means owner, group, other can read and edit the file.

chmod 644 /tmp/LinuxStall/linuxstall.txt

will change the file permissions of file linuxstall.txt to

-r w – r – – r – – which means owner can read and edit the file, group and other can only read the file.

Hence we saw that first numeric bit is for owner, second numeric bit is for group and third one is for other users. You can set the numeric bits to 1,2,4 for execute, write and read purpose respectively.

Method 2: Alphabetical Method

There are 4 alphabets associated with it, they are u,g,o and a.

u stands for owner

g stands for group

o stands for others

a stands for all

To add permission we use + and to remove permission we use

Example 1: To grant read permission to all group users

chmod g+r /tmp/LinuxStall/linuxstall.txt

Example 2: To remove read permission from group users

chmod g-r /tmp/LinuxStall/linuxstall.txt

Example 3: To grant read,write,execute permission to owner

chmod u+rwx /tmp/LinuxStall/linuxstall.txt

Example 4: To remove execute permission from other users

chmod o-x /tmp/LinuxStall/linuxstall.txt

Example 5: To grant read and write permission to all users

chmod a+rw /tmp/LinuxStall/linuxstall.txt

Noticed use of a? It sets permission for all users.

 

Conclusion: We saw that each file in Linux has access rights associated with it. It is divided in 3 parts, one is owner, second is group users and third are other users. We can use chmod command to change the file permission. There are two methods, one is numerical and other is alphabetical method. Numerical method of setting permission is preferred though.