View log files in real time

log files in realtime in linux

Most of the log files in linux can be found in /var/log . You can go to this directory with cd command but root permission is required.

cd /var/log

To view the log files in real time:

tail -f /path/toFile.log

For example, to view the live version of yum log files execute tht following command:

tail -f /var/log/yum.log

How to block a country using iptables?

If you are an admin of a website and you see a lot of bogus traffic coming from some countries which give no profit to you, and you want to block those countries from accessing your website then you can use the bash script given below.

There are two ways to block countries. First is to configure your Apache server and second is to set iptables commands. We will do this using iptables. First of all download the list of IP zone files of the country which you want to block from here.

[Warning]The script will not work if people of that country are using any proxy server or they have spoofed their IP address.[/Warning]

#!/bin/bash
### Block all traffic from AFGHANISTAN (af) and CHINA (CN). Use ISO code ###
ISO="af cn"
 
### Set PATH ###
IPT=/sbin/iptables
WGET=/usr/bin/wget
EGREP=/bin/egrep
 
### No editing below ###
SPAMLIST="countrydrop"
ZONEROOT="/root/iptables"
DLROOT="http://www.ipdeny.com/ipblocks/data/countries"
 
cleanOldRules(){
$IPT -F
$IPT -X
$IPT -t nat -F
$IPT -t nat -X
$IPT -t mangle -F
$IPT -t mangle -X
$IPT -P INPUT ACCEPT
$IPT -P OUTPUT ACCEPT
$IPT -P FORWARD ACCEPT
}
 
# create a dir
[ ! -d $ZONEROOT ] && /bin/mkdir -p $ZONEROOT
 
# clean old rules
cleanOldRules
 
# create a new iptables list
$IPT -N $SPAMLIST
 
for c  in $ISO
do
	# local zone file
	tDB=$ZONEROOT/$c.zone
 
	# get fresh zone file
	$WGET -O $tDB $DLROOT/$c.zone
 
	# country specific log message
	SPAMDROPMSG="$c Country Drop"
 
	# get
	BADIPS=$(egrep -v "^#|^$" $tDB)
	for ipblock in $BADIPS
	do
	   $IPT -A $SPAMLIST -s $ipblock -j LOG --log-prefix "$SPAMDROPMSG"
	   $IPT -A $SPAMLIST -s $ipblock -j DROP
	done
done
 
# Drop everything
$IPT -I INPUT -j $SPAMLIST
$IPT -I OUTPUT -j $SPAMLIST
$IPT -I FORWARD -j $SPAMLIST
 
# call your other iptable script
# /path/to/other/iptables.sh
 
exit 0

You must be logged in as a ‘root’ user to run this script. Mention the country names which you want to block in ‘ISO’.

To run the script

# /path/block_country.sh

You can add this script to crontab so that it will run automatically.

@weekly /path/block_country.sh

Below is an another script which does the same work:

#!/bin/bash
###PUT HERE COMA SEPARATED LIST OF COUNTRY CODE###
COUNTRIES="AK,AR"
WORKDIR="/root"
#######################################
cd $WORKDIR
wget -c --output-document=iptables-blocklist.txt http://blogama.org/country_query.php?country=$COUNTRIES
if [ -f iptables-blocklist.txt ]; then
  iptables -F
  BLOCKDB="iptables-blocklist.txt"
  IPS=$(grep -Ev "^#" $BLOCKDB)
  for i in $IPS
  do
    iptables -A INPUT -s $i -j DROP
    iptables -A OUTPUT -d $i -j DROP
  done
fi
rm $WORKDIR/iptables-blocklist.txt

No big useless shortcuts in Unity Dash of Ubuntu 12.04

If you don’t like those 8 big shortcuts in Unity Dash then this is a good news for you. There will be no big shortcuts in Unity Dash of Ubuntu 12.04. The reason of removing these is that they are useless, yes useless! Ask yourself how many times do you use them? Do they help you? If your answer is no then you should be happy now.

Old look of Unity Dash

The Unity Dash of Ubuntu 12.04 will consist of 3 sections:

1. Recent Apps: This section will display the recently used applications.

2. Recent Files: This section will display recently accessed files.

3. Downloads: This section will display the items of your ‘Downloads’ folder.

Here is how it will look:

New look of Unity Dash

Isn’t this cool? You can get access to your recently used Apps, files and downloaded items from just one place. I think this is a good and necessary decision which Ubuntu 12.04 developers have taken.

Microsoft’s attack on Linux: The hardware way

Microsoft restricts Linux on ARM

Microsoft did it again! This time by putting the gun on shoulders of ARM hardware makers. Their hatred for Linux is not a new issue, they have always been figuring out ways to edge out Linux. It has been in news that, Microsoft wishes ARM hardware manufacturers to make it impossible disabling UEFI secure boot feature which is required by Windows 8.

What is secure boot?

Secure boot is a scheme that checks all the stages of booting cryptographically maintaining the integrity of code under execution. It restricts the malicious or unauthorized software from running.

A Walkthrough

Later in the last year, Microsoft stated that secure boot would be required in order to boot Windows 8. Secure boot, as mentioned above, keeps ‘keys’ by itself inside the system. These keys are required if a software needs to run, Operating systems included. If the operating system is not ‘signed’ by a respective key, it cannot boot!

Microsoft is totally using secure boot the wrong way, with the prime intention of locking out Linux. Microsoft made crystal clear instructions for vendors:

MANDATORY: Disabling Secure MUST NOT be possible on ARM systems.

Which is in contrast to statement they made it on their blog:

Who is in control?
At the end of the day, the customer is in control of their PC. Microsoft’s philosophy is to provide customers with the best experience first, and allow them to make decisions themselves.

How the customer is in control if he is not allowed to disable features on the device he paid for! If it goes on this way, Linux users would be ‘forced’ to buy a Windows license which is so not fair.

How to delete spaces from file name in Linux?

If your file name has spaces in it and there are many such files that it is not possible for you to rename each file’s name manually then you may use the command given below. It will rename all the file names which have spaces by replacing the spaces with an underscore ( _ ).

for FILE in *; do mv "$FILE" "$(echo "$FILE"|tr ' ' '_')"; done

or

for file in *; do mv "$file" `echo $file | sed -e 's/ */_/g' -e 's/_-_/-/g'`; done

I had two files named a b c.txt and x y z.txt in my directory. I used the command given above to delete spaces from file name. It changed the file names to a_b_c.txt and x_y_z.txt respectively. See the screenshot below:

Difference between Unix and Linux

difference between Unix and Linux

When it comes to operating systems, Unix is the mother of operating systems. Members of rich Unix family are:

  • SVR4 (by AT&T)
  • BSD 4.4 (by University of California)
  • HP-UX (Hewlett-Packard)
  • Solaris (Sun Microsystems)

Mostly, Linux is considered to be a copy of UNIX. Lets hear the actual story now. Linux was actually a late addition to the family. It was written by Linus Torvalds back in 1991 and it was meant for IBM computers. As a matter of fact, in the world of operating systems, Linux has come up as a great operating system and it is welcomed with huge popularity. Commercial enterprise servers are running on Linux. Another cherry on the cake, Laptop and PC companies are also providing GNU Linux as a pre-installed OS on their systems so that individual users can also get a bite of it.

An extensive brain storming confusion is if Linux a kernel or Operating System. After some time, people just start ignoring this doubt as it never gets answered, well keep reading for the solution. What Linus Torvalds wrote was Linux kernel and it had a lot of features similar to UNIX one. Why it is confused with OS, actually commercially available distributions that provide graphical interface, compilers and other utilities along with Linux kernel are referred to as Linux Operating System.

Linux is, as they say, a UNIX-like kernel, because it has ‘some’ common features but still there are areas where they are not same. Difference between Unix and Linux can be understood by going through following points.

1. The difference in approach: There is a class of kernels that fall in Monolithic category, Monolithic ones are those which operate in one and only one process, they don’t have any other process for any kind of task. Another category is called Micro-kernel where core of the kernel is assigned one process while other processes are there for its side tasks like drivers, etc. Linux lies within Monolithic category with few exceptions in Micro-kernel.

2. Loadable kernel Modules: Unix systems are bound to have static links of new modules to be added or recently added. Linux differs here too, it supports new additions on the fly. For example, drivers can be loaded dynamically whenever they are needed. This feature is recognized as Loadable kernel Modules (LKM). This feature enables us to add any component dynamically without arising the need of compiling the whole kernel again. Unloading can also be performed in similar fashion. This adds to the flexibility of Linux.

3. Kernel threads: Kernel thread is an independent execution flow. It can be used to execute some user process or any kernel code. Most of the UNIX-like systems constitute kernel thread sets. Threads always operate in same address space, so it is not expensive to perform context switching on kernel threads in comparison to processes- this explains why Unix-like systems have kernel threads. Kernel threads are used, on linux, to execute kernel code.

4. Multi-threading: Multi-threaded applications are those which create multiple execution flows. These flows are also known as threads and are light weight processes. Nowadays all operating systems have support for multi-threading. Most of the UNIX-like operating systems LWP (read light weight process) are kernel thread based, while on the other hand Linux handles them a bit differently. In linux, LWP are created by calling the clone() function which leads to creation of separate processes. Same task can also be carried out with fork() function but clone() lets recently created processes share their memory, address space etc. Their working in shared environment gave them a name “threads”. Hence, multi-threading is supported by both of them but they differ in internal handling of it.

5. Preemption and non-preemption: We have a category of kernels which are able to pre-empt currently executing process and hence we call the preemptive kernels. Processes are run on a priority basis. If currently executing process is a low-priority one and a high-priority one process comes up, it can interrupt the current process and start executing itself. Non-preemptive kernels are those which don’t forcibly interrupt the current process even if a high-priority process has to wait. Linux based operating systems are non-preemptive while UNIX systems are fully preemptive. Linux based Realtime operating systems are found to be fully-preemptive.

Bottom line, despite taking the basic idea from UNIX it still differs in some aspects because it has inherited a lot from it and will always be considered a part of Unix family.

Linux Command Line tips that every Linux user should know

Below is the collection of Linux command line tips which I’ve found useful for Linux users. To get more information about the command mentioned below just open your terminal and type man <command>.

Things a Linux user must learn

  • Learn bash: No need to refer a lengthy bash guide or something else. Just read the complete man page of bash (man bash).
  • Learn vim: You might be using Emacs or Eclipse for your work all the time but nothing can compete vim.
  • Learn ssh: Learn the basics of passwordless authentication.
  • Learn basics of bash job management: Using &, Ctrl-C, fg, bg, Ctrl-Z, jobs, kill.
  • Learn basic commands for file management: ls and ls -l, less, head, tail and tail -f, ln and ln -s (learn the differences between hard links and soft links), chown, mount, chmod, df, du (du -sk *).
  • Learn basic commands for network management: dig, ifconfig.
  • Learn how to use grep, find and sed.
  • Learn how to use aptitude or yum (depends on the distro) to find and install packages.

For daily use

  • In bash, you may use Ctrl+R to search in command history.
  • In bash, you may use Ctrl+W to delete the last word, and Ctrl+U to delete the complete line.
  • Use cd – command to go back to the previous working directory.
  • Learn how to use xargs.

$ find . -name \*.py | xargs grep some_function

$ cat hosts | xargs -I{} ssh [email protected]{} hostnameX

  • Use pstree -p command to get see the process tree.
  • Learn various signals. For example, to suspend a process, use kill -STOP [pid]. Type man 7 signal in terminal for complete guide.
  • If you want to keep running a background process forever then you can use nohup or disown.
  • Use netstat -lntp command to see what the processes are listening. You should check about lsof also.
  • In your bash script you can use subshells to group commands.

# do something in current dir

(cd /some/other/dir; other-command)

# continue in original dir

  • Trimming of strings: ${var%suffix} and ${var#prefix}. For example if var=foo.pdf, then echo ${var%.pdf}.txt prints “foo.txt”.
  • The output of a command can be treated like a file via <(some command). For example, compare local /etc/hosts with a remote one: diff /etc/hosts <(ssh somehost cat /etc/hosts)
  • Know about “here documents” in bash.
  • Learn how to redirect both standard output and standard error via: some-command >logfile 2>&1.
  • You should know about ASCII table (with hex and decimal values). Type man ascii in terminal.
  • While working remotely via ssh, you should use screen or dtach to save your session.
  • For web deveopers use of curl and curl -I, wget etc is useful.
  • To convert HTML page to text file: lynx -dump -stdin
  • If you must handle XML, xmlstarlet is good.
  • In ssh, learn how to port tunnel with -L or -D (and occasionally -R). Also learn how to access web sites from a remote server.
  • If you were typing a command but then changed your mind, Press Alt+shift+3. It will add # at the beginning and enter it as a comment.

Data processing

  • Learn about sort and uniq.
  • Learn about cut, paste, and join.
  • Learn how to get union, intersection and difference of text files.

cat a b | sort | uniq > c # c is a union b

cat a b | sort | uniq -d > c # c is a intersect b

cat a b b | sort | uniq -u > c # c is set difference a – b

  • Summing all numbers in the second column of a text file, code given below is probably 3X faster and 3X shorter than equivalent Python.

awk ‘{ x += $2 } END { print x }’

  • Learn about strings and grep command.
  • To split files into different parts learn about split (to split by size) and csplit (to split by a pattern).

System debugging

  • To know the status of your disk, cpu or network use iostat, netstat, top (or the better htop), and (especially) dstat.
  • To know your system’s memory status use free and vmstat command.
  • Use mtr which is a network diagnostic tool.
  • To find out which process or socket is using bandwidth, try iftop or nethogs.
  • You may use ab tool which is helpful for quick checking of web server performance.
  • For more serious network debugging take use of wireshark or tshark.
  • Learn how to use strace, and that you can strace a running process (with -p). This is helpful if your program is failing, hanging, or crashing, and you don’t know why.
  • Use the ldd command to check shared libraries.
  • Learn how to connect to a running process with gdb and get its stack traces.
  • Knowledge of /proc is very helpful. Examples: /proc/cpuinfo, /proc/xxx/smaps, /proc/xxx/exe, /proc/xxx/cwd, /proc/xxx/fd/.
  • When debugging why something went wrong in the past? To know about this use the sar command. It collects, reports and saves system activity information.

PS: I think I have missed some tips because they didn’t come in my mind at the moment. If you know some good command line tips then please share them in the comment. Thank you :)

Edit: I found some useful tips from reddit users which they gave after reading this post.

1. ifconfig is deprecated, alternative of it is ip.
2. Use of aliases is also an important thing which I forgot to mention.

Cheat Sheet

Download or bookmark the cheat sheet given below. It is very useful.

linux command line cheat sheet

Noticed “rm -rf / – make computer faster” under file commands in the cheat sheet? Don’t ever do that, that will delete all of your files. Do it on your enemy’s system :P

How to read a CSV file in Perl?

read csv file perl

Perl is used for manipulating text files and in this article I will teach how to read a CSV file in Perl? First of all let’s start with

What is a CSV file?

A comma-separated values (CSV) file stores tabular data (numbers and text) in plain-text form. As a result, such a file is easily human-readable (e.g., in a text editor).

CSV is a simple file format that is widely supported by consumer, business, and scientific applications. Among its most common uses is to move tabular data between programs that naturally operate on a more efficient or complete proprietary format. For example: a CSV file might be used to transfer information from a database program to a spreadsheet. -Wikipedia

Using Perl to read a CSV file

Now suppose we have a CSV file which has 4 rows and each row has fields which are separated by comma as given below:

Linux,Unix,1,Ubuntu
RHEL,Kubuntu,2,Fedora
Mint,Puppy,3,ArchLinux
Mandriva,OpenSUSE,4,CentOS

If you want to extract out the data from 3rd column and calculate the sum of all the numbers then the approach to do this will be as follows:

Read the first line of the CSV file and extract out the 3rd column. Assign the extracted value to a scalar variable and then read the next line, extract out the value of 3rd column, add it to the scalar variable which contains the first value and store result in the same variable. Then read the 3rd line and so on…

Below is the Perl code for this purpose using split function.

  #!/usr/bin/perl
  use strict;
  use warnings;
 
  my $filename = $ARGV[0];
  my $total = 0;
 
  open(my $FH, '<', $filename) or die "Couldn't open $filename $!";
 
  while (my $fetchline = <$FH>){
    chomp $fetchline;
    my @elements = split "," , $fetchline;
    $total += $elements[2];
  }
 
  print "$total\n";

Save the file as text.plx and if the CSV file name is filename.csv then you may run the code as

$ perl text.plx filename.csv

In this case we get 10 (1+2+3+4) as output.

In this article we used split function to extract out the columns from CSV file using comma (,) as separator. But this code is error prone because of 2 reasons.

Reason 1: If the filename.csv contains something like

Linux,Unix,1,Ubuntu
RHEL,Kubuntu,2,Fedora
Mint,"Puppy,Linux",3,ArchLinux
Mandriva,OpenSUSE,4,CentOS

Notice the 3rd row. The 2nd field of it has a comma “Puppy,Linux”. If we use the same approach of splitting fields by comma separator then it will throw an error because split function will split out Puppy and Linux since they are separated by a comma. Then it will do something like 1+2+Linux+4 which is wrong.

Reason 2: If the filename.csv contains something like

Linux,Unix,1,Ubuntu
RHEL,Kubuntu,2,Fedora
Mint,"Puppy,
Linux",3,ArchLinux
Mandriva,OpenSUSE,4,CentOS

Here the 3rd row contains multi-line fields. When the above code will be used it will treat the CSV file as it contains 5 lines instead of 4.

So we can conclude that if the CSV file is in good format AND if it does not contain field separators within quote AND if there are no multi-line fields in CSV file then our code (which we have written above) will work perfectly and will give correct result. Take use of Text::CSV_XS module which is used for reading and writing CSV files, it will deal with those error prone situations very efficiently.

How to install rpm package on linux ?

install rpm package

install rpm package

RPM was the most famous way of installing packages on a Red Hat based linux box, nowadays other distros are also using RPM to support their software, Suse and Mandriva fall in the zone. RPM stands for Red Hat Package Manager. RPM lets you install a package, upgrade and uninstall the same. It also allows a querying functionality which lets one know about already packages installed on the system.

RPM packages are those with a .rpm extension. RPM packages contains the actual software to be installed and other files that are needed to carry out its installation. RPM packages are distributed distro specifically. Any attempt to install one distro-specific package on another distro might result in undesired consequences. You have been warned !

Graphical installation tools install a package in few clicks but here discussed is old school command line method which has its own legacy.

From here on, you need to have root’s privileges.

To install rpm package, rpm command is used along with -i option which clearly stands for “install”:

rpm -i package.rpm

Packages are subjected to regular upgradations, to update an installed rpm package, engage following code.

rpm -U package.rpm

On a fine day, everything will go well and the rpm package will be installed on your system ready to be launched from command line. Installed software, rather than creating its own directory, goes into pre-created linux directories. Executable files find their path to specific bin directories. To shoot the program from command line, the whole path is not needed just the program name is sufficient.

If things go south during installation, it can be very irritating. Most famous error that comes up is due to failed dependency. Dependencies are the packages that are needed for installing package to mover further in installation process. They are just software that are needed for proper functioning of the package being installed. While installing, the database is checked for needed packages, if not present, it stops the installation process due to failed dependency.

To rectify, one way is to install/upgrade those missing packages (don’t worry it will tell you which packages are not present ) then continue with the current installation. Some times, this error is as meaningless as ‘ay’ in ‘okay’. An entry is made in database only when the package is installed via rpm. If any other method is used, the entry is not made and despite presence of files, “failed dependency” error shows up.

When you are sure that the needed files are already on your system, you can skip the dependency check and install rpm package anyway.

rpm -i package.rpm --nodeps –nodeps option installs it without checking for dependency.

If you try to fool your system by going with –nodeps right away, the program won’t work at all.

Removing the rpm package

rpm command along with -e option is used to remove a package installed with rpm.

rpm -e package.rpm -e is for “erase”

Note: While installing rpm package we included the whole package name but at the time of removing- only the program name can be used ignoring the extension.

rpm -e package This code works the same as above.

It removes all the files related to that package and also deletes the entry from database. Manually removing individual files will not affect database which can lead to dependency errors in future.

Making a query to database:

As mentioned earlier in the very article, the database knows what packages are installed on the system. The database can be queried to check if certain package exists on the database. For a single package query, command goes like:

rpm -q package

If the package is present, it returns the version of package installed, else, it says the package is not installed.

To list all the rpm packages installed on the system, run :

rpm -qa

To check for certain package in the list, “grep” can be used. We will cover that later, stay tuned.

How to find the CPU information from terminal in Linux?

lscpu output

You can use lscpu command or can view the file /proc/cpuinfo to get the CPU information from terminal.

Man page description of lscpu command:

lscpu gathers CPU architecture information like number of CPUs, threads, cores, sockets, NUMA nodes, information about CPU caches, CPU family, model and stepping from sysfs and /proc/cpuinfo, and prints it in human-readable format. Alternatively, it can print out in parsable format including how different caches are shared by different CPUs, which can also be fed to other programs.

$ lscpu

or

$ cat /proc/cpuinfo

Output: