Linux Tutorial - 12. Process Management

 

Process Management!

The wheels are in motion

Introduction

Linux in general is a fairly stable system. Occasionally, things do go wrong however and sometimes we also wish to tweak the running of the system to better suit our needs. In this section we will take a brief look at how we may manage programs, or processes on a Linux system.

So what are they?

A program is a series of instructions that tell the computer what to do. When we run a program, those instructions are copied into memory and space is allocated for variables and other stuff required to manage its execution. This running instance of a program is called a process and it's processes which we manage.

What is Currently Running?

Linux, like most modern OS's is a multitasking operating system. This means that many processes can be running at the same time. As well as the processes we are running, there may be other users on the system also running stuff and the OS itself will usually also be running various processes which it uses to manage everything in general. If we would like to get a snapshot of what is currently happening on the system we may use a program called top.

top

Below is a simplified version of what you should see when you run this program.

1. top
2. Tasks: 174 total, 3 running, 171 sleeping, 0 stopped
3. KiB Mem: 4050604 total, 3114428 used, 936176 free
4. Kib Swap: 2104476 total, 18132 used, 2086344 free
5.  
6.  PID USER %CPU %MEM COMMAND
7. 6978 vamsi 3.0  21.2 firefox
8.   11 root 0.3   0.0 rcu_preempt
9. 6601 vamsi 2.0   2.4 kwin
10. ...

Let's break it down:

• Line 2 Tasks is just another name for processes. It's typical to have quite a few processes running on your system at any given time. Most of them will be system processes. Many of them will typically be sleeping. This is ok. It just means they are waiting until a particular event occurs, which they will then act upon.
• Line 3 This is a breakdown of working memory (RAM). Don't worry if a large amount of your memory is used. Linux keeps recently used programs in memory to speed up performance if they are run again. If another process needs that memory, they can easily be cleared to accommodate this.
• Line 4 This is a breakdown of Virtual memory on your system. If a large amount of this is in use, you may want to consider increasing it's size. For most people with most modern systems having gigabytes of RAM you shouldn't experience any issues here.
• Lines 6 - 10 Finally is a listing of the most resource intensive processes on the system (in order of resource usage). This list will update in real time and so is interesting to watch to get an idea of what is happening on your system. The two important columns to consider are memory and CPU usage. If either of these is high for a particular process over a period of time, it may be worth looking into why this is so.The USER column shows who owns the process and the PID column identifies a process's Process ID which is a unique identifier for that process.

Top will give you a realtime view of the system and only show the number of processes which will fit on the screen. Another program to look at processes is called ps which stands for processes. In it's normal usage it will show you just the processes running in your current terminal (which is usually not very much). If we add the argument aux then it will show a complete system view which is a bit more helpful.

ps [aux]

It does give quite a bit of output so people usually pipe the output to grep to filter out just the data they are after. We will see in the next bit an example of this.

Killing a Crashed Process

It doesn't happen often, but when a program crashes, it can be quite annoying. Let's say we've got our browser running and all of a sudden it locks up. You try and close the window but nothing happens, it has become completely unresponsive. No worries, we can easily kill Firefox and then reopen it. To start off we need to identify the process id.

1. ps aux | grep 'firefox'
2. vamsi 6978 8.8 23.5 2344096 945452 ? Sl 08:03 49:53 /usr/lib64/firefox/firefox

It is the number next to the owner of the process that is the PID (Process ID). We will use this to identify which process to kill. To do so we use a program which is appropriately called kill.

kill [signal] <PID>

1. kill 6978
2. ps aux | grep 'firefox'
3. vamsi 6978 8.8 23.5 2344096 945452 ? Sl 08:03 49:53 /usr/lib64/firefox/firefox

Sometimes you are lucky and just running kill normally will get the process to stop and exit. When you do this kill sends the default signal ( 1 ) to the process which effectively asks the process nicely to quit. We always try this option first as a clean quit is the best option. Sometimes this does not work however. In the example above we ran ps again and saw that the process was still running. No worries, we can run kill again but this time supply a signal of 9 which effectively means, go in with a sledge hammer and make sure the process is well and truly gone.

1. kill -9 6978
2. ps aux | grep 'firefox'

Normal users may only kill processes which they are the owner for. The root user on the system may kill anyones processes.

My Desktop has locked up

On rare occassions, when a process crashes and locks up, it can lock up the entire desktop. If this happens there is still hope.

Linux actually runs several virtual consoles. Most of the time we only see console 7 which is the GUI but we can easily get to the others. If the GUI has locked up, and we are in luck, we can get to another console and kill the offending process from there. To switch between consoles you use the keyboard sequence CTRL + ALT + F<Console>. So CTRL + ALT F2 will get you to a console (if all goes well) where you can run the commands as above to identify process ids and kill them. Then CTRL + ALT F7 will get you back to the GUI to see if it has been fixed. The general approach is to keep killing processes until the lock up is fixed. Normally you can look for tell tale signs such as high CPU or Memory usage and start with those processes first. Sometimes this approach works, sometimes it doesn't and you need to restart the computer. Just depends how lucky you are.

Foreground and Background Jobs

You probably won't need to do too much with foreground and background jobs but it's worth knowing about them just for those rare occassions. When we run a program normally (like we have been doing so far) they are run in the foreground. Most of them run to completion in a fraction of a second as well. Maybe we wish to start a process that will take a bit of time and will happily do it's thing without intervention from us (processing a very large text file or compiling a program for instance). What we can do is run the program in the background and then we can continue working. We'll demonstrate this with a program called sleep. All sleep does is wait a given number of seconds and then quit. We can also use a program called jobs which lists currently running background jobs for us.

jobs

1. sleep 5

If you run the above example yourself, you will notice that the terminal waits 5 seconds before presenting you with a prompt again. Now if we run the same command but instead put an ampersand ( & ) at the end of the command then we are telling the terminal to run this process in the background.

1. sleep 5 &
2. [1] 21634
5. [1]+ Done sleep 5

This time you will notice that it assigns the process a job number, and tells us what that number is, and gives us the prompt back straight away. We can continue working while the process runs in the background. If you wait 5 seconds or so and then hit ENTER you will see a message come up telling you the job has completed.

We can move jobs between the foreground and background as well. If you press CTRL + z then the currently running foreground process will be paused and moved into the background. We can then use a program called fg which stands for foreground to bring background processes into the foreground.

fg <job number>

1. sleep 15 &
2. [1] 21637
3. sleep 10
4. (you press CTRL + z, notice the prompt comes back.)
5. jobs
6. [1]- Running sleep 15 &
7. [2]+ Stopped sleep 10
8. fg 2
9. [1] Done sleep 15

CTRL + z is used in Windows but for the purpose of running the undo command. It is not uncommon for people coming from the Windows world to accidentally hit the key combo (especially in the editor VI for instance) and wonder why their program just dissappeared and the prompt returned. If you do this, don't worry, you can use jobs to identify which job it has been assigned to and then fg to bring it back and continue working.

Summary

top

View real-time data about processes running on the system.

ps

Get a listing of processes running on the system.

kill

End the running of a process.

jobs

Display a list of current jobs running in the background.

fg

Move a background process into the foreground.

ctrl + z

Pause the current foreground process and move it into the background.

Control

We have quite a bit of control over the running of our programs.

Activities

Time for some fun:

• First off, start a few programs in your desktop. Then use ps to identify their PID and kill them.
• Now see if you can do the same, but switch to another virtual console first.
• Finally, play about with the command sleep and moving processes between the foreground and background.

Linux Tutorial - 11. Piping and Redirection

 

Piping and Redirection!

Keeping the data flowing

Introduction

Learn how easy it is to use piping and redirection to create powerful workflows that will automate your work, saving you time and effort.

In the previous two sections we looked at a collection of filters that would manipulate data for us. In this section we will see how we may join them together to do more powerful data manipulation.

There is a bit of reading involved in this section. Even though the mechanisms and their use are quite simple, it is important to understand various characteristics about their behaviour if you wish to use them effectively.

So what are they?

Every program we run on the command line automatically has three data streams connected to it.

• STDIN (0) - Standard input (data fed into the program)
• STDOUT (1) - Standard output (data printed by the program, defaults to the terminal)
• STDERR (2) - Standard error (for error messages, also defaults to the terminal)

Piping and redirection is the means by which we may connect these streams between programs and files to direct data in interesting and useful ways.

We'll demonstrate piping and redirection below with several examples but these mechanisms will work with every program on the command line, not just the ones we have used in the examples.

Redirecting to a File

Normally, we will get our output on the screen, which is convenient most of the time, but sometimes we may wish to save it into a file to keep as a record, feed into another system, or send to someone else. The greater than operator ( > ) indicates to the command line that we wish the programs output (or whatever it sends to STDOUT) to be saved in a file instead of printed to the screen. Let's see an example.

1. ls
2. barry.txt bob example.png firstfile foo1 video.mpeg
3. ls > myoutput
4. ls
5. barry.txt bob example.png firstfile foo1 myoutput video.mpeg
6. cat myoutput
7. barry.txt
8. bob
9. example.png
10. firstfile
11. foo1
12. myoutput
13. video.mpeg

Let's break it down:

• Line 1 Let's start off by seeing what's in our current directory.
• Line 3 Now we'll run the same command but this time we use the > to tell the terminal to save the output into the file myoutput. You'll notice that we don't need to create the file before saving to it. The terminal will create it automatically if it does not exist.
• Line 4 As you can see, our new file has been created.
• Line 6 Let's have a look at what was saved in there.

Some Observations

You'll notice that in the above example, the output saved in the file was one file per line instead of all across one line when printed to the screen. The reason for this is that the screen is a known width and the program can format its output to suit that. When we are redirecting, it may be to a file, or it could be somewhere else, so the safest option is to format it as one entry per line. This also allows us to easier manipulate that data later on as we'll see further down the page.

When piping and redirecting, the actual data will always be the same, but the formatting of that data may be slightly different to what is normally printed to the screen. Keep this in mind.

You'll also notice that the file we created to save the data into is also in our listing. The way the mechanism works, the file is created first (if it does not exist already) and then the program is run and output saved into the file.

Saving to an Existing File

If we redirect to a file which does not exist, it will be created automatically for us. If we save into a file which already exists, however, then it's contents will be cleared, then the new output saved to it.

1. cat myoutput
2. barry.txt
3. bob
4. example.png
5. firstfile
6. foo1
7. myoutput
8. video.mpeg
9. wc -l barry.txt > myoutput
10. cat myoutput
11. 7 barry.txt

We can instead get the new data to be appended to the file by using the double greater than operator ( >> ).

1. cat myoutput
2. 7 barry.txt
3. ls >> myoutput
4. cat myoutput
5. 7 barry.txt
6. barry.txt
7. bob
8. example.png
9. firstfile
10. foo1
11. myoutput
12. video.mpeg

Redirecting from a File

If we use the less than operator ( < ) then we can send data the other way. We will read data from the file and feed it into the program via it's STDIN stream.

1. wc -l myoutput
2. 8 myoutput
3. wc -l < myoutput
4. 8

A lot of programs (as we've seen in previous sections) allow us to supply a file as a command line argument and it will read and process the contents of that file. Given this, you may be asking why we would need to use this operator. The above example illustrates a subtle but useful difference. You'll notice that when we ran wc supplying the file to process as a command line argument, the output from the program included the name of the file that was processed. When we ran it redirecting the contents of the file into wc the file name was not printed. This is because whenever we use redirection or piping, the data is sent anonymously. So in the above example, wc recieved some content to process, but it has no knowledge of where it came from so it may not print this information. As a result, this mechanism is often used in order to get ancillary data (which may not be required) to not be printed.

We may easily combine the two forms of redirection we have seen so far into a single command as seen in the example below.

1. wc -l < barry.txt > myoutput
2. cat myoutput
3. 7

Redirecting STDERR

Now let's look at the third stream which is Standard Error or STDERR. The three streams actually have numbers associated with them (in brackets in the list at the top of the page). STDERR is stream number 2 and we may use these numbers to identify the streams. If we place a number before the > operator then it will redirect that stream (if we don't use a number, like we have been doing so far, then it defaults to stream 1).

1. ls -l video.mpg blah.foo
2. ls: cannot access blah.foo: No such file or directory
3. -rwxr--r-- 1 vamsi users 6 May 16 09:14 video.mpg
4. ls -l video.mpg blah.foo 2> errors.txt
5. -rwxr--r-- 1 vamsi users 6 May 16 09:14 video.mpg
6. cat errors.txt
7. ls: cannot access blah.foo: No such file or directory

Maybe we wish to save both normal output and error messages into a single file. This can be done by redirecting the STDERR stream to the STDOUT stream and redirecting STDOUT to a file. We redirect to a file first then redirect the error stream. We identify the redirection to a stream by placing an & in front of the stream number (otherwise it would redirect to a file called 1).

1. ls -l video.mpg blah.foo > myoutput 2>&1
2. cat myoutput
3. ls: cannot access blah.foo: No such file or directory
4. -rwxr--r-- 1 vamsi users 6 May 16 09:14 video.mpg
5. 
   

Piping

So far we've dealt with sending data to and from files. Now we'll take a look at a mechanism for sending data from one program to another. It's called piping and the operator we use is ( | ) (found above the backslash ( \ ) key on most keyboards). What this operator does is feed the output from the program on the left as input to the program on the right. In the example below we will list only the first 3 files in the directory.

1. ls
2. barry.txt bob example.png firstfile foo1 myoutput video.mpeg
3. ls | head -3
4. barry.txt
5. bob
6. example.png

We may pipe as many programs together as we like. In the below example we have then piped the output to tail so as to get only the third file.

1. ls | head -3 | tail -1
2. example.png

Any command line arguments we supply for a program must be next to that program.

I often find people try and write their pipes all out in one go and make a mistake somewhere along the line. They then think it is in one point but in fact it is another point. They waste a lot of time trying to fix a problem that is not there while not seeing the problem that is there. If you build your pipes up incrementally then you won't fall into this trap. Run the first program and make sure it provides the output you were expecting. Then add the second program and check again before adding the third and so on. This will save you a lot of frustration.

You may combine pipes and redirection too.

1. ls | head -3 | tail -1 > myoutput
2. cat myoutput
3. example.png

More Examples

Below are some more examples to give an idea of the sorts of things you can do with piping. There are many things you can achieve with piping and these are just a few of them. With experience and a little creative thinking I'm sure you'll find many more ways to use piping to make your life easier.

All the programs used in the examples are programs we have seen before. I have used some command line arguments that we haven't covered yet however. Look up the relevant man pages to find out what they do. Also you can try the commands yourself, building up incrementally to see exactly what each step is doing.

In this example we are sorting the listing of a directory so that all the directories are listed first.

1. ls -l /etc | tail -n +2 | sort
2. drwxrwxr-x 3 nagios nagcmd 4096 Mar 29 08:52 nagios
3. drwxr-x--- 2 news news 4096 Jan 27 02:22 news
4. drwxr-x--- 2 root mysql 4096 Mar 6 22:39 mysql
5. ...

In this example we will feed the output of a program into the program less so that we can view it easier.

1. ls -l /etc | less
2. (Full screen of output you may scroll. Try it yourself to see.)

Identify all files in your home directory which the group has write permission for.

1. ls -l ~ | grep '^.....w'
2. drwxrwxr-x 3 vamsi users 4096 Jan 21 04:12 dropbox

Create a listing of every user which owns a file in a given directory as well as how many files and directories they own.

1. ls -l /projects/ghosttrail | tail -n +2 | sed 's/\s\s*/ /g' | cut -d ' ' -f 3 | sort | uniq -c
2. 8 anne
3. 34 harry
4. 37 tina
5. 18 vamsi

Summary

>

Save output to a file.

>>

Append output to a file.

<

Read input from a file.

2>

Redirect error messages.

|

Send the output from one program as input to another program.

Streams

Every program you may run on the command line has 3 streams, STDIN, STDOUT and STDERR.

Activities

Let's mangle some data:

• First off, experiment with saving output from various commands to a file. Overwrite the file and append to it as well. Make sure you are using both absolute and relative paths as you go.
• Now see if you can list only the 20th last file in the directory /etc.
• Finally, see if you can get a count of how many files and directories you have the execute permission for in your home directory.

Linux Tutorial - 10. Grep and Regular Expressions

 

Grep and Regular Expressions!

What the $[+*.

Introduction

Discover the power of grep and regular expressions with this easy to follow beginners tutorial with plenty of examples to guide you.

In the previous section we looked at a collection of filters that would manipulate data for us. In this section we will look at another filter which is quite powerful when combined with a concept called regular expressions or re's for short. Re's can be a little hard to get your head around at first so don't worry if this stuff is a little confusing. I find the best approach is to go over the material and experiment on the command line a little, then leave it for a day or 3, then come back and have another go. You will be surprised but it will start to make more sense the second time. Mastering re's just takes practice and time so don't give up.

So what are they?

Regular expressions are similar to the wildcards that we looked at in section 7. They allow us to create a pattern. They are a bit more powerful however. Re's are typically used to identify and manipulate specific pieces of data. eg. we may wish to identify every line which contains an email address or a url in a set of data.

Re's are used all over the place. We will be demonstrating them here with grep but many other programs use them (including sed and vi which you learned about in previous sections) and many programming languages make use of them too.

I'll give you an introduction to them here in this section but there is much more they can do. If you are interested then I highly recommend going through our regular expression tutorial which goes into more detail.

The characters used in regular expressions are the same as those used in wildcards. Their behaviour is slightly different however. A common mistake is to forget this and get their functions mixed up.

eGrep

egrep is a program which will search a given set of data and print every line which contains a given pattern. It is an extension of a program called grep. It's name is odd but based upon a command which did a similar function, in a text editor called ed. It has many command line options which modify it's behaviour so it's worth checking out it's man page. ie the -v option tells grep to instead print every line which does not match the pattern.

egrep [command line options] <pattern> [path]

In the examples below we will use a similar sample file as in the last section. It is included below as a reference.

1. cat mysampledata.txt
2. Fred apples 20
3. Susy oranges 5
4. Mark watermellons 12
5. Robert pears 4
6. Terry oranges 9
7. Lisa peaches 7
8. Susy oranges 12
9. Mark grapes 39
10. Anne mangoes 7
11. Greg pineapples 3
12. Oliver rockmellons 2
13. Betty limes 14

Let's say we wished to identify every line which contained the string mellon

1. egrep 'mellon' mysampledata.txt
2. Mark watermellons 12
3. Oliver rockmellons 2

The basic behaviour of egrep is that it will print the entire line for every line which contains a string of characters matching the given pattern. This is important to note, we are not searching for a word but a string of characters.

Also note that we included the pattern within quotes. This is not always required but it is safer to get in the habit of always using them. They are required if your pattern contains characters which have a special meaning on the command line.

Sometimes we want to know not only which lines matched but their line number as well.

1. egrep -n 'mellon' mysampledata.txt
2. 3:Mark watermellons 12
3. 11:Oliver rockmellons 2

Or maybe we are not interested in seeing the matched lines but wish to know how many lines did match.

1. egrep -c 'mellon' mysampledata.txt
2. 2

Learning Regular Expressions

The best way to learn regular expressions is to give the examples a try yourself, then modify them slightly to test your understanding. It is common to make mistakes in your patterns while you are learning. When this happens typically every line will be matched or no lines will be matched or some obscure set. Don't worry if this happens you haven't done any damage and you can easily go back and have another go. Remember you may hit the up arrow on your keyboard to get at your recent commands and also modify them so you don't need to retype the whole command each time.

If you're not getting the output you would like then here are some basic strategies.

• First off, check for typo's. If you're like me then you're prone to making them.
• Re read the content here. Maybe what you thought a particular operator did was slightly different to what it actually does and re reading you will notice a point you may have missed the first time.
• Break your pattern down into individual components and test each of these individually. This will help you to get a feel for which parts of the pattern is right and which parts you need to adjust.
• Examine your output. Your current pattern may not have worked the way you want but we can still learn from it. Looking at what we actually did match and using it to help understand what actually did happen will help us to work out what we should try changing to get closer to what we actually want.

Debuggex is an on-line tool that allows you to experiment with regular expressions and allows you to visualise their behaviour. It can be a good way to better understand how they work.

Regular Expression Overview

I will outline the basic building blocks of re's below then follow on with a set of examples to demonstrate their usage.

• . (dot) - a single character.
• ? - the preceding character matches 0 or 1 times only.
• * - the preceding character matches 0 or more times.
• + - the preceding character matches 1 or more times.
• {n} - the preceding character matches exactly n times.
• {n,m} - the preceding character matches at least n times and not more than m times.
• [agd] - the character is one of those included within the square brackets.
• [^agd] - the character is not one of those included within the square brackets.
• [c-f] - the dash within the square brackets operates as a range. In this case it means either the letters c, d, e or f.
• () - allows us to group several characters to behave as one.
• | (pipe symbol) - the logical OR operation.
• ^ - matches the beginning of the line.
• $ - matches the end of the line.

Some Examples

We'll start with something simple. Let's say we wish to identify any line with two or more vowels in a row. In the example below the multiplier {2,} applies to the preceding item which is the range.

1. egrep '[aeiou]{2,}' mysampledata.txt
2. Robert pears 4
3. Lisa peaches 7
4. Anne mangoes 7
5. Greg pineapples 3

How about any line with a 2 on it which is not the end of the line. In this example the multiplier + applies to the . which is any character.

1. egrep '2.+' mysampledata.txt
2. Fred apples 20

The number 2 as the last character on the line.

1. egrep '2$' mysampledata.txt
2. Mark watermellons 12
3. Susy oranges 12
4. Oliver rockmellons 2

And now each line which contains either 'is' or 'go' or 'or'.

1. egrep 'or|is|go' mysampledata.txt
2. Susy oranges 5
3. Terry oranges 9
4. Lisa peaches 7
5. Susy oranges 12
6. Anne mangoes 7

Maybe we wish to see orders for everyone who's name begins with A - K.

1. egrep '^[A-K]' mysampledata.txt
2. Fred apples 20
3. Anne mangoes 7
4. Greg pineapples 3
5. Betty limes 14

Summary

egrep

View lines of data which match a particular pattern.

Regular Expressions

A powerful way to identify particular pieces of information.

Activities

Let's identify some information.

• First off, you may want to make a file with data similar to our sample file.
• Now play with some of the examples we looked at above.
• Have a look at the man page for egrep and try atleast 2 of the command line options for them.