39.5 Why Is the System So Slow?
To a user, performance means: "How much time does it take to run my job?" For a system manager, this question is much too simple: a user's job may take a long time to execute because it is badly written or because it doesn't really use the computer appropriately. Furthermore, a system manager must optimize performance for all system users - which is much more complicated than optimizing performance for a single user. Here are some of the things that affect performance.
The UNIX utility /bin/time ( 39.2 ) reports the amount of time required to execute a program, breaking down the total time into several important components. For example, consider the report below:
This report shows that the program ran in roughly 4.8 seconds. This is the elapsed or wallclock time: it is the actual time that the program runs as it would be measured by a user sitting at the terminal with a stopwatch. The amount of time that the system spent working on your program is much smaller. It spent 0.5 seconds of user time , which is time spent executing code in the user state, and about 0.7 seconds of system time , which is time spent in the system state (i.e., time spent executing UNIX system code) on behalf of the user. The total amount of CPU time (actual execution time on the main processor) was only 1.2 seconds, or only one-quarter of the elapsed time. 
Where did the rest of the time go? Some time was spent performing I/O (text input/output) operations, which /bin/time doesn't report. Handling I/O requires some computation, which is attributed to system time. But time that is spent by disk drives, network interfaces, terminal controllers, or other hardware isn't accounted for; most of the time was spent running jobs on behalf of other users. This entails its own performance overhead (context-switch time, swapping time, etc.).
Many different components contribute to a program's total running time. When you understand the roles these components play, you will understand the problem. Here is a summary of the different components:
If you spend most of your time running standard utilities and commercial applications, you can't do much about user-state or system-state time. To make a significant dent in these, you have to rewrite the program. But you can do a lot to improve your memory and I/O performance, and you can do a lot to run your big applications more efficiently.
Keyboard response is an extremely important issue to users, although it really doesn't contribute to a program's execution time. If there is a noticeable gap between the time when a user types a character and the time when the system echoes that character, the user will think performance is bad, regardless of how much time it takes to run a job. In order to prevent terminal buffers from overflowing and losing characters, most UNIX systems give terminal drivers ( 42.1 ) very high priority. As a side effect, the high priority of terminals means that keyboard response should be bad only under exceptionally high loads. If you are accessing a remote system across a network, however, network delays can cause poor keyboard response. Network performance is an extremely complex issue.
- from O'Reilly & Associates' System Performance Tuning , Chapter 1