Abstract: This text introduces ancient Unix systems setup with a special emphasis on the terminal subsystem. Based on that, application support for multiple terminal types is discussed before we switch over to the X Window System. We show basic concepts, and how it still doesn't render the traditional Unix concepts outdated.

Introduction

In NetBSD, the wscons console driver can either emulate a vt100 or a sun display. I was asked by a novice user why there was no vt220 offered, and what's the difference between the terminal type emulated by wscons and the terminal type set in /etc/tty was. I realized that there was a lack of background information that I'd like to give here.

Starting with a description of the hardware setup the Unix operating system was designed for, the article moves to the support that the system offers applications to talk to widely different hardware and how information on the hardware can be determined and set. Then a jump is made to the modern world, explaining how the X Window System works in comparison to the terminal based I/O of traditional Unix systems, giving information of the basic concepts of X. After that, the class of terminal emulators as much-used applications is examined to display that the ancient Unix terminal system is still used even with in the world of modern desktop systems.

Examples in this text use the NetBSD multiplatform operating system, but the concepts are common to all Unix systems, ranging from PDS/Cadmus that the author made his first contact with Unix on to modern Unix versions like NetBSD, Solaris and Linux.

Ancient Unix Systems

Back in the stone ages of Unix, computer systems consisted of a mainframe, a big box of blinking lights which had memory, mass storage and computing units, and that run processes started by users or operators. As the hardware was very expensive, the systems were used as true multiuser systems, with many people interacting with the system at the same time. What it usually didn't have - unlike today's Unix workstations - was a fixed monitor and keyboard. Instead, issuing commands to the machine and retrieving output was done over serial lines, using teletypers first, and CRT (cathode ray tube) terminals later. Teletypers - that's where the "ttys" in Unix come from - are electronic typewriters that send keys pressed over the serial line to the host, and replies were sent back to the teletyper char by char over the serial line, with the built-in printer putting the reply on paper, much like a typewriter. For a picture of such a typewriter, see the lower left corner of the image at http://www.freeunix.de/.

* Early support of input devices

Some typewriters only knew capital chars, while others were able to input and print both cases. A traditional way that's rarely found in today's Unix flavours was the way to tell the system to use or not use mixed case: if you entered your login name in all lower case chars, the system assumed your terminal was capable of doing both upper and lower case. But if you entered your login name in all UPPER CASE chars (or your teletyper just sent everything you typed in upper case), the system assumed your terminal only can do upper case, and printed (real) upper chars with a backslash prepended in it's reply. This behaviour is still documented in the 4.4BSD's chapter 11 of the UNIX User's Supplementary Documents (USD), the "edit" tutorial (see /usr/share/doc/usd/11.edit).

Image #1: (click to enlarge!)

* How a terminal works

More advanced typewriters offered changing from black to red ribbon, bold face was available everywhere by overstriking - printing a character, moving the printer-head back (by printing a "backspace" character), and then printing it again on the same spot, resulting in a bold look. Not too long after that, the "printer" part of the teletypes was replaced with a cathode ray tube. The CRT displayed characters by moving an electron beam on a fluorescent screen of glass with a phosphorous coat, which gave them a greenish look that you can still see in movies today. The combination of CRT and keyboard is now widely known as "(serial) terminal". The functionality is the same as for the teletyper - keypresses get sent to the processing host over the serial line, and replies are sent back char by char, and the terminal displays them. The terminal itself has no processing capabilities, and is therefore often described as "dumb" terminal. When the host sent a "f", an "o" and another "o", the screen said "foo", with the cursor blinking after the second "o", waiting for further output from the host. The character set was 7bit ASCII, which is also still in widespread use in the Unix world today, and which are only slowly superceded by various competing standards like ISO 8859-1, Unicode and whatnot.

* Command sets

Staying at the blinking cursor after our "foo" a bit longer, the host had to send a command char that said "move to the next line", often even followed by another command char asking it to "move to the start of line", depending on the terminal's setting . These control characters are usually known as "carriage return" (CR) and "line feed" (LF). Command chars were originally encoded as ASCII values below 32, see "man ascii" for your nearest ASCII table. Other control characters for flow control etc. were used, and quickly it became obvious that the 32 chars were not enough. Command sets that consisted of several chars were used next, which often had a control sequence introducer (CSI), followed by one or more command bytes (e.g. "move cursor") and data bytes (e.g. row and column). What made this worse was that each manufacturer of terminals started to invent his own set of command set, that were not compatible with each other. Representatives of the more prominent ones are the DEC (now Compaq) VT100/VT220 escape sequences (named so because they had an Escape char, ASCII 27, as first char), ANSI codes and many others.

Image #2: (click to enlarge!)

* Handling many different terminals in applications

As Unix machines were used as real multiuser machines, it often happened that terminals from different vendors were hooked up using the standard serial protocol, but the applications that wanted to use the different terminals' special capabilities had to handle these differences in some way. Compiling hardcoded support for a certain terminal type was not an option as it meant adding a number of programs for each new terminal type - a maintenance nightmare. A different approach was made by putting a "translation table" in between, that defined common operations, and a mapping to the vendor specific command sets. The translation table is part of the hosts's operating system. When an application running on the Unix host wanted to e.g. clear a terminal's screen or move the cursor, it looked up the terminal-specific command via a system or library call, and then sent that string to the terminal. There's no need in the application to keep terminal-specific code, and if a new terminal gets added, only the translation table needs to be adjusted for each operation. A big win!

Image #3: (click to enlarge!)

In today's world there are two libraries available for doing these mappings of terminal attributes. One is the terminfo library found on many System V and Linux systems, the other is the BSD-originated termcap library which stores all the translation information all in a single file, usually /usr/share/misc/termcap. Be sure to have a look!

* $TERM and ttys

There's one question left. If a users logs in from one terminal, and starts an application, how does the system know what terminal type and with that what command sequences are needed? The answer is, the system just knows, because each terminal is hooked up to a certain serial port (/dev/ttySomething), and the /etc/ttys file tells which type the terminal is (Footnote: This is for the BSD world; the System V world uses /etc/inittab for a similar purpose):

Image #4: (click to enlarge!)

This is from a NetBSD 1.5/sparc64 system, with ttya and ttyb being the machine's two built in serial ports, and the "console" line being used for all system input and output. The first column specifies the terminal line, and 3rd column, "type", tells the type of the terminal, "vt220" for everything but the system console here. Have a look at your ttys(5) manpage for all the details.

The terminal type from column 3 is passed through the login process to the user's login shell, and by that to any applications that the user starts. The application then uses termcap to lookup the specific command sequences they may need. When someone logs in from a terminal hooked up to ttya, it is assumed a VT220 (or compatible) terminal is used.

The information on the terminal type is kept in the TERM environment variable, which can be set to a different value if the user needs to do so for some reason. In addition, the terminal line one's logged in via can be determined with the tty(1) command:

Image #5: (click to enlarge!)

In the above example, I'm logged in on the ttya serial port. I do not have a (hardware) terminal, but use another computer that's connected to ttya via a serial line, and that has a terminal emulation program running. The terminal program talks to the Sun over the serial interface, displays the characters it sends, and interprets any special command sequences. As i know from my terminal program's handbook, it does not emulate a vt220 properly but works fine for vt100 commands. The TERM variable is set here to reflect this in order to make applications behave properly.

Back to modern life

Modern workstation nowadays don't have seperate hardware for the computer itself and one or more terminals. Instead, (usually) a CRT is directly attached to a gfx board, and input is done with keyboard and mouse, thus doing user input and output no longer over a slow serial line.

* Console Drivers & Virtual Consoles

The model described above with the terminal being able to handle a certain set of commands still applies to modern console drivers, though. The console driver is a piece of software that runs as part of the kernel, and that handles input from the keyboard and output to the display. It passes data via a (software) terminal device, using the same interface inside the kernel as a serial device would do, and that of course needs to be configured in /etc/ttys. Output to the console is interpreted, and just as ancient terminals, today's consoles handle a wide range of character sets, among them various VT100/VT220 variants, ANSI, or more specialized ones like the FreeBSD "cons25" console.

Most popular PC operating systems further offer a mechanism called "virtual consoles" which allow more than one "virtual" consoles, among which the user sitting in front of the machine can switch between, usually by pressing the Control+Alt and a function key (F1, ...). The system handles these as seperate lines, and each of them must be configured in /etc/ttys.

The NetBSD operating system, which uses wscons as the console driver on many ports, offers a way to configure which terminal type it should emulate, with "vt100" being used on most ports, and the sun3, sparc and sparc64 port using "sun".

* Graphics

So far we have always talked about text/console data being passed between computer and terminal, and being displayed on the console. The question how graphics works still stands unanswered so far.

Let's go back in time a bit again for this. When terminal vendors added new capabilities to their hardware, one of the new features was to not only display 80x24 textual information, but also bitmap data of some sort. Of course the command sequences for this were as non-standard as they could be. There was no application-level standard like terminfo or termcap that applications could use to make device independent use of these graphic terminals' capabilities.

Time evolved, machines were built which had main unit and gfx hardware connected with faster connection than serial line, and at the same time networking several main (computer) units to local area networks for data exchange and communication started to get used.

A group of researchers at MIT were exploring modern computer capabilities, and came up with a graphical user interface based on similar parts as the classic setup between computers and terminals was. It included an application process that was issuing commands like "clear screen" or "open window". The commands were transported to a "presentation server" that decoded the commands, and did whatever steps were necessary to e.g. clear a bitmap or draw a rectangular area with a border. The "presentation server" usually got it's commands from a local or remote computer system, and it knew how to talk to gfx board (for output), keyboard and mouse (for input). Events generated by the user operating mouse and keyboard were sent back to the application process for interpretation and action.

Image #6: (click to enlarge!)

* Calling Names

Are you familiar with the scheme described so far? No? Maybe it will when we call the components by their names:

The window system described above is the X Window System. The application process is usually known as "X client", the "presentation server" handling device access is the "X server". Communication between X client and server is done using the X Window System Protocol, which is based on the TCP/IP stack today.

Using Unix' capabilities to run multiple processes, one machine running X clients can issue display requests to one or more X servers running either on the local machine using the local gfx board, keyboard and mouse, or using a X server that's running on a remote machine, and that thus uses a remote machine's gfx card, mouse and keyboard.

And just as vendors started to sell VT220 compatible ASCII terminals when that was mostly a standard, there were things called "X terminals". They were equivalent to their text-only cousins which only know how to handle input and output, without any capabilities to run actual application code themselves. Usually, X terminals consist of monitor, keyboard and mouse and a small box that ran (only) the X server process. The X terminal's X server knew how to talk to keyboard mouse and graphics card, and passed all input and output to an X client connecting to the X server.

An alternative to X terminals are today's Unix workstations, which have computer, gfx and IO-hardware all built into e.g. a PC or a traditional workstation machine. The workstation then runs it's local instance of Unix, including the X server process to talk to the hardware and a number of X client applications, all in one case.

* An X Terminal-Emulation Application - xterm

So far, we have talked about the classic terminal concept of unix, and about the X window system, which follows similar principles. Now that we have the X window system and everyone can turn his PC into a workstation, the base for modern user interfaces like GNOME and KDE is there. We can forget about all that old cruft about terminals and passing single chars back and forth, escape sequences and all this. Whee!

But... can we really? Let's see how communication with the Unix system works today.

Most (real :-) interaction with the operating system today still happens via a command line interface, which is used to type commands, start processes, and view their output, either via X or on the command line environment that the command was started from. That "command line environment" is usually a shell running either on the system's console, one of it's virtual consoles as described above, or a terminal window on the X desktop.

If the system console or virtual consoles are used, it's quite obvious that the traditional terminal handling is still needed, as it is the base for input and output done by all commands. This applies to both line oriented programs like ls(1) or cat(1) as well as so-called "screen oriented" programs like the "vi"sual editor vi(1), the top(1) process monitor, etc., which still use system libraries like termcap or terminfo to address the console's capabilities.

But we just agreed that noone needs consoles any more, and that we have the X Window System to interact with the system. There, we open up a terminal window, using programs like xterm, or one of it's newer cousins like the KDE "konsole" or GNOME's "gnome-terminal". The programs' names make it quite obvious what they do, and if we take a closer look at how things work, it will be even more obvious where they got their names from.

In the following example, we will use "xterm", but others do about the same. When you are using the X Window System and start "xterm", a rectangular window opens, with a "shell" command line interpreter running. You can type commands into the shell window, and output from the shell commands will be displayed in the xterm window. What's happening in more detail is, that the X server takes the keyboard events, and sends them to the xterm process. The xterm process has a shell process connected to it. The ASCII presentation of the keys pressed are sent from the xterm process to the shell process, just like it happens for both the systems or virtual consoles and ancient, dumb serial hardware terminals. The shell process reads it's input via it's standard input, processes the commands given, and output is sent back to the xterm via the standard output channel. What happens if you run a program that needs to e.g. clear the screen or move the cursor to a certain position? Exactly the same as in the cases described above, the application knows which terminal type it's connected to, it uses the termcap or terminfo libraries to query that terminal's capabilities, and then sends the necessary escape sequences to the xterm. The xterm process in turn recognizes the command sequences, and does not simply echo the command sequence's characters but knows how to interpret them, and does not send character-generating events to the X server for display, but does what the command sequence says - a "clear screen" sequence is sent to the X server as a command to draw a plain white rectangle, and another small black rectangle in the upper left corner, representing the cursor.

Image #7: (click to enlarge!)

Two details are worth having another look here. First, communication between the xterm and shell process. This is done via Unix' ordinary terminal input/output routines, no special mechanism was invented in the X Window System for user interaction between the X server and an application running in a command line environment, with the xterm program "translating" between the two environments. This means that the terminal handling that the Unix operating system has had for more than 30 years now is still useful even in modern environments.

The other detail that's interresting here is the interaction between the xterm process and the X server. The xterm process is just like any graphical application, it sends drawing requests to the X server - a big white filled rectangle, a small black filled rectangle, commands to move parts of the on-screen image around (i.e. do scrolling), of course use the X font mechanism to use a certain font and display chars from it on the X "presentation" server. Although the xterm terminal program does not provide special command sequences so set e.g. the font of the terminal used, xterm allows changing the font it asks the X server to use, and a number of other things. To try this, hold down the control-key and press one of the three mouse buttons at the same time - you'll see that the popup-menu of ctrl+left mouse allows selecting the font that the xterm process asks the X server for.

The Circle Closes

As you see from the previous discussion, graphical user interfaces did not supersede the traditional concepts of the Unix operating system, but interact with them so that existing applications can be used in window environments without any migration effort. The concept of terminals used in hardware once and mostly in software nowadays continues to live on, and the circle closes, giving both "old" and "new" technology an equal right to exist. They do not compete but play together perfectly well.

It is because of the simplicity and usefullnes of the basic Unix ideas that allowed the system to grow and gain maturity over all these years.

I'd like to close this rather lengthy overview with mentioning that I left some things like networking out deliberately - the (pseudo) terminals that are used in communication between xterm processes and their shells are also involved in network communication for things like rsh and ssh, and to pass data between different machines as if the data source and destination were processes on the same machine. Also, the description of the X Window System left out *many* of the concepts of the window system. Window managers and session management are only two of many things that come to mind, but going into detail on these is something left for another time. For now, I hope you had as much fun reading this as I had writing. Enjoy!

* Recommended Reading

Richard Stevens: Advanced Programming the Unix Environment; Addison Wesley, 1992. ISBN 0-201-56317-7.