Unit10 Fault Tolerance and Security
Unit10 Fault Tolerance and Security
10.0. Introduction
As more and more of this information is stored in computer systems, the need to protect it is
becoming increasingly important Protecting this information against unauthorized usage is
therefore a major concern of all operating systems. Unfortunately, it is also becoming
increasingly difficult due to the widespread acceptance of system bloat as being a normal and
acceptable phenomenon. In the following sections we will look at a variety of issues concerned
with security and protection, some of which have analogies to real-world protection of
information on paper, but some of which are unique to computer systems. In this chapter we will
examine computer security as it applies to operating systems.
Some people use the terms “security” and “protection” interchangeably. Nevertheless, it is
frequently useful to make a distinction between the general problems involved in making sure
that files are not read or modified by unauthorized persons, which include technical,
administrative, legal, and political issues on the one hand, and the specific operating system
mechanisms used to provide security, on the other. To avoid confusion, we will use the term
security to refer to the overall problem, and the term protection mechanisms to refer to the
specific operating system mechanisms used to safeguard information in the computer. The
boundary between them is not well defined, however. First we will look at security to see what
the nature of the problem is. Later on in the chapter we will look at the protection mechanisms
and models available to help achieve security. Security has many facets. Three of the more
important ones are the nature of the threats, the nature of intruders, and accidental data loss. We
will now look at these in turn.
A fault is a malfunction, the cause of an error. It can be a hardware or a software fault. It can be
transient, intermittent, or permanent.A transient fault occurs once only. It is usually a hardware
fault, such as a random cosmic ray flipping a bit in memory. There is not much that can be done
about identifying the fault and repairing it, as it does not occur again. The emphasis is on
detecting it and recovering from it.An intermittent fault occurs again and again, at unpredictable
intervals. It is the most difficult type of fault to diagnose, as you never know when it will occur
again. It can be a failing hardware component or a bug in software.A permanent fault means that
something is broken and must be replaced. This can be a hardware component or a piece of code
which is not doing what it is supposed to do.
The first requirement for a fault-tolerant system is that it be able to detect a fault. This should be
done as soon as possible, as the longer it goesundetected the more errors will be caused, reducing
the chance of identifying the underlying fault.The general approach to this is to use redundancy.
Data faults can be detected by using information redundancy, such as check-bits or checksums.
They can indicate that a data item has been corrupted. A step up from this is to use error-
correcting codes. There is a greater overhead involved here, but they make it possible to recreate
the correct value.Another possibility is to use time redundancy by repeating the operation. For
example, two different copies of a file can be read and compared. If they do not match exactly,
we know that an error has occurred. This more than doubles the time involved. Sometimes it is
possible to use physical redundancy by installing extra equipment, e.g. duplicate processing on
two different CPUs.
When a fault-tolerant system detects a fault, it either fails gracefully or masks it. Graceful failure
means that it informs the user, notifies any other processes it is communicating with, closes all
I/O streams it has open, returns memory, and stops the process. Masking a fault would include
retrying an operation on a different CPU, attempting to contact a different server on a network,
using a different software routine, or using an alternative source for data.
At the very least, faults should be confined to the process in which they occur. They must not
spread to other processes or to the operating system itself.
10.2.4 Faults in file systems
The following are some common faults which are specific to file systems.
Bad read. One or more sectors of a disk cannot be read. At best, the user loses part of a
file. At worst, a whole directory, an index block or a bitmap can be lost.
Bad write. Information is written to the wrong sector. Chains of pointers can be corrupted.
One file can be linked into another or, even worse, into the free list, resulting in chaos.
When a system uses a disk buffer cache, a power failure can leave the disk in an
inconsistent state.
Viruses can cause corruption or even total loss of data.
There are always faults attributable to humans, whether intentional or not.
Precautions
In designing safeguards against these faults, the following factors have to be balanced.
What is the mean time between failure (MTBF) for the hardware? In other words, what
are the odds against the system crashing?
What is the operational cost of making backup copies? If it only requires a click on an
icon or can be done automatically, then why not do it? But in some systems it may mean
shutting the computer down.
What is the cost of loss of information? The loss of the latest version of a student program
is very different from the loss of banking information.
One hundred per cent protection requires that everything be recorded in duplicate or triplicate.
This involves two or three similar drives, all writing in unison. Reads are also duplicated and
compared.Another security feature is to read after every write to check that the data has been
properly written. This involves a heavy time overhead.
Generally some degree of loss can be tolerated, and the policy adopted is regular backups.One
approach to this is to do a total backup at fixed intervals. The entire file system is copied to tape
or to another disk. Recovery from a crash is easy when using a total backup. The disk is
reformatted or replaced, and the backup tape is copied to the disk. We then have a copy of the file
system as it was a day, a week or a month ago.
A security policy is a statement of the rules and practices that regulate how a computer system
manages, protects and distributes sensitiveinformation. The security policy decides whether a
given subject (user, process etc.) can be permitted to gain access to a specific object (resource,
data, file etc.). A computer system should have sufficient hardware and software to enforce such
a security policy.Each user has a set of privileges which give rights to access certain objects
through operating system functions. These privileges are acquired when a user logs on to the
system, and are normally inherited by each new process that the user creates.
Protection can be implemented in many different ways, from an all-or-nothing level down to a
very fine granularity. Generally there is a tradeoff between the granularity of the protection and
the overhead of implementing it.
Physical exclusion. If the system can only be accessed from specific terminals in a designated
area, then traditional security methods such as locks or identity checks can be used to control
access to those terminals. This level of protection is normally only used in the highest security
sites.
Exclusion of unauthorized users.The traditional approach to this is to issue each user with an
account name and a password. The problems with passwords are well known. Many systems will
not allow a user to set a password which is easily cracked. It is common practice that only
encrypted versions of passwords are stored, and the encryption algorithm cannot be reversed.
Distinguishing between users. We do not, however, want even authorized users to have access
to everything. So we must make some further distinctions after access. The system maintains a
list of the privileges granted to each user and checks every request for resources against this.This
is certainly an improvement, but it still has the weakness that access rights remain unchanged
during the lifetime of a process. A process may need access to a particular resource just once, for
example at initialisation. But it retains that right, even though it is unneeded. This is a potential
security hole.
Access for current activity only. The most fine-grained protection is based on the idea of need-
to-know, or access rights for the current activity only.For example, consider two processes
performing two-way communication through a shared buffer. Access privileges for the buffer
segment should be dependent on whether a process is currently engaged in reading or writing. So
we need mechanisms for granting and revoking privileges while a process is running.
The most general way of tracking who can access what, and how, in a system is to use an access
matrix.
The rows of the matrix represent processes, also known as subjects or domains. This is the ‘who’
part. The columns represent the resources, or objects. This is the ‘what’. The entries in the array
represent the ‘how’ part.
The information in the access matrix should not be accessible to user-level processes. It itself is
highly protected by the operating system.
An access matrix for even a small system can grow quite large. Most of the entries will be empty.
Such a sparse matrix is very wasteful of space. So they are rarely implemented as actual matrices.
Other methods are used, purely to pack the relevant data in more concisely. We will now
consider some of these.
Global table
Each entry in such a table is a set of ordered triples <Domain, Object, RightsSet>. For example,
the information from the first row of Figure 8.1 would be encapsulated as <Domain1, File1,
Read>, <Domain1, File3, Read>.
If the triple corresponding to a particular operation exists, then that operation is valid. Otherwise
it is invalid.
Even though it does not take up as much space as a full access matrix, such a table can be quite
large. Also, if a particular object can be accessed by every subject, then it must have a separate
entry corresponding to each domain. This tends to inflate the size of the table.
Capability lists
One way of slicing the access matrix is to have a separate list for each domain. This would
consist of the couples <Object, RightsSet>. As such a list specifies what a subject operating in
that domain can do, not surprisingly it is known as a capability list. The capability list for
Domain1 in Figure 10.1 would be <File1, Read>, <File3, Read>.
Another way of compacting an access matrix is to store each column in the matrix as a separate
list of ordered pairs <Domain, RightsSet>. With this scheme, each object has its own access
control list. For example, the
access list for File1 in Figure 10.1 would be <Domain1, Read>, <Domain4, Read/Write>.
If there are any default access rights, these could be put at the head of the list, e.g. <AllDomains,
RightsSet>. So the defaults would be checked first before going on to scan the remainder of the
list.
Activity 10.1
10.4.1 Authentication
This involves the set of techniques whereby a server can identify a remote user or a client can
verify that it is dealing with a legitimate server.
Servers themselves are always in danger of being infiltrated. The best known method is still
password cracking. Viruses are another way to get control of a server. A virus (in this context) is
a program which can modify another program, such as the part of the operating system
implementing the security policy.
One-way authentication
The classic mechanism for a user to authenticate itself to a server has been passwords. An
improvement for standalone machines would be non-forgeable identifiers, such as fingerprints or
voice patterns. These are difficult to transmit over a network, and so are of less value in a
distributed system.
Two-way authentication
It is not sufficient for a machine to authenticate a user; the user must also be able to authenticate
the machine. It is important to know that it is the legitimate server and not a fake. A common
method is to have one trusted authentication server in the system. Each machine can agree a
password with this server.
For example, a user wishing to log on to a file server would send its request through the
authentication server. This can verify the user and authenticate the request to the file server. It
can also verify the file server and assure the user that it is dealing with the authorized server.
10.4.2 Cryptography
For example, authorisation to an ATM to issue money could be replayed later in the hope of
getting the machine to issue the money again.
The whole aim of cryptography is to conceal private information from unauthorized eyes. The
sender uses a rule to transform the data into an unintelligible encrypted form; the recipient uses
an inverse rule. However, if an intruder discovers the rule, then all secrecy is lost.
An improvement on a rule is a function with a key as a parameter. This relies on the secure
distribution and storage of keys. Cryptography using keys also involves authentication, as
possession of the appropriate encryption or decryption key can be used to authenticate the sender
or receiver. Modern systems are moving towards authentication servers which both authenticate
users and issue keys. Of course this is putting all of the eggs in one basket. If an intruder gains
access to this server, all security is broken. It is now accepted that such security systems need to
be rigidly designed using formal methods.
The key is issued in two forms. One is used to encrypt messages. Another, which can be sent
securely to the recipient, is used to decrypt them. But secure distribution of keys is a problem.
One possibility is to use an authentication server to distribute them. Such a server maintains a
table of <name, secret key> pairs, which is used to authenticate clients.
Suppose, for example, a client A wishes to communicate secretly with B. It first authenticates
itself to the server by sending a message encrypted with its secret key. This message asks for a
key to communicate with its destination B. This is the first of the sequence of messages
illustrated in Figure 10.2.
The server uses A’s secret key to decrypt this message, so proving that it must have come from
A. It then generates a one-off key for this communication between A and B, and sends this key,
as well as a copy of this key encrypted with B’s secret key, back to A. This whole message is
encrypted with A’s secret key, so A can decrypt it. This is the second message in Figure 10.2.
A then sends the encrypted version of the one-off key to B. As the server originally encrypted
this with B’s secret key, B is able to decrypt it and extract the one-off key. A never learns B’s
secret key.
Then A encrypts its message using the one-off key, and sends this encrypted message to B. B
uses the one-off key to decrypt it, thus proving that it must have come from A.
Each recipient has two keys, such that either can be used to decrypt a message encrypted with the
other. One, the private key, is kept very secret. The other, the public key, is freely available, e.g.
on the Web. Anyone can encrypt a message with the public key, but only the recipient can
decrypt it, using the private key.
With the growth in the number of computer documents, we need to be able to authenticate such
documents. Again, either public or secret keys can be used for this.
Public keys
The document is encrypted with the private key. Anyone can decrypt it using the public key, but
only the originator could have encrypted it. So its authenticity is guaranteed.
Secret keys
This requires the use of an authentication server. The source process sends the message,
encrypted with its secret key, to this server, which verifies the sender. The server then adds a
certificate of authenticity, and encrypts the message with the secret key of the destination
process. The receiver has the assurance of the server it trusts that the message is authentic.
Activity 10.2
1. Explain the problem of mutual authentication in a distributed system, and some of the
approaches taken.
2. Outline how secret keys and public keys can be used to encrypt data passing over
communication lines.
3. Explain how digital signatures work.
UNIX security aims to protect users from each other and the system’s trusted computing base
(TCB) from all users. Informally, the UNIX TCB consists of the kernel and several processes that
run with the identity of the privileged user, root or superuser. These root processes provide a
variety of services, including system boot, user authentication, administration, network services,
etc. Both the kernel and root processes have full system access. All other processes have limited
access based on their associated user’s identity.
10.5.1. Unix Protection System
UNIX implements a classical protection system, not the secure protection system. As stated in
before, a UNIX protection system consists of a protection state and a set of operations that enable
processes to modify that state. Thus, UNIX is a discretionary access control (DAC) system.
However, UNIX does have some aspects of the secure protection system in Definition 2.4. First,
the UNIX protection system defines a transition state that describes how processes change
between protection domains. Second, the labelling state is largely ad hoc. Trusted services
associate processes with user identities, but users can control the assignment of permissions to
system resources (i.e., files). In the final analysis, these mechanisms and the discretionary
protection system are insufficient to build a system that satisfies the secure operating system
requirements.
Recall that a protection state describes the operations that the system’s subjects can perform on
that system’s objects. The UNIX protection state associates process identities (subjects) with their
access to files (objects). Each UNIX process identity consists of a user id (UID), a group id
(GID), and a set of supplementary groups. These are used in combination to determine access as
described below.
All UNIX resources are represented as files. The protection state specifies that subjects may
perform read, write, and execute operations on files, with the standard meaning of these
operations. While directories are not files, they are represented as files in the UNIX protection
state, although the operations have different semantics (e.g., execute means search for a
directory). Files are also associated with an owner UID and an owner GID which conveys special
privileges to processes with these identities. A process with the owner UID can modify any
aspect of the protection state for this file. Processes with either the owner UID or group GID may
obtain additional rights to access the file as described below.
The limited set of objects and operations enabled UNIX designers to use a compressed access
control list format called UNIX mode bits, to specify the access rights of identities to files. Mode
bits define the rights of three types of subjects: (1) the file owner UID; (2) the file group GID;
and (3) all other subjects. Using mode bits authorization is performed as follows. First, the UNIX
authorization mechanism checks whether the process identity’s UID corresponds to the owner
UID of the file, and if so, uses the mode bits for the owner to authorize access. If the process
identity’s GID or supplementary groups correspond to the file’s group GID, then the mode bits
for the group permissions are used. Otherwise, the permissions assigned to all others are used.
Example 10.1. UNIX mode bits are of the form {owner bits, group bits, others bits} where each
element in the tuple consists of a read bit, a write bit, and an execute bit. The mode bits:
rwxr--r--mean that a process with the same UID as the owner can read, write, or execute the file,
a process with a GID or supplementary group that corresponds to the file’s group can read the
file, and others can also only read the file. Suppose a set of files have the following owners,
groups, and others mode bits as described below:
Name Owner Group Mode Bits
foo alice faculty rwxr--r--
Bar bob students rw-rw-r--
baz charlie faculty rwxrwxrwx
Then, processes running as alice with the group faculty can read,write, or execute foo and baz,
but only read bar. For bar, Alice does not match the UID (bob), nor have the associated group
(students). The process has the appropriate owner to gain all privileges for foo and the
appropriate group to gain privileges to baz.
As described above, the UNIX protection system is a discretionary access control system.
Specifically, this means that a file’s mode bits, owner UID, or group GID may be changed by any
UNIX processes run by the file’s owner (i.e., that have the same UID as the file owner). If we
trust all user processes to act in the best interests of the user, then the user’s security goals can be
enforced. However, this is no longer a reasonable assumption. Nowadays, users run a variety of
processes, some of which may be supplied by attackers and others may be vulnerable to
compromise from attackers, so the user will have no guarantee that these processes will behave
consistently with the user’s security goals. As a result, a secure operating system cannot use
discretionary access control to enforce user security goals.
Since discretionary access control permits users to change their files owner UID and group GID
in addition to the mode bits, file labelling is also discretionary. A secure protection system
requires a mandatory labelling state, so this is another reason that UNIX systems cannot satisfy
the requirements of a secure operating system.
UNIX processes are labelled by trusted services from a set of labels (i.e., user UIDs and group
GIDs) defined by trusted administrators, and child processes inherit their process identity from
their parent. This mandatory approach to labelling processes with identities would satisfy the
secure protection system requirements, although it is rather inflexible.
Finally, UNIX mode bits also include a specification for protection domain transitions, called the
setuid bit. When this bit is set on a file, any process that executes the file with automatically
perform a protection domain transition to the file’s owner UID and group GID. For example, if a
root process sets the setuid bit on a file that it owns, then any process that executes that file will
run under the root UID. Since the setuid bit is a mode bit, it can be set by the file’s owner, so it is
also managed in a discretionary manner. A secure protection state requires a mandatory transition
state describe all protection domain transitions, so the use of discretionary setuid bits is
insufficient.
File descriptors are stored in the kernel, and only an index is returned to the process. Thus, file
descriptors are a form of capability (see Chapter 2 for the definition and Chapter 10 for a
discussion on capability-based systems). User processes present their file descriptor index to the
kernel when they request operations on the files that they have opened. UNIX authorization
controls traditional file operations by mediating file open for read, write, and execute
permissions. However, the use of these permissions does not always have the expected effect: (1)
these permissions and their semantics do not always enable adequate control and (2) some objects
are not represented as files, so they are unmediated. If a user has read access to a file, this is
sufficient to perform a wide-variety of operations on the file besides reading. For example,
simply via possession of a file descriptor, a user process can perform any ad hoc command on the
file using the system calls ioctl or fcntl, as well as read and modify file metadata. Further, UNIX
does not mediate all security-sensitive objects, such as network communications. Host firewalls
provide some control of network communication, but they do not restrict network communication
by process identity.
The UNIX authorization mechanism depends on user-level authentication services, such as login
and sshd, to determine the process identity (i.e., UID, GID, and supplementary groups, see
Section 4.2.1).When a user logs in to a system, her processes are assigned her login identity. All
subsequent processes created in this login session inherit this identity unless there is a domain
transition (see below). Such user-level services also need root privileges in order to change the
identity of a process, so they run with this special UID. However, several UNIX services need to
run as root in order to have the privileges necessary to perform their tasks. These privileges
include the ability to change process identity, access system files and directories, change file
permissions, etc.
Some of these services are critical to the correct operation of UNIX authorization, such as sshd
and passwd, but others are not, such as inetd and ftp. However, a UNIX system’s trusted
computing base must include all root processes, thus risking compromise of security critical
services and the kernel itself. UNIX protection domain transitions are performed by the setuid
mechanism. setuid is used in two ways: (1) a root process can invoke the setuid system call to
change the UID of a process 4 and (2) a file can have its setuid mode bit set, such that whenever
it is executed its identity is set to the owner of the file. In the first case, a privileged process, such
as login or sshd, can change the identity of a process. For example, when a user logs in, the login
program must change the process identity of the user’s first process, her shell, to the user to
ensure correct access control. In the second case, the use of the setuid bit on a file is typically
used to permit a lower privileged entity to execute a higher privileged program, almost always as
root.
For example, when a user wishes to change her password, she uses the passwd program. Since
the passwd program modifies the password file, it must be privileged, so a process running with
the user’s identity could not change the password file. The setuid bit on the root-owned, passwd
executable’s file is set, so when any user executes passwd, the resultant process identity
transitions to root.While the identity transition does not impact the user’s other processes, the
writers of the passwd program must be careful not to allow the program to be tricked into
allowing the user to control how passwd uses its additional privileges.
UNIX also has a couple of mechanisms that enable a user to run a process with a reduced set of
permissions. Unfortunately, these mechanisms are difficult to use correctly, are only available to
root processes, and can only implement modest restrictions. First, UNIX systems have a special
principal nobody that owns no files and belongs to no groups. Therefore, a process’s permissions
can be restricted by running as nobody since it never has owner or group privileges.
Unfortunately, nobody, like all subjects, has others privileges. Also, since only root can do a
setuid only a superuser process can change the process identity to nobody. Second, UNIX chroot
can be used to limit a process to a subtree of the file system [262]. Thus, the process is limited to
only its rights to files within that subtree. Unfortunately, a chroot environment must be setup
carefully to prevent the process from escaping the limited domain. For example, if an attacker can
create /etc/passwd and /etc/shadow files in the subtree, she can add an entry for root, login as this
root, and escape the chroot environment (e.g., using root access to kernel memory). Also, a
chroot environment can only be setup by a root process, so it is not usable to regular system
users. In practice, neither of these approaches has proven to be an effective way to limit process
permissions.
1. Complete Mediation: How does the reference monitor interface ensure that all security
sensitive operations are mediated correctly? The UNIX reference monitor interface
consists of hooks to check access for file or inode permission on some system calls. The
UNIX reference monitor interface authorizes access to the objects that the kernel will use
in its operations. A problem is that the limited set of UNIX operations (read, write, and
execute) is not expressive enough to control access to information. UNIX permits
modifications to files without the need for write permission (e.g., fcntl).
2. Complete Mediation: Does the reference monitor interface mediate security-sensitive
operations on all system resources? UNIX authorization does not provide complete
mediation of all system resources. For some objects, such as network communications,
UNIX itself provides no authorization at all.
3. Complete Mediation: How do we verify that the reference monitor interface provides
complete mediation? Since the UNIX reference monitor interface is placed where the
security-sensitive operations are performed, it difficult to know whether all operations
have been identified and all paths have been mediated. No specific approach has been
used to verify complete mediation.
4. Tamperproof: How does the system protect the reference monitor, including its
protection system, from modification? The reference monitor and protection system are
stored in the kernel, but this does not guarantee tamper-protection. First, the protection
system is discretionary, so it may be tampered by any running process. Untrusted user
processes can modify permissions to their user’s data arbitrarily, so enforcing security
goals on user data is not possible. Second, the UNIX kernel is not as protected from
untrusted user processes as the Multics kernel is. Both use protection rings for isolation,
but the Multics system also explicitly specifies gates for verifying the legality of the ring
transition arguments. While UNIX kernels often provide procedures to verify system call
arguments, such procedures are may be misplaced. Finally, user-level processes have a
variety of interfaces to access and modify the kernel itself above and beyond system calls,
ranging from the ability to install kernel modules to special file systems (e.g., /proc or
sysfs) to interfaces through netlink sockets to direct access to kernel memory (e.g., via the
device file/dev/kmem). Ensuring that these interfaces can only be accessed by trusted
code has become impractical.
5. Tamperproof: Does the system’s protection system protect the trusted computing base
programs? In addition to the kernel, the UNIX TCB consists of all root processes,
including all processes run by a user logged in as a root user. Since these processes could
run any program, guaranteeing the tamper-protection of the TCB is not possible. Even
ignoring root users, the amount of TCB code is far too large and faces far too many
threats to claim a tamperproof trusting computing base. For example, several root
processes have open network ports that may be used as avenues to compromise these
processes. If any of these processes is compromised, the UNIX system is effectively
compromised as there is no effective protection among root processes. Also, any root
process can modify any aspect of the protection system. As we show below, UNIX root
processes may not be sufficiently trusted or protected, so unauthorized modification of the
protection system, in general, is possible. As a result, we cannot depend on a tamperproof
protection system in a UNIX system.
6. Verifiable: What is basis for the correctness of the system’s TCB? Any basis for
correctness in a UNIX system is informal. The effectively unbounded size of the TCB
prevents any effective formal verification. Further, the size and extensible nature of the
kernel (e.g., via new device drivers and other kernel modules) makes it impractical to
verify its correctness.
7. Verifiable: Does the protection system enforce the system’s security goals? Verifiability
enforcement of security goals is not possible because of the lack of complete mediation
and the lack of tamper proofing. Since we cannot express a policy rich enough to prevent
unauthorized data leakage or modification, we cannot enforce secrecy or integrity security
goals. Since we cannot prove that the TCB is protected from attackers, we cannot prove
that the system will be remain able to enforce our intended security goals, even if they
could be expressed properly.
However, several vulnerabilities have been reported for such processes, particularly due to buffer
overflows [232, 318], enabling remote attackers to compromise the system TCB. Some of these
daemons have been redesigned to remove many of such vulnerabilities (e.g., Postfix [317, 73] as
a replacement for sendmail and privilege-separated SSH [251]), but a comprehensive justification
of integrity protection for the resulting daemons is not provided. Thus, integrity protection of
network facing dameons in UNIX is incomplete and ad hoc.
Further, some network-facing daemons, such as remote login daemons (e.g.,telnet, rlogin,etc.)
ftpd, and NFS, puts an undo amount of trust in the network. The remote login daemons and ftpd
are notorious for sending passwords in the clear. Fortunately, such daemons have been obsoleted
or replaced by more secure versions (e.g., vsftpd for ftpd). Also, NFS is notorious for accepting
any response to a remote file system request as being from a legitimate server. Network-facing
daemons must additionally protect the integrity of their secrets and authenticate the sources of
remote data whose integrity is crucial to the process.
Rootkits Modern UNIX systems support extension via kernel modules that may be loaded
dynamically into the kernel. However, a malicious or buggy module may enable an attacker to
execute code in the kernel, with full system privileges. A variety of malware packages, called
rootkits, have been created for taking advantage of kernel module loading or other interfaces to
the kernel available to root processes. Such rootkits enable the implementation of attacker
function and provide measures to evade from detection. Despite efforts to detect malware in the
kernel, such rootkits are difficult to detect, in general.
Environment Variables UNIX systems support environment variables, system variables that are
available to processes to convey state across applications. One such variable is LIBPATH whose
value determines the search order for dynamic libraries. A common vulnerability is that an
attacker can change LIBPATH to load an attacker-provided file as a dynamic library. Since
environment variables are inherited when a child process is created, an untrusted process can
invoke a TCB program (e.g., a program file which setuid’s to root on invocation, under an
untrusted environment. If the TCB process depends on dynamic libraries and does not set the
LIBPATH itself, it may be vulnerable to running malicious code. As many TCB programs can be
invoked via setuid, this is a widespread issue.
Further, TCB programs may be vulnerable to any input value supplied by an untrusted process,
such as malicious input arguments. For example, a variety of program permit the caller to define
the configuration file of the process. A configuration file typically describes all the other places
that the program should look for inputs to describe how it should function, sometimes including
the location of libraries that it should use and the location of hosts that provide network
information. If the attack can control the choice of a program’s configuration file, she often has a
variety of ways to compromise the running process. Any TCB program must ensure their
integrity regardless of how they are invoked.
Shared Resources If TCB processes share resources with untrusted processes, then they may be
vulnerable to attack. A common problem is the sharing of the /tmp directory. Since any process
can create files in this directory, an untrusted process is able to create files in this directory and
grant other processes, in particular a TCB process, access to such files as well. If the untrusted
process can guess the name of TCB process’s /tmp file, it can create this file in advance, grant
access to the TCB process, and then have access itself to a TCB file. TCB processes can prevent
this problem by checking for the existence of such files upon creation (e.g., using the O_CREAT
flag). However, programmers have been prone to forget such safeguards. TCB process must take
care when using any objects shared by untrusted processes.
As a result of the discretionary protection system, the size of the system TCB, and these types of
vulnerabilities, converting a UNIX system to a secure operating system is a significant challenge.
Ensuring that TCB processes protect themselves, and thus protect a reference monitor from
tampering, is a complex undertaking as untrusted processes can control how TCB processes are
invoked and provide inputs in multiple ways: network, environment, and arguments. Further,
untrusted processes may use system interfaces to manipulate any shared resources and may even
change the binding between object name and the actual object.
Specifically, the Windows protection system differs from UNIX mainly in the variety of its
objects and operations and the additional flexibility it provides for assigning them to subjects.
When the Windows 2000 access control model was being developed, there were a variety of
security systems being developed that provided administrators with extensible policy languages
that permitted flexible policy specification, such as the Java 2 model. While these models address
some of the shortcomings of the UNIX model by enabling the expression of any protection state,
they do not ensure a secure system.
Subjects in Windows are similar to subjects in UNIX. In Windows, each process is assigned a
token that describes the process’s identity. A process identity consists of user security identifier
(principal SID, analogous to a UNIX UID), a set of group SIDs (rather than a single UNIX GID
and a set of supplementary groups), a set of alias SIDs (to enable actions on behalf of another
identity), and a set of privileges (ad hoc privileges just associated with this token). A Windows
identity is still associated with a single user identity, but a process token for that user may contain
any combination of rights.
Unlike UNIX, Windows objects can belong to a number of different data types besides files. In
fact, applications may define new data types, and add them to the active directory, the
hierarchical name space for all objects known to the system. From an access control perspective,
object types are defined by their set of operations. The Windows model also supports a more
general view of the operations that an object type may possess. Windows defines up to 30
operations per object type, including some operations that are specific to the data type [74]. This
contrasts markedly with the read, write, and execute operations in the UNIX protection state.
Even for file objects, the Windows protection system defines many more operations, such as
operations to access file attributes and synchronize file operations. In addition, application may
add new object types and define their own operations.
The other major difference between a Windows and UNIX protection state is that Windows
supports arbitrary access control lists (ACLs) rather than the limited mode bits approach of
UNIX. A Windows ACL stores a set of access control entries (ACEs) that describe which
operations an SID (user, group, or alias) can perform on that object 6. The operations in an ACE
are interpreted based on the object type of the target object. In Windows, ACEs may either grant
or deny an operation. Thus, Windows uses negative access rights, whereas UNIX does not,
generating some difference in their authorization mechanisms.
Example 10.2. Figure 10.3 shows an example ACL for an object foo. foo’s ACL contains three
ACEs. The field principal SID specifies the SID to which the ACE applies. These ACE apply to
the SIDs Alice, Bob, and Group1. The other two important fields in an ACE are its type (grant or
deny) and the access rights (a bitmask). The Alice and Bob ACEs grant rights, and the Group1
ACE denies access to certain rights. The access rights bitmask is interpreted based on the object
type field in the ACE.We describe how the ACL is used in authorization in the next section.
Windows authorization queries are processed by a specific component called the Security
Reference Monitor (SRM). The SRM is a kernel component that takes a process token, an object
SID, and a set of operations, and it returns a boolean result of an authorization query. The SRM
uses the object SID to retrieve its ACL from which it determines the query result.
Because of the negative permissions, the way that the SRM processes authorization queries is
more complicated than in the UNIX case. The main difference is that the ACEs in an ACL are
ordered, and the ACEs are examined in that order. The SRM searches the ACEs until it finds a
set of ACEs that permits the operation or a single ACE that denies the operation. If an ACE
grants the necessary operations 7, then the request is authorized. However, if a deny ACE is
encountered that includes one of the requested operations, then the entire request is denied.
Example 10.3. Returning to Example 10.2 above, the ACEs of the object’s ACL are ordered as
shown in Figure 10.3.Note that the ACE field for access rights is really a bitmap, but we list the
operations to simplify understanding. Further, we specify the process tokens for two processes,
P1 and P2. Below, we show the authorization results for a set of queries by these processes for
the target object.
P1, read: ok
P1, read, write: no
P2: read: ok
P2: read, write: no
Both P1 and P2 can read the target object, but neither can write the object. P1 cannot write the
object because the P1 token include Group1 which matches the deny ACE for writing. P2 cannot
write the object because the ACE for Bob does not permit writing.
Mediation in Windows is determined by a set of object managers. Rather than a monolithic set of
system calls to access homogeneous objects (i.e., files) in UNIX, each object type in Windows
has an object manager that implements the functions of that type. While the Windows object
managers all run in the kernel, the object managers are independent entities. This can be
advantageous from a modularity perspective, but the fact that object managers may extend the
system presents some challenges for mediation. We need to know that each new object manager
mediates all operations and determines the rights for those operations correctly. There is no
process for ensuring this in Windows.
In Windows, the trusted computing base consists of all system services and processing running as
a trusted user identity, such as Administrator 8.Windows provides a setuid-like mechanism for
invoking Windows Services that run at a predefined privilege, at least sufficient to support all
clients. Thus, vulnerabilities in such services would lead to system compromise. Further, the ease
of software installation and complexity of the discretionary Windows access control model often
result in users running as Administrator. In this case, any user program would be able to take
control of the system. This is often a problem on Windows systems. With the release of Windows
Vista, the Windows model is extended to prevent programs downloaded from the Internet from
automatically being able to write Windows applications and the Windows system, regardless of
the user’s process identity [152]. While this does provide some integrity protection, it does not
fully protect the system’s integrity. It prevents low integrity processes from writing to high
integrity files, but does not prevent invocation, malicious requests, or spoofing the high integrity
code into using a low integrity file.
Windows also provides a means for restricting the permissions available to a process flexibly,
called restricted contexts. By defining a restricted context for a process, the permissions
necessary to perform an operation must be available to both the process using its token and to the
restricted context. That is, the permissions of a process running in a restricted context are the
intersection of the restricted context and the process’s normal permissions. Since a restricted
context may be assigned an arbitrary set of permissions, this mechanism is much more flexible
than the UNIX option of running as nobody. Also, since restricted contexts are built into the
access control system, it less error-prone than and chroot .Nonetheless, restricted contexts are
difficult for administrators to define correctly, so they are not used commonly, and not at all by
the user community.
1. Complete Mediation: How does the reference monitor interface ensure that all security
sensitive operations are mediated correctly? In Windows, mediation is provided by object
managers. Without the source code, it is difficult to know where mediation is performed,
but we would presume that object managers would authorize the actual objects used in the
security-sensitive operations, similarly to UNIX.
2. Complete Mediation: Does the reference monitor interface mediate security-sensitive
operations on all system resources? Object managers provide an opportunity for complete
mediation, but provide no guarantee of mediation. Further, the set of managers may be
extended, resulting in the addition of potentially insecure object managers. Without a
formal approach that defines what each manager does and how it is to be secured, it will
not be possible to provide a guarantee of complete mediation.
3. Complete Mediation: How do we verify that the reference monitor interface provides
complete mediation? As for UNIX, no specific approach has been used to verify complete
mediation.
4. Tamperproof: How does the system protect the reference monitor, including its
protection system, for modification? Windows suffers from the same problems as UNIX
when it comes to tampering. First, the protection system is discretionary, so it may be
tampered by any running process. Untrusted user processes can modify permissions to
their user’s data arbitrarily, so enforcing security goals on user data is not possible. Since
users have often run as Administrator to enable ease of system administration, any aspect
of the protection system may be modified. Second, there are limited protections for the
kernel itself. Like UNIX, a Windows kernel can be modified through kernel modules. In
Microsoft Vista, a code signing process can be used to determine the certifier of a kernel
module (i.e., the signer, not necessarily the writer of the module). Of course, the
administrator (typically an end user) must be able to determine the trustworthiness of the
signer. Security procedures that depend on the decision-making of users are often prone to
failure, as users are often ignorant of the security implications of such decisions. Also,
like UNIX, the Windows kernel also does not define protections for system calls (e.g.,
Multics gates).
5. Tamperproof: Does the system’s protection system protect the trusted computing base
programs? The TCB of Windows system is no better than that of UNIX. Nearly any
program may be part of the Windows TCB, and any process running these programs can
modify other TCB programs invalidating the TCB. Like UNIX, any compromised TCB
process can modify the protection system invalidating the enforcement of system security
goals, and modify the Windows kernel itself through the variety of interfaces provided to
TCB processes to access kernel state. Unlike UNIX, Windows provides APIs to tamper
with other processes in ways that UNIX does not .For example, Windows provides the
CreateRemoteThread function, which enables a process to initiate a thread in another
process.Windows also provides functions for writing a processes memory via
OpenProcess and WriteProcessMemory, so one process can also write the desired code
into that process prior to initiating a thread in that process. While all of these operations
require the necessary access rights to the other process, usually requiring a change in
privileges necessary for debugging a process (via the AdjustTokenPrivileges). While such
privileges are typically only available to processes under the same SID, we must verify
that these privileges cannot be misused in order to ensure tamper-protection of our TCB.
6. Verifiable: What is basis for the correctness of the system’s trusted computing base? As
for UNIX, any basis for correctness is informal. Windows also has an unbounded TCB
and extensible kernel system that prevent any effective formal verification.
7. Verifiable: Does the protection system enforce the system’s security goals? The general
Windows model enables any permission combination to be specified, but no particular
security goals are defined in the system. Thus, it is not possible to tell whether a system is
secure. Since the model is more complex than the UNIX model and can be extended
arbitrarily, this makes verifying security even more difficult.
Not surprisingly given its common limitations, Windows suffers from the same kinds of
vulnerabilities as the UNIX system. For example, there are books devoted to constructing
Windows rootkits.Here we highlight a few vulnerabilities that are specific to Windows systems
or are more profound in Windows systems.
The Windows Registry The Windows Registry is a global, hierarchical database to store data for
all programs.When a new application is loaded it may update the registry with application
specific, such as security-sensitive information such as the paths to libraries and executables to be
loaded for the application. While each registry entry can be associated with a security context that
limits access, such limitations are generally not effectively used. For example, the standard
configuration of AOL adds a registry entry that specifies the name of a Windows library file (i.e.,
DLL) to be loaded with AOL software. However, the permissions were set such that any user
could write the entry.
This use of the registry is not uncommon, as vendors have to ensure that their software will
execute when it is downloaded. Naturally, a user will be upset if she downloads some newly-
purchased software, and it does not execute correctly because it could not access its necessary
libraries. Since the application vendors cannot know the ad hoc ways that a Windows system is
administered, they must turn on permissions to ensure that whatever the user does the software
runs. If the registry entry is later used by an attacker to compromise the Windows system, that is
not really the application vendor’s problem—selling applications is.
Administrator Users We mentioned in the Windows security evaluation that traditionally users
ran under the identity Administrator or at least with administrative privileges enabled. The reason
for this is similar to the reason that broad access is granted to registry entries: the user also wants
to be sure that they can use what function is necessary to enable the system to run. If the user
downloads some computer game, the user would need special privileges to install the game, and
likely need special privileges to run the device-intensive game program. The last thing the user
wants is to have to figure out why the game will not run, so enabling all privileges works around
this issue. UNIX systems are generally used by more experienced computer users who
understand the difference between installing software (e.g., run sudo) and the normal operation of
the computer. As a result, the distinction between root users and sudo operations has been utilized
more effectively in UNIX.
Enabled By Default Like users and software vendors, Windows deployments also came with full
permissions and functionality enabled. This resulted in the famous Code Red worms [88] which
attacked the SQL server component of the Microsoft IIS web server. Many people who ran IIS
did not have an SQL server running or even knew that the SQL server was enabled by default in
their IIS system. But in these halcyon times, IIS web servers ran with all software enabled, so
attackers could send malicious requests to SQL servers on any system, triggering a buffer
overflow that was the basis for this worm’s launch. Subsequent versions of IIS are now “locked
down” , such that software has to be manually enabled to be accessible.
Activity 10.3
REFERENCES
Gray, J. (1997) Interprocess Communications in Unix. Upper Saddle River, NJ: Prentice Hall.
Robbins, K. and Robbins, S. (1996) Practical Unix Programming. Upper Saddle River, NJ:
Prentice Hall.
Silberschatz, A. and Galvin, P. (1998) Operating System Concepts, 5th edn. Reading, MA:
Addison-Wesley.
Stallings, W. (1995) Operating Systems, 2nd edn. Englewood Cliffs, NJ: Prentice Hall. Stevens,
W. R. (1992) Advanced Programming in the Unix Environment. Reading, MA: Addison-Wesley.
Tanenbaum, A. (1992) Modern Operating Systems. Englewood Cliffs, NJ: Prentice Hall.
Tanenbaum, A. (1995) Distributed Operating Systems. Englewood Cliffs, NJ: Prentice Hall.
Tanenbaum, A. and Woodhull, A. (1997) Operating Systems: Design and Implementation, 2nd
edn. Upper Saddle River, NJ: Prentice Hall.