Operating Systems

Start Lecture #11

3.6.4 Locking (Pinning) Pages in Memory

We discussed pinning jobs already. The same (mostly I/O) considerations apply to pages.

3.6.5 Backing Store

The issue is where on disk do we put pages that are not in frames.

Homework: Assume every memory reference takes 0.1 microseconds to execute providing the reference page is memory resident. Assume a page fault takes 10 milliseconds to service providing the necessary disk block is actually on the disk. Assume a disk block fault takes 10 seconds service. So the worst case time for a memory reference is 10.0100001 seconds. Finally assume the program requires that a billion memory references be executed.

  1. If the program is always completely resident, how long does it take to execute?
  2. If 0.1% of the memory references cause a page fault, but all the disk blocks are on the disk, how long does the program take to execute and what percentage of the time is the program waiting for a page fault to complete?
  3. If 0.1% of the memory references cause a page fault and 0.1% of the page faults cause a disk block fault, how long does the program take to execute and what percentage of the time is the program waiting for a disk block fault to complete?

3.6.6 Separation of Policy and Mechanism

Skipped.

3.7 Segmentation

Up to now, the virtual address space has been contiguous. In segmentation the virtual address space is divided into a number of variable-size segments. One can view the designs we have studied so far as having just one segment, the entire process.

Homework: Explain the difference between internal fragmentation and external fragmentation. Which one occurs in paging systems? Which one occurs in systems using pure segmentation?

** Two Segments

Late PDP-10s and TOPS-10

** Three Segments

Traditional (early) Unix had three segments as shown on the right.

  1. Shared text marked execute only.
  2. Data segment (global and static variables).
  3. Stack segment (automatic variables).

Since the text doesn't grow, this was sometimes treated as 2 segments by combining text and data into one segment. But then the text could not be shared.

** General (Not Necessarily Demand) Segmentation

Segmentation is a user-visible division of a process into multiple variable-size segments, whose sizes change dynamically during execution. It enables fine-grained sharing and protection. For example, one can share the text segment as done in early unix.

With segmentation, the virtual address has two components: the segment number and the offset in the segment.

Segmentation does not mandate how the program is stored in memory.

Any segmentation implementation requires a segment table with one entry for each segment.

The address translation for segmentation is
    (seg#, offset) --> if (offset<limit) base+offset else error.

3.7.1: Implementation of Pure Segmentation

Pure Segmentation means segmentation without paging.

Segmentation, like whole program swapping, exhibits external fragmentation (sometimes called checkerboarding). (See the treatment of OS/MVT for a review of external fragmentation and whole program swapping). Since segments are smaller than programs (several segments make up one program), the external fragmentation is not as bad as with whole program swapping. But it is still a serious problem.

As with whole program swapping, compaction can be employed.

ConsiderationDemand
Paging		Demand
Segmentation
Programmer awareNoYes
How many addr spaces1Many
VA size > PA sizeYesYes
Protect individual
procedures separately 		NoYes
Accommodate elements
with changing sizes 		NoYes
Ease user sharingNoYes
Why invented let the VA size
exceed the PA size 		Sharing, Protection,
independent addr spaces
Internal fragmentation YesNo, in principle
External fragmentationNoYes
Placement questionNoYes
Replacement questionYesYes

** Demand Segmentation

Same idea as demand paging, but applied to segments.

The table on the right compares demand paging with demand segmentation. The portion above the double line is from Tanenbaum.

** 3.7.2 and 3.7.3 Segmentation With (Demand) Paging

These two sections of the book cover segmentation combined with demand paging in two different systems. Section 3.7.2 covers the historic Multics system of the 1960s (it was coming up at MIT when I was an undergraduate there). Multics was complicated and revolutionary. Indeed, Thompson and Richie developed (and named) Unix partially in rebellion to the complexity of Multics. Multics is no longer used.

Section 3.7.3 covers the Intel Pentium hardware, which offers a segmentation+demand-paging scheme that is not used by any of the current operating systems (OS/2 used it in the past). The Pentium design permits one to convert the system into a pure damand-paging scheme and that is the common usage today.

I will present the material in the following order.

  1. Describe segmentation+paging (not demand paging) generically, i.e. not tied to any specific hardware or software.
  2. Note the possibility of using demand paging (again generically).
  3. Give some details of the Multics implementation.
  4. Give some details of the Pentium hardware, especially how it can emulate straight demand paging.

** Segmentation With (non-demand) Paging

One can combine segmentation and paging to get advantages of both at a cost in complexity. In particular, user-visible, variable-size segments are the most appropriate units for protection and sharing; the addition of (non-demand) paging eliminates the placement question and external fragmentation (at the smaller average cost of 1/2-page internal fragmentation per segment).

The basic idea is to employ (non-demand) paging on each segment. A segmentation plus paging scheme has the following properties.

Although it is possible to combine segmentation with non-demand paging, I do not know of any system that did this.

Homework: 36.

Homework: Consider a 32-bit address machine using paging with 8KB pages and 4 byte PTEs. How many bits are used for the offset and what is the size of the largest page table? Repeat the question for 128KB pages. So far this question has been asked before. Repeat both parts assuming the system also has segmentation with at most 128 segments. Remind me to do this in class next time.

Homework: Consider a system with 36-bit addresses that employs both segmentation and paging. Assume each PTE and STE is 4-bytes in size.

  1. Assume the system has a page size of 8K and each process can have up to 256 segments. How large in bytes is the largest possible page table? How large in pages is the largest possible segment?
  2. Assume the system has a page size of 4K and each segment can have up to 1024 pages. What is the maximum number of segments a process can have? How large in bytes is the largest possible segment table? How large in bytes is the largest possible process.
  3. Assume the largest possible segment table is 213 bytes and the largest possible page table is 216 bytes. How large is a page? How large in bytes is the largest possible segment?

** Segmentation With Demand Paging

There is very little to say. The previous section employed (non-demand) paging on each segment. For the present scheme, we employ demand paging on each segment, that is we perform fetch-on-demand for the pages of each segment.

The Multics Scheme

Multics was the first system to employ segmentation plus demand paging. The implementation was as described above with just a few wrinkles, some of which we discuss now together with some of the parameter values.

The Pentium Scheme

The Pentium design implements a trifecta: Depending on the setting of a various control bits the Pentium scheme can be pure demand-paging (current OSes use this mode), pure segmentation, or segmentation with demand-paging.

The Pentium supports 214=16K segments, each of size up to 232 bytes.

Once the 32-bit segment base and the segment limit are determined, the 32-bit address from the instruction itself is compared with the limit and, if valid, is added to the base and the sum is called the 32-bit linear address. Now we have three possibilities depending on whether the system is running in pure segmentation, pure demand-paging, or segmentation plus demand-paging mode.

  1. In pure segmentation mode the linear address is treated as the physical address and memory is accessed.

  2. In segmentation plus demand-paging mode, the linear address is broken into three parts since the system implements 2-level-paging. That is, the high-order 10 bits are used to index into the 1st-level page table (called the page directory). The directory entry found points to a 2nd-level page table and the next 10 bits index that table (called the page table). The PTE referenced points to the frame containing the desired page and the lowest 12 bits of the linear address (the offset) finally point to the referenced word. If either the 2nd-level page table or the desired page are not resident, a page fault occurs and the page is made resident using the standard demand paging model.

  3. In pure demand-paging mode all the segment bases are zero and the limits are set to the maximum. Thus the 32-bit address in the instruction become the linear address without change (i.e., the segmentation part is effectively) disabled. Then the (2-level) demand paging procedure just described is applied.

    Current operating systems for the Pentium use this last mode.

3.8 Research on Memory Management

Skipped

3.9 Summary

Read

Some Last Words on Memory Management

We have studied the following concepts.

Chapter 4 File Systems

There are three basic requirements for file systems.

  1. Size: Store very large amounts of data.
  2. Persistence: Data survives the creating process.
  3. Concurrent Access: Multiple processes can access the data concurrently.

High level solution: Store data in files that together form a file system.

4.1 Files

4.1.1 File Naming

Very important. A major function of the file system is to supply uniform naming. As with files themselves, important characteristics of the file name space are that it is persistent and concurrently accessible.

Unix-like operating systems extend the file name space to encompass devices as well

Does each file have a unique name?
Answer: Often no. We will discuss this below when we study links.

File Name Extensions

The extensions are suffixes attached to the file names and are intended to in some why describe the high-level structure of the file's contents.

For example, consider the .html extension in class-notes.html, the name of the file we are viewing. Depending on the system and application, these extensions can have little or great significance. The extensions can be

  1. Conventions just for humans. For example letter.teq (my convention) signifies to me that this letter is written using the troff text-formatting language and employs the tbl preprocessor to handle tables and the eqn preprocessor to handle mathematical equations. Neither linux, troff, tbl, nor equ place any significance in the .teq extension.

  2. Conventions giving default behavior for some programs.
  3. Default behaviors for the operating system or window manager or desktop environment.
  4. Required for certain programs.
  5. Required by the operating system

Case Sensitive?

Should file names be case sensitive. For example, do x.y, X.Y, x.Y all name the same file? There is no clear answer.

4.1.2 File Structure

How should the file be structured. Said another way, how does the OS interpret the contents of a file.

A file can be interpreted as a

  1. Byte stream
  2. (fixed size-) Record stream: Out of date
  3. Varied and complicated beast.

4.1.3 File Types

The traditional file is simply a collection of data that forms the unit of sharing for processes, even concurrent processes. These are called regular files.

The advantages of uniform naming have encouraged the inclusion in the file system of objects that are not simply collections of data.

Regular Files

Text vs Binary Files

Some regular files contain lines of text and are called (not surprisingly) text files or ascii files. Each text line concludes with some end of line indication: on unix and recent MacOS this is a newline (a.k.a line feed), in MS-DOS it is the two character sequence carriage return followed by newline, and in earlier MacOS it was carriage return.

Ascii, with only 7 bits per character, is poorly suited for most human languages other than English. Latin-1 (8 bits) is a little better with support for most Western European Languages.

Perhaps, with growing support for more varied character sets, ascii files will be replaced by unicode (16 bits) files. The Java and Ada programming languages (and perhaps others) already support unicode.

An advantage of all these formats is that they can be directly printed on a terminal or printer.

Other regular files, often referred to as binary files, do not represent a sequence of characters. For example, a four-byte, twos-complement representation of integers in the range from roughly -2 billion to +2 billion is definitely not to be thought of as 4 latin-1 characters, one per byte.

Application Imposed File Structure

Just because a file is unstructured (i.e., is a byte stream) from the OS perspective does not mean that applications cannot impose structure on the bytes. So a document written without any explicit formatting in MS word is not simply a sequence of ascii (or latin-1 or unicode) characters.

On unix, an executable file must begin with one of certain magic numbers in the first few bytes. For a native executable, the remainder of the file has a well defined format.

Another option is for the magic number to be the ascii representation of the two characters #! in which case the next several characters specify the location of the executable program that is to be run with the current file fed in as input. That is how interpreted (as opposed to compiled) languages work in unix.
    #!/usr/bin/perl
    perl script

Strongly Typed Files

In some systems the type of the file (which is often specified by the extension) determines what you can do with the file. This make the easy and (hopefully) common case easier and, more importantly, safer.

It tends to make the unusual case harder. For example, you have a program that turns out data (.dat) files. Now you want to use it to turn out a java file, but the type of the output is data and cannot be easily converted to type java and hence cannot be given to the java compiler.

Other-Than-Regular Files

We will discuss several file types that are not called regular.

  1. Directories.
  2. Symbolic Links, which are used to give alternate names to files.
  3. Special files (for devices). These use the naming power of files to unify many actions. 
            dir               # prints on screen
        dir > file        # result put in a file
        dir > /dev/audio  # results sent to speaker (sounds awful)

4.1.4 File Access

There are two possibilities, sequential access and random access (a.k.a. direct access).

With sequential access, each access to a given file starts where the previous access to that file finished (the first access to the file starts at the beginning of the file). Sequential access is the most common and gives the highest performance. For some devices (e.g. magnetic or paper tape) access must be sequential.

With random access, the bytes are accessed in any order. Thus each access must specify which bytes are desired. This is done either by having each read/write specify the starting location or by defining another system call (often named seek) that specifies the starting location for the next read/write.

For example, in unix, if no seek occurs between two read/write operations, then the second begins where the first finished. That is, unix treats a sequences of reads and writes as sequential, but supports seeking to achieve random access.

Previously, files were declared to be sequential or random. Modern systems do not do this. Instead all files are random, and optimizations are applied as the system dynamically determines that a file is (probably) being accessed sequentially.

4.1.5 File Attributes

Various properties that can be specified for a file For example:

4.1.6 File Operations

Homework: 4, 5.