Evolution & Development HistoryIn the late 1960s a combined project between researchers at MIT, Bell Labs and General Electric led to the design of a third generation of computer operating system known as MULTICS (MULTiplexed Information and Computing Service). It was envisaged as a computer utility, a machine that would support hundreds of simultaneous timesharing users. They envisaged one huge machine providing computing power for everyone in Boston. The idea that machines as powerful as their GE-645 would be sold as personal computers costing only a few thousand dollars only 20 years later would have seemed like science fiction to them. However MULTICS proved more difficult than imagined to implement and Bell Labs withdrew from the project in 1969 as did General Electric, dropping out of the computer business altogether.One of the Bell Labs researchers (Ken Thompson) then decided to rewrite a stripped down version of MULTICS, initially as a hobby. He used a PDP-7 minico mputer that no was using and wrote the code in assembly. It was initially a stripped down, single user version of MULTICS but Thompson actually got the system to work and one of his colleagues jokingly called it UNICS (UNiplexed Information and Computing Service). The name stuck but the spelling was later changed to UNIX. Soon Thompson was joined on the project by Dennis Richie and later by his entire department.UNIX was moved from the now obsolete PDP-7 to the much more modern PDP-11/20 and then later to the PDP-11/45 and PDP-11/70. These two latter computers had large memories as well as memory protection hardware, making it possible to support multiple users at the same time. Thompson then decided to rewrite UNIX in a high-level language called B. Unfortunately this attempt was not successful and Richie designed a successor to B called C. Together, Thompson and Richie rewrote UNIX in C and subsequently C has dominated system programming ever since. In 1974, Thompson and R ichie published a paper about UNIX and this publication stimulated many universities to ask Bell Labs for a copy of UNIX. As it happened the PDP-11 was the computer of choice at nearly all university computer science departments and the operating systems that came with this computer was widely regarded as being dreadful and hence UNIX quickly came to replace them. The version that first became the standard in universities was Version 6 and within a few years this was replaced by Version 7. By the mid 1980s, UNIX was in widespread use on minicomputers and engineering workstations from a variety of vendors.In 1984, AT&T released the first commercial version of UNIX, System III, based on Version 7. Over a number of years this was improved and upgraded to System V. Meanwhile the University of California at Berkeley modified the original Version 6 substantially. They called their version 1BSD (First Berkeley Software Distribution). This was modified over time to 4BSD and improvem ents were made such as the use of paging, file names longer than 14 characters and a new networking protocol, TCP/IP. Some computer vendors like DEC and Sun Microsystems based their version of UNIX on Berkeley's rather than AT&T's. There was a few attempts to standardise UNIX in the late 1980s, but only the POSIX committee had any real success, and this was limited. During the 1980s, most computing environments became much more heterogeneous, and customers began to ask for greater application portability and interoperability from systems and software vendors. Many customers turned to UNIX to help address those concerns and systems vendors gradually began to offer commercial UNIX-based systems. UNIX was a portable operating system whose source could easily be licensed, and it had already established a reputation and a small but loyal customer base among R&D organisations and universities. Most vendors licensed source bases from either the University of California at Berkeley or AT&T(r) (two completely different source bases). Licensees extensively modified the source and tightly coupled them to their own systems architectures to produce as many as 100 proprietary UNIX variants. Most of these systems were (and still are) neither source nor binary compatible with one another, and most are hardware specific.With the emergence of RISC technology and the breakup of AT&T, the UNIX systems category began to grow significantly during the 1980s. The term "open systems" was coined. Customers began demanding better portability and interoperability between the many incompatible UNIX variants. Over the years, a variety of coalitions (e.g. UNIX International) were formed to try to gain control over and consolidate the UNIX systems category, but their success was always limited. Gradually, the industry turned to standards as a way of achieving the portability and interoperability benefits that customers wanted. However, UNIX standards and standards organisatio ns proliferated (just as vendor coalitions had), resulting in more confusion and aggravation for UNIX customers.The UNIX systems category is primarily an application-driven systems category, not an operating systems category. Customers choose an application first-for example, a high-end CAD package-then find out which different systems it runs on, and select one. The final selection involves a variety of criteria, such as price/performance, service, and support. Customers generally don't choose UNIX itself, or which UNIX variant they want. UNIX just comes with the package when they buy a system to run their chosen applications. The UNIX category can be divided into technical and business markets: 87% of technical UNIX systems purchased are RISC workstations purchased to run specific technical applications; 74% of business UNIX systems sold are multiuser/server/midrange systems, primarily for running line-of-business or vertical market applications. The UNIX systems category is extremely fragmented. Only two vendors have more than a 10% share of UNIX variant license shipments (Sun(r) and SCO); 12 of the top 15 vendors have shares of 5% or less (based on actual 1991 unit shipments, source: IDC). This fragmentation reflects the fact that most customers who end up buying UNIX are not actually choosing UNIX itself, so most UNIX variants have small and not very committed customer bases.
Operating System ArchitectureWindows NT was designed with the goal of maintaining compatibility with applications written for MS-DOS, Windows for MS-DOS, OS/2, and POSIX. This was an ambitious goal, because it meant that Windows NT would have to provide the applications with the application programming interfaces (API) and the execution environments that their native operating systems would normally provide. The Windows NT developers accomplished their compatibility goal by implementing a suite of operating system environment emulators, called environment subsystems. The emulators form an intermediate layer between user applications and the underlying NT operating system core.User applications and environment subsystems work together in a client/server relationship. Each environment subsystem acts as a server that supports the application programming interfaces of a different operating system. Each user application acts as the client of an environment subsystem because it uses the application programming interface provided by the subsystem. Client applications and environment subsystem servers communicate with each other using a message-based protocol.At the core of the Windows NT operating system is a collection of operating system components called the NT Executive. The executive's components work together to form a highly sophisticated, general purpose operating system. They provide mechanisms for:Interprocess communication.Pre-emptive multitasking.Symmetric multiprocessing.Virtual memory management.Device Input/Output.Security.Each component of the executive provides a set of functions, commonly referred to as native services or executive services. Collectively, these services form the application programming interface (API) of the NT executive.Environment subsystems are applications that call NT executive services. Each one emulates a different operating system environment. For example, the OS/2 environment subsystem supports all of the application programming interface functions used by OS/2 character mode applications. It provides these applications with an execution environment that looks and acts like a native OS/2 system. Internally, environment subsystems call NT executive services to do most of their work. The NT executive services provide general-purpose mechanisms for doing most operating system tasks. However the subsystems must implement any features that are unique to the their operating system environments.User applications, like environment subsystems, are run on the NT Executive. Unlike environment subsystems, user applications do not directly call executive services. Instead, they call application programming interfaces provided by the environment subsystems. The subsystems then call executive services as needed to implement their application programming interface functions.Windows NT presents users with an interface that looks like that of Windows 3.1. This user interface is provided by Win dows NT's 32-bit Windows subsystem (Win32). The Win32 subsystem has exclusive responsibility for displaying output on the system's monitor and managing user input. Architecturally, this means that the other environment subsystems must call Win32 subsystem functions to produce output on the display. It also means that the Win32 subsystem must pass user input actions to the other environment subsystems when the user interacts with their windows.Windows NT does not maintain compatibility with device drivers written for MS-DOS or Windows for MS-DOS. Instead, it adopts a new layered device-driver architecture that provides many advantages in terms of flexibility, maintainability, and portability. Windows NT's device driver architecture requires that new drivers be written before Windows NT can be compatible with existing hardware. While writing new drivers involves a lot of development effort on the part of Microsoft and independent hardware vendors (IHV), most of the hardware de vices supported by Windows for MS-DOS will be supported by new drivers shipped with the final Windows NT product.The device driver architecture is modular in design. It allows big (monolithic) device drivers to be broken up into layers of smaller independent device drivers. A driver that provides common functionality must only be written once. Drivers in adjacent layers can then simply call the common device driver to get their work done. Adding support for new devices is easier under Windows NT than most operating systems because only the hardware-specific drivers need to be rewritten.Windows NT's new device driver architecture provides a structure on top of which compatibility with existing installable file systems (for example, FAT and HPFS) and existing networks (for example, Novell and Banyan Vines) was relatively easy to achieve. File systems and network redirectors are implemented as layered drivers that plug easily into the new Windows NT device driver architecture.I n any Windows NT multiprocessor platform, the following conditions must hold:All CPUs are identical, and either all have identical coprocessors or none has a coprocessor.All CPUs share memory and have uniform access to memory.In a symmetric platform, every CPU can access memory, take an interrupt, and access I/O control registers. In an asymmetric platform, one CPU takes all interrupts for a set of slave CPUs. Windows NT is designed to run unchanged on uniprocessor and symmetric multiprocessor platformsA UNIX system can be regarded as hierarchical in nature. At the highest level is the physical hardware, consisting of the CPU or CPUs, memory and disk storage, terminals and other devices.On the next layer is the UNIX operating system itself. The function of the operating system is to allow access to and control the hardware and provide an interface that other software can use to access the hardware resources within the machine, without having to have complete knowledge of wha t the machine contains. These system calls allow user programs to create and manage processes, files and other resources. Programs make system calls by loading arguments into memory registers and then issuing trap instructions to switch from user mode to kernel mode to start up UNIX. Since there is no way to trap instructions in C, a standard library is provided on top of the operating system, with one procedure per system call.The next layer consists of the standard utility programs, such as the shell, editors, compilers, etc., and it is these programs that a user at a terminal invokes. They use the operating system to access the hardware to perform their functions and generally are able to run on different hardware configurations without specific knowledge of them.There are two main parts to the UNIX kernel which are more or less distinguishable. At the lowest level is the machine dependent kernel. This is a piece of code which consists of the interrupt handlers, the low-l evel I/O system device drivers and some of the memory management software. As with most of the Unix operating system it is mostly written in C, but since it interacts directly with the machine and processor specific hardware, it has to be rewritten from scratch whenever UNIX is ported to a new machine. This kernel uses the lowest level machine instructions for the processor which is why it must be changed for each different processor.In contrast, the machine independent kernel runs the same on all machine types because it is not as closely reliant on any specific piece of hardware it is running on. The machine independent code includes system call handling, process management, scheduling, pipes, signals, memory paging and memory swapping functions, the file system and the higher level part of the I/O system. The machine independent part of the kernel is by far the larger of the two sections, which is why it UNIX can be ported to new hardware with relative ease.Unix does not use the DOS and Windows idea of independently loaded device drivers for each additional hardware item that is not under BIOS control in the machine which is why it must be recompiled whenever hardware is added or removed, the kernel needing to be updated with the new information. This is the equivalent of adding a device driver to a configuration file in DOS or Windows and then rebooting the machine. It is however a longer process to undertake.
Memory ManagementWindows NT provides a flat 32-bit address space, half of which is reserved for the OS, and half available to the process. This provides a separate 2 gigabytes of demand-paged virtual memory per process. This memory is accessible to the software developer through the usual malloc() and free() memory allocation and deallocation routines, as well as some advanced Windows NT-specific mechanisms.For a programmer desiring greater functionality for memory control, Windows NT also provides Virtual and Heap memory management APIs.The advantage of using the virtual memory programming interface (VirtualAlloc(), VirtualLock(), VirtualQuery(), etc.) is that the developer has much more control over whether backing store (memory committed in the paging (swap) file to handle physical memory overcommitment) is explicitly marked, and removed from the available pool of free blocks. With malloc(), every call is assumed to require the memory to be available upon return from th e function call to be used. With VirtualAlloc() and related functions, the memory is reserved, but not committed, until the page on which an access occurs is touched. By allowing the application to control the commitment policy through access, less system resources are used. The trade-off is that the application must also be able to handle the condition (presumably with structured exception handling) of an actual memory access forcing commitment.Heap APIs are provided to make life easier for applications with memory-using stack discipline. Multiple heaps can be initialised, each growing/shrinking with subsequent accesses. Synchronisation of access to allocated heaps can be done either explicitly through Windows NT synchronisation objects, or by using an appropriate parameter at the creation of a heap. All access to memory in that particular heap is synchronised between threads in the process.Memory-mapped files are also provided in Windows NT. This provides a convenient way to access disk data as memory, with the Windows NT kernel managing paging. This memory may be shared between processes by using CreateFileMapping() followed by MapViewOfFile().Windows NT provides thread local storage (TLS) to accommodate the needs of multithreaded applications. Each thread of a subprocess has its own stack, and may have its own memory to keep various information.Windows NT is the first operating system to provide a consistent multithreading API across multiple platforms. A thread is unit of execution in a process context that shares a global memory state with other threads in that context (if any). When a process is created in Windows NT, memory is allocated for it, a state is set up in the system, and a thread object is created. To start a thread in a currently executing process, the CreateThread() call is used as a function pointer is passed in through lpStartAddr; this address may be any valid procedure address in an application. Windows NT supports a num ber of different types of multiprocessing hardware. On these designs, it's possible for different processors to be running different threads an application simultaneously. Take care to use threads in an application to synchronise access to common resources between threads. Fortunately, Windows NT has very rich synchronisation facilities.Most UNIX developers don't use threads in their applications since support is not consistent between UNIX platforms.Handles don't have a direct mapping from UNIX; however, they're very important to Win32 applications and deserve discussion. When kernel objects (such as threads, processes, files, semaphores, mutexes, events, pipes, mailslots, and communications devices) are created or opened using the Win32 API, a HANDLE is returned. This handle is a 32-bit quantity that is an index into a handle table specific to that process. Handles have associated ACLs, or Access Control Lists, that Windows NT uses to check against the security credentials of the process. Handles can be obtained by explicitly creating them (usually when an object is created), as the result of an open operation (e.g. OpenEvent()) on a named object in the system, inherited as the result of a CreateProcess() operation (a child process inherits an open handle from its parent process if inheritance was specified when the original handle was created and if the child process was created with the "inherit handles" flag set), or "given away" by DuplicateHandle(). It is important to note that unless one of these mechanisms is used, a handle will be meaningless in the context of a process. For example, suppose process 1 calls CreateEvent() to return a handle that happens to have the ordinal value 0x1FFE. This event will be used to co-ordinate an operation between different processes. Process 2 must somehow get a handle to the event that process 1 created. If process 1 somehow "conjures" that the right value to use is 0x1FFE, it still won't have access t o the event created by process 1, since that handle value means nothing in the context of process 2. If instead, process 1 calls DuplicateHandle()with the handle of process 2 (acquired through calling OpenProcess() with the integral id of process 2), a handle that can be used by process 2 is created. This handle value can be communicated back to process 1 through some IPC mechanism.Handles that are used for synchronisation (semaphores, mutexes, events) as well as those that may be involved in asynchronous I/O (named pipes, files, communications) may be used with WaitForObject() and WaitForMultipleObject(), which are functionally similar to the select() call in UNIX.Prior to 3BSD most UNIX systems were based on swapping. When more processes existed than could be kept in physical memory, some of them were swapped out to disk or drum storage. A swapped out process was always swapped out in its entirety and hence any current process was always either in memory or on disk as a co mplete unit.All movement between memory and disk was handled by the upper level of a split level scheduler, known as the (memory) swapper. Swapping from memory to disk was initiated when the kernel ran out of free physical memory. In order to choose a victim to evict, the swapper would first look at the processes that were being blocked by having to wait for something such as terminal input or a print job to respond. If more than one process was found, that process whose priority plus residence time was the highest was chosen as a candidate for swapping to disk. Thus a process that had consumed a large amount of CPU time recently was a good candidate, as was one that had been in memory a long time, even if it was mostly doing I/O. If no blocked process was available in memory then a ready process was chosen based on the same criteria of priority plus residence time. Starting with 3BSD, memory paging was added to the operating system to handle the ever larger programs that we re being written. Both 4BSD and System V implemented demand paging in a similar fashion. The theory of demand paging is that a process need not necessarily be entirely resident in memory in order to continue execution. All that is actually required is the user structure and the page tables. If these are swapped into memory, the process is then deemed to be sufficiently in memory and can be scheduled to execute. The pages of the text, data and stack segments are brought in dynamically, one at a time, as they are referenced, thus leaving memory free for other tasks rather than filling it with tables of data which may be referenced only once. If the user structure and page table are not in memory, the process cannot be executed until the swapper swaps them into memory from disk.Paging is implemented partly by the main kernel and partly by a process called the page daemon. Like all daemons, the page daemon is started up periodically so that it can look around to see if there is any work for it to do. If it discovers that the number of free pages in memory is too low, it initiates action to free up more pages.When a process is started it may cause a page fault due to one of its pages is not being resident in memory. When a page fault occurs, the operating system takes the first page frame free on the list of page frames, removes it from the list and reads the needed page into it. If the free page frame list is empty, the process must be suspended until the page daemon has had time to free a page frame from another process.The page replacement algorithm is executed by the page daemon. At a set interval (commonly 250 millisec but varying from system to system) it is activated to see if the number of free page frames is at least equal to a system parameter known as lotsfree (typically set to 1/4 of memory). If there are insufficient page frames, the page daemon will start transferring pages from memory to disk until the lotsfree parameter value of page frames are available. Alternatively, if the page daemon discovers that more than lotsfree page frames are on the free list, it has no need to perform any function and terminates until its next call by the system. If the machine has plenty of memory and few active processes, it will be inactive for most of the time.The page daemon uses a modified version of the clock algorithm. It is a global algorithm, which means that when removing a page it does not take into account whose page is being removed. Thus the number of pages each process has assigned to it varies in time, depending both on its own requirements and other process requirements. The size of the data segment may vary depending upon what has been requested, the operating system tracking allocated and unallocated memory blocks while the memalloc function manages the content of the data segment.
Process Management, Inter-process Communication and ControlThe Windows NT process model differs from that of UNIX in a number of aspects, including process groups, terminal groups, setuid, memory layout, etc. For some programs, such as shells, a re-architecture of certain portions of the code is inevitable. Fortunately, most applications don't inherently rely on the specific semantics of UNIX processes, since even this differs between UNIX versions.Quoting from the online help provided with the Windows NT SDK:Win32 exposes processes and threads of execution within a process as objects. Functions exist to create, manipulate, and delete these objects. A process object represents a virtual address space, a security profile, a set of threads that execute in the address space of the process, and a set of resources or objects visible to all threads executing in the process. A thread object is the agent that executes program code (and has its own stack and machine state). Each th read is associated with a process object which specifies the virtual address space mapping for the thread. Several thread objects can be associated with a single process object which enables concurrent execution of multiple threads in a single address space (possible simultaneous execution in a multiprocessor system running Windows NT). On multiprocessor systems running Windows NT, multiple threads may execute at the same time but on different processors. In order to support the process structure of Windows NT, APIs include: * Support for process and thread creation and manipulation. * Support for synchronisation between threads within a process and synchronisation objects that can be shared by multiple processes to allow synchronisation between threads whose processes have access to the synchronisation objects. * A uniform sharing mechanism that provides security features that limit/control the sharing of objects between processes. Windows NT provides the ability to create new processes (CreateProcess) and threads (CreateThread). Rather than "inherit" everything always, as is done in UNIX with the fork call, CreateProcess accepts explicit arguments that control aspects of process creation such as file handle inheritance, security attributes, debugging of the child process, environment, default directory, etc. It is through the explicit creation of a thread or process with appropriate security descriptors that credentials are granted to the created entity. Win32 does not provide the capability to "clone" a running process (and it's associated in-memory contents); this is not such a hardship, since most UNIX code forks and then immediately calls exec. Applications that depend on the cloning semantics of fork may have to be rearchitected a bit to use threads (especially where large amounts of data sharing between parent and child occurs), or in some cases, to use IPC mechanisms to copy the relevant data between two distinct processes after the Cr eateProcess call is executed.If a child process is to inherit the handles of the creator process, the bInherit flag of the CreateProcess call can be set. In this case, the child's handle table is filled in with handles valid in the context of the child process. If this flag is not specified, handles must be given away by using the DuplicateHandlecall.Windows NT was not designed to support "dumb terminals" as a primary emphasis, so the concept of terminal process groups and associated semantics are not implemented. Applications making assumptions about groups of applications (for example, killing the parent process kills all child processes), will have to investigate the GenerateConsoleCtrlEvent API, which provides a mechanism to signal groups of applications controlled by a parent process using the CREATE_NEW_PROCESS_GROUP flag in the CreateProcess API.Programs making assumptions about the layout of processes in memory (GNU EMACS, for example, which executes, then "dumps" th e image of variables in memory to disk, which is subsequently "overlayed" on start-up to reduce initialisation time), especially the relationship of code segments to data and stack, will likely require modification. Generally, practices such as these are used to get around some operating system limitation or restriction. At this level, a rethinking of the structure of that part of the application is generally in order, to examine supported alternatives to the "hack" that was used (perhaps memory mapped files for particular cases like this). For those who must deal with an application's pages on this level, there is a mechanism by which a process may be opened (OpenProcess), and individual memory pages, threads, and stacks examined or modified.There is no direct equivalent of the UNIX setuid. There are, however, a number of Windows NT alternatives to use depending on the task to be accomplished. If the task at hand is a daemon that runs with a fixed user context, it would be best to use a Windows NT service (again, the online help is invaluable for this information). A Windows NT service is equivalent to a "daemon" running with fixed user credentials, with the added benefit of being administrable locally or remotely through standard Windows NT administration facilities. For instances when a process must "impersonate" a particular user, it's suggested that a server program be written that communicates thr
iAutoblog the premier
autoblogger software
