Intel MultiProcessor User Manual

MultiProcessor

Specification

Version 1.4

May

1997

THIS SPECIFICATION IS PROVIDED "AS IS" WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY WARRANTY
OF MERCHANTABILITY, FITNESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY OTHERWISE ARISING
OUT OF ANY PROPOSAL, SPECIFICATION OR SAMPLE.

A license is hereby granted to copy and reproduce this specification for internal use only.
No other license, express or implied, by estoppel or otherwise, to any other intellectual property rights is granted herein.
Intel disclaims all liability, including liability for infringement of any proprietary rights, relating to implementation of information in

this specification. Intel does not warrant or represent that such implementation(s) will not infringe such rights.
Intel Corporation and Intel's FASTPATH are not affiliated with Kinetics, a division of Excelan, Inc. or its FASTPATH trademark

or products.
* Third-party trademarks are the property of their respective owners.
Additional copies of this document or other Intel literature may be obtained from:

Intel Corporation
Literature Center
P.O. Box 7641
Mt. Prospect IL 60056-7641

or call 800-879-4683

printed on

recycled paper

Copyright  1993-1997 . Intel Corporation, All Rights Reserved.

Revision History

Revision Revision History Date

Pre-release Version 1.0. Formerly called “PC+MP Specification” 10/27/93

-001 Version 1.1. Resolves conflicts with MCA-based systems. The following
changes have been made:

1. Two MP feature information bytes were moved from the BIOS System
Configuration Table to the RESERVED area of the MP Floating Pointer
Structure.

2. If the MP Floating Pointer Structure is present, it indicates that the system
is MP-compliant, in accordance with this specification. (Previously, this was
indicated by bit 0 of MP Feature Information Byte 1.)

3. One more hardware default system configuration was added for MCA+PCI
with the integrated APIC.

-002 Minor technical corrections. 6/1/94

-003 Added Appendix D: Multiple I/O APIC Multiple PCI Bus Systems. 9/1/94

-004 Version 1.4. Added extended configuration table to improve support for
multiple PCI bus configurations and improve future expandability. Made
clarifications to Appendix B.

-005 Added Appendix E: Errata. Includes information for hierarchical PCI bus. 08/15/96

-006 Appendix E Update. Included Byte 2, Bit 6 information to MP Floating Point
Structure section (4.1).

4/11/94

7/1/95

05/12/97

iii

Table of Contents

Chapter 1 Introduction

1.1 Goals ........................................................................................................1-1

1.2 Features of the Specification....................................................................1-2

1.3 Scope........................................................................................................1-2

1.4 Target Audience .......................................................................................1-3

1.5 Organization of This Document................................................................ 1-3

1.6 Conventions Used in This Document ....................................................... 1-4

1.7 For More Information................................................................................1-4

Chapter 2 System Overview

2.1 Hardware Overview.................................................................................. 2-2

2.1.1 System Processors......................................................................2-2

2.1.2 Advanced Programmable Interrupt Controller .............................2-3

2.1.3 System Memory...........................................................................2-4

2.1.4 I/O Expansion Bus....................................................................... 2-4

2.2 BIOS Overview.........................................................................................2-5

2.3 Operating System Overview.....................................................................2-5

Chapter 3 Hardware Specification

3.1 System Memory Configuration .................................................................3-1

3.2 System Memory Cacheability and Shareability.........................................3-2

3.3 External Cache Subsystem ...................................................................... 3-4

3.4 Locking .....................................................................................................3-4

3.5 Posted Memory Write............................................................................... 3-5

3.6 Multiprocessor Interrupt Control ...............................................................3-5

3.6.1 APIC Architecture ........................................................................3-5

3.6.2 Interrupt Modes............................................................................3-6

3.6.2.1 PIC Mode......................................................................3-7

3.6.2.2 Virtual Wire Mode .........................................................3-9

3.6.2.3 Symmetric I/O Mode...................................................3-11

3.6.3 Assignment of System Interrupts to the APIC Local Unit...........3-12

3.6.4 Floating Point Exception Interrupt..............................................3-12

3.6.5 APIC Memory Mapping..............................................................3-12

v

Contents

3.6.6 APIC Identification .....................................................................3-13

3.6.7 APIC Interval Timers..................................................................3-13

3.6.8 Multiple I/O APIC Configurations...............................................3-13

3.7 RESET Support......................................................................................3-14

3.7.1 System-wide RESET ................................................................. 3-14

3.7.2 System-wide INIT.......................................................................3-15

3.7.3 Processor-specific INIT..............................................................3-15

3.8 System Initial State.................................................................................3-16

3.9 Support for Fault-resilient Booting.......................................................... 3-16

Chapter 4 MP Configuration Table

4.1 MP Floating Pointer Structure...................................................................4-3

4.2 MP Configuration Table Header............................................................... 4-5

4.3 Base MP Configuration Table Entries....................................................... 4-6

4.3.1 Processor Entries.........................................................................4-7

4.3.2 Bus Entries.................................................................................4-10

4.3.3 I/O APIC Entries.........................................................................4-12

4.3.4 I/O Interrupt Assignment Entries................................................4-12

4.3.5 Local Interrupt Assignment Entries............................................4-15

4.4 Extended MP Configuration Table Entries.............................................. 4-17

4.4.1 System Address Space Mapping Entries................................... 4-18

4.4.2 Bus Hierarchy Descriptor Entries...............................................4-21

4.4.3 Compatibility Bus Address Space Modifier Entries....................4-22

Chapter 5 Default Configurations

5.1 Discrete APIC Configurations...................................................................5-2

5.2 Integrated APIC Configurations................................................................5-4

5.3 Assignment Of I/O Interrupts To The APIC I/O Unit .................................5-6

5.3.1 EISA and IRQ13 ..........................................................................5-7

5.3.2 Level-triggered Interrupt Support.................................................5-7

5.4 Assignment Of System Interrupts To The APIC Local Unit ......................5-7

vi

Appendix A System BIOS Programming Guidelines

A.1 BIOS Post Initialization ............................................................................A-1

A.2 Controlling the Application Processors....................................................A-2

A.3 Programming the APIC for Virtual Wire Mode.........................................A-2

A.4 Constructing the MP Configuration Table................................................A-4

Appendix B Operating System Programming Guidelines

B.1 Operating System Boot-up .......................................................................B-1

B.2 Operating System Booting and Self-configuration....................................B-2

B.3 Interrupt Mode Initialization and Handling ................................................B-2

B.4 Application Processor Startup ..................................................................B-3

B.4.1 USING INIT IPI ............................................................................B-4

B.4.2 USING STARTUP IPI...................................................................B-5

B.5 AP Shutdown Handling.............................................................................B-5

B.6 Other IPI Applications...............................................................................B-6

Contents

B.6.1 Handling Cache Flush..................................................................B-6

B.6.2 Handling TLB Invalidation............................................................ B-6

B.6.3 Handling PTE Invalidation............................................................B-6

B.7 Spurious APIC Interrupts.......................................................................... B-6

B.8 Supporting Unequal Processors...............................................................B-7

Appendix C System Compliance Checklist
Appendix D Multiple I/O APIC Multiple PCI Bus Systems

D.1 Interrupt Routing with Multiple APICs.......................................................D-1

D.1.1 Variable Interrupt Routing............................................................D-1

D.1.2 Fixed Interrupt Routing................................................................D-2

D.2 Bus Entries in Systems with More Than One PCI Bus .............................D-3

D.3 I/O Interrupt Assignment Entries for PCI Devices.....................................D-3

Appendix E Errata
Glossary

vii

Contents

Figures

1-1. Conceptual Overview......................................................................1-1

1-2. Memory Layout Conventions .......................................................... 1-4

2-1. Multiprocessor System Architecture................................................2-2

2-2. APIC Configuration .........................................................................2-3

3-1. System Memory Address Map........................................................3-2

3-2. PIC Mode.......................................................................................3-8

3-3. Virtual Wire Mode via Local APIC...................................................3-9

3-4. Virtual Wire Mode via I/O APIC.....................................................3-10

3-5. Symmetric I/O Mode .....................................................................3-11

3-6. Multiple I/O APIC Configurations .................................................. 3-14

4-1. MP Configuration Data Structures ..................................................4-1

4-2. MP Floating Pointer Structure.........................................................4-3

4-3. MP Configuration Table Header......................................................4-5

Tables

4-4. Processor Entry...............................................................................4-7

4-5. Bus Entry.......................................................................................4-10

4-6. I/O APIC Entry...............................................................................4-12

4-7. I/O Interrupt Entry ......................................................................... 4-13

4-8. Local Interrupt Entry...................................................................... 4-15

4-9. System Address Space Entry .......................................................4-18

4-10. Example System with Multiple Bus Types and Bridge Types .......4-19

4-11. Bus Hierarchy Descriptor Entry.....................................................4-21

4-12. Compatibility Bus Address Space Modifier Entry..........................4-23

5-1. Default Configuration for Discrete APIC..........................................5-3

5-2. Default Configuration for Integrated APIC.......................................5-5

1-1. Document Organization .................................................................. 1-3

3-1. Memory Cacheability Map............................................................... 3-3

3-2. APIC Versions.................................................................................3-6

4-1. MP Floating Pointer Structure Fields ..............................................4-3

viii

4-2. MP Configuration Table Header Fields...........................................4-6

4-3. Base MP Configuration Table Entry Types..................................... 4-7

4-4. Processor Entry Fields....................................................................4-8

4-5. Intel486™ and Pentium® Processor Signatures.............................4-9

Contents

4-6. Feature Flags from CPUID Instruction............................................4-9

4-7. Bus Entry Fields............................................................................4-10

4-8. Bus Type String Values.................................................................4-11

4-9. I/O APIC Entry Fields.................................................................... 4-12

4-10. I/O Interrupt Entry Fields............................................................... 4-14

4-11. Interrupt Type Values.................................................................... 4-15

4-12. Local Interrupt Entry Fields...........................................................4-16

4-13. Extended MP Configuration Table Entry Types............................4-17

4-14. System Address Space Mapping Entry Fields..............................4-19

4-15. Bus Hierarchy Descriptor Entry Fields..........................................4-22

4-16. Compatibility Bus Address Space Modifier Entry Fields ...............4-24

4-17. Predefined Range Lists.................................................................4-24

5-1. Default Configurations..................................................................... 5-2

5-2. Default Configuration Interrupt Assignments ..................................5-6

5-3. Assignment of System Interrupts to APIC Local Unit......................5-7

Examples

D-1. I/O Interrupt Entry Source Bus IRQ Field for PCI Devices..............D-3

A-1. Programming Local APIC for Virtual Wire Mode.............................A-3

B-1. Universal Start-up Algorithm...........................................................B-3

ix

Introduction

HARDWARE

DRIVER/

HARDWARE

ABSTRACTION LAYER

MP BIOS

OPERATING SYSTEM

MP

CONFIGURATION

DATA

STRUCTURES

The MultiProcessor Specification, hereafter known as the “MP specification,” defines an
enhancement to the standard to which PC manufacturers design DOS-compatible systems.
MP-capable operating systems will be able to run without special customization on multiprocessor
systems that comply with this specification. End users who purchase a compliant multiprocessor
system will be able to run their choice of operating systems.

The MP specification covers PC/AT-compatible MP platform designs based on Intel processor
architectures and Advanced Programmable Interrupt Controller (APIC) architectures. The term
“PC/AT-compatible” here refers to the software-visible components of the PC/AT, not to hardware
features, such as the bus implementation, that are not visible to software. An implementation of
this specification may incorporate one or more industry standard buses, such as ISA, EISA, MCA,
PCI, or other OEM-specific buses.

1.1 Goals

The intent of this specification is to establish an MP Platform interface standard that extends the
performance of the existing PC/AT platform beyond the traditional single processor limit, while
maintaining 100% PC/AT binary compatibility.

1

The ultimate goal is to enable scalable, high-end workstations and enterprise server systems that
provide computer users with superior price/performance and have the ability to execute all existing
AT binaries, as well as MP-ready software packages on shrink-wrapped MP operating systems.
Figure 1-1 shows that at the heart of the specification are the data structures that define the
configuration of the MP system. The BIOS constructs the MP configuration data structures,
presenting the hardware in a known format to the standard device drivers or to the hardware
abstraction layer of the operating system. The specification details default hardware
configurations, and, for added flexibility, outlines extensions to the standard BIOS.

Figure 1-1. Conceptual Overview

Version 1.4 1-1

MultiProcessor Specification

1.2 Features of the Specification

The MP specification includes the following features:

• A multiprocessor extension to the PC/AT platform that runs all existing uniprocessor shrink-

wrapped binaries, as well as MP binaries.

• Support for symmetric multiprocessing with one or more processors that are Intel architecture

instruction set compatible, such as the CPUs in the Intel486™ and the Pentium
family.

• Support for symmetric I/O interrupt handling with the APIC, a multiprocessor interrupt

controller.

• Flexibility to use a BIOS with minimal MP-specific support.

• An optional MP configuration table to communicate configuration information to an MP

operating system.

• Incorporation of ISA and other industry standard buses, such as EISA, MCA, VL and PCI

buses in MP-compliant systems.

• Requirements that make secondary cache and memory bus implementation transparent to

software.

®

processor

1.3 Scope

There are many vendors building innovative MP systems based on Intel architectures today. Intel
supports and encourages vendors to develop advanced approaches to the technological
requirements of today's computing environments. In no way does the MP specification prevent
system manufacturers from adding their own unique value to MP systems. This specification does
not define the only way that multiprocessor systems can be implemented. Vendors may, for
example, create noncompliant, high-performance, scalable multiprocessor systems that do not have
the PC/AT compatibility required by this specification. Hardware vendors will always have the
option of offering one or more customized operating systems that support the unique, value-added
capabilities that they have designed into their systems. End users will be able to benefit from the
additional capabilities that these vendors offer.

The MP specification benefits PC vendors who wish to offer MP-enabled systems without having
to invest in the customization of one or more operating systems. This specification focuses on
providing a standard mechanism to add MP support to personal computers built around the PC/AT
hardware standard. With that goal, the specification covers only the minimum set of capabilities
required to extend the PC/AT platform to be MP-capable. The hardware required to implement the
MP specification is kept to a minimum, as follows:

• One or more processors that are Intel architecture instruction set compatible, such as the CPUs

in the Intel486 or Pentium processor family.

• One or more APICs, such as the Intel 82489DX Advanced Programmable Interrupt Controller

or the integrated APIC, s

together with a discrete I/O APIC unit.

• The necessary supporting electronics to duplicate the initialization and signaling protocol

described in this specification.

• PC/AT-compliant hardware.

uch as that on the Intel Pentium 735\90 and 815\100 processors,

1-2 Version 1.4

In addition to the hardware requirements, this document also specifies MP features that are visible
to the BIOS and operating system. However, it is important to understand that as hardware
technology progresses, the functions performed by the BIOS may change in accordance with the
hardware technology. ONLY THE INTERFACE TO THE OPERATING SYSTEM LEVEL

IS EXPECTED TO REMAIN CONSTANT.

This specification does not address issues relating to the processor's System Management Mode
(SMM).

1.4 Target Audience

This document is intended for the following users:

• OEMs who will be creating PC/AT-compatible, MP-ready hardware based on this

specification.

• BIOS developers, either those who create general-purpose BIOS products or those who modify

these products to suit specific MP hardware.

• Operating-system developers who will be adapting MP operating systems to run on the class of

MP-ready platform specified here.

Introduction

1.5 Organization of This Document

Table 1-1 shows the organization of this document.

Table 1-1. Document Organization

Chapter Description

2 Overview of the MultiProcessor Specification
3 Specification of the MP hardware
4 Specification of MP configuration information available to OS
5 Specification of default hardware configurations
Appendix A Guidelines for MP BIOS programming
Appendix B Guidelines for MP operating system programming
Appendix C Checklist for hardware compliance to the specification

Appendix D Clarifications for multiple I/O APIC, multiple PCI bus systems

Glossary Definitions of specialized terms used in this document

Version 1.4 1-3

MultiProcessor Specification

RESERVED

RESERVED

THREE-BYTE FIELD

RESERVED

00H

04H

08H

0CH

31 07815162324

31 07815162324

ONE-BYTE

FIELD

TWO-BYTE FIELD

LOW-ORDER BITSHIGH-ORDER BITS

INCREASING
ADDRESSES

1.6 Conventions Used in This Document

Signal names that are followed by the character # represent active low signals. For example,
FERR# is active when at its low-voltage state.

Throughout this document, the Intel 82489DX APIC is referred to as the “discrete APIC.” The
term “integrated APIC” is used to refer to an APIC integrated with other system components, such
as the Pentium 735\90 and 815\100 processors. This specification uses the term APIC to refer to
both discrete and integrated versions.

The processors of the Intel486 and Pentium processor family are “little endian” machines. This
means that the low-order byte of a multibyte data item in memory is at the lowest address, while
the high-order byte is at the highest address. Illustrations of data structures in memory show the
lowest addresses at the bottom and the highest addresses at the top of the illustration, as shown in
Figure 1-2. Bit positions are numbered from right to left.

In some memory layout descriptions, certain fields are marked RESERVED. Software should
initialize these fields as binary zeros, but should otherwise treat them as having a future, though
unknown effect. SOFTWARE SHOULD AVOID ANY DEPENDENCE ON THE VALUES

IN THE RESERVED FIELDS.

1.7 For More Information

Figure 1-2. Memory Layout Conventions

For more information, refer to any of the following documents:

• 82489DX Advanced Programmable Interrupt Controller (data book), Intel order number

290446

• Intel486 Microprocessor Programmer's Reference Manual, Intel order number 240486

• Intel Processor Identification with the CPUID Instruction AP-485, Intel order number 241618

• Pentium Processor User’s Manual, Intel order number 241428

1-4 Version 1.4

2

System Overview

In the realm of multiprocessor architectures, there are several conceptual models for tying together
computing elements, and there are a variety of interconnection schemes and details of
implementation. Figure 2-1 shows the general structure of a design based on the MP specification.
The MP specification’s model of multiprocessor systems incorporates a tightly-coupled, shared­memory architecture with a distributed interprocessor and I/O interrupt capability. It is fully
symmetric; that is, all processors are functionally identical and of equal status, and each processor
can communicate with every other processor. There is no hierarchy, no master-slave relationship,
no geometry that limits communication only to “neighboring” processors. The model is symmetric
in two important respects:

• Memory symmetry. Memory is symmetric when all processors share the same memory space

and access that space by the same addresses. Memory symmetry offers a very important
feature—the ability for all processors to execute a single copy of the operating system. Any
existing system and application software will execute the same, regardless of the number of
processors installed in a system.

• I/O symmetry. I/O is symmetric when all processors share access to the same I/O subsystem

(including I/O ports and interrupt controllers) and any processor can receive interrupts from
any source. Some multiprocessor systems that have symmetric access to memory are actually
asymmetric with regard to I/O interrupts, because they dedicate one processor to interrupt
functions. I/O symmetry helps eliminate the potential of an I/O bottleneck, thereby increasing
system scalability.

With both memory and I/O symmetry, a system that complies with the MP specification can
achieve hardware scalability as well as support software standardization. Based on this kind of
scalable architecture, systems developers can produce systems that span a wide range of price and
performance, and that all execute the same binaries.

Version 1.4 2-1

MultiProcessor Specification

HIGH-BANDWIDTH MEMORY BUS

APIC

ADVANCED PRO G RAMMABLE
INTERRUPT CONTROLLER

ICC INTERRUPT CONTROLLER

COMMUNICATIONS

CPU CPU CPU

SHARED

MEMORY

MODULE

GRAPHICS

FRAME

BUFFER

I/O

INTERFACE

APIC

I/O

INTERFACE

APIC

ICC BUS

I/O EXPANSION BUSI/O EXPANSION BUS

2.1 Hardware Overview

The MP specification defines a system architecture based on the following hardware components:

• One or more processors that are Intel architecture instruction set compatible, such as the CPUs

• One or more APICs, such as the Intel 82489DX Advanced Programmable Interrupt Controller

• Software-transparent cache and shared memory subsystem.

• Software-visible components of the PC/AT platform.

2.1.1 System Processors

Figure 2-1. Multiprocessor System Architecture

in the Intel486 and the Pentium processor family.

or the integrated APIC on the Pentium 735\90 and 815\100 processors.

To maintain compatibility with existing PC/AT software products, this specification is based on
the Intel486 and the Pentium processor family. To achieve a minimum level of MP system
performance, two or more processors that are Intel architecture instruction set compatible are
required.

Figure 2-2 gives a different point of view of a compliant system, showing the configuration of the
APICs with respect to the CPUs. While all processors in a compliant system are functionally
identical, this specification classifies them into two types: the bootstrap processor (BSP) and the
application processors (AP). Which processor is the BSP is determined by the hardware or by the
hardware in conjunction with the BIOS. This differentiation is for convenience and is in effect

2-2 Version 1.4

System Overview

ICC BUS

LOCAL

APIC

1

CPU 1

LOCAL

APIC

2

LOCAL

APIC

3

CPU 2 CPU 3

BSP AP1 AP2

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

I/O

APIC

INTERRUPT

REQUESTS

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

I/O

APIC

INTERRUPT

REQUESTS

only during the initialization and shutdown processes. The BSP is responsible for initializing the
system and for booting the operating system; APs are activated only after the operating system is
up and running. CPU1 is designated as the BSP. CPU2, CPU3, and so on, are designated as the
APs.

Figure 2-2. APIC Configuration

2.1.2 Advanced Programmable Interrupt Controller

The Advanced Programmable Interrupt Controller (APIC) is based on a distributed architecture in
which interrupt control functions are distributed between two basic functional units, the local unit
and the I/O unit. The local and I/O units communicate through a bus called the Interrupt Controller
Communications (ICC) bus, as shown in Figure 2-2.

In a multiprocessor system, multiple local and I/O APIC units operate together as a single entity,
communicating with one another over the ICC bus. The APIC units are collectively responsible
for delivering interrupts from interrupt sources to interrupt destinations throughout the
multiprocessor system.

The APICs help achieve the goal of scalability by doing the following:

• Off-loading interrupt-related traffic from the memory bus, making the memory bus more

available for processor use.

• Helping processors share the interrupt processing load with other processors.
The APICs help achieve the goal of AT-compatibility by cooperating with 8259A-equivalent PICs

in the system.

Version 1.4 2-3

MultiProcessor Specification

The local APIC units also provide interprocessor interrupts (IPIs), which allow any processor to
interrupt any other processor or set of processors. There are several types of IPIs. Among them,
the INIT IPI and the STARTUP IPI are specifically designed for system startup and shutdown.

Each local APIC has a Local Unit ID Register and each I/O APIC has an I/O Unit ID Register.
The ID serves as a physical name for each APIC unit. It is used by software to specify destination
information for I/O interrupts and interprocessor interrupts, and is also used internally for accessing
the ICC bus.

Due to the distributed architecture, the APIC local and I/O units can be implemented in either a
single chip, such as Intel’s 82489DX interrupt controller, or they can be integrated with other parts
of the system’s components. For example, the local APIC may be integrated with the CPU chip,
such as Intel’s Pentium processors (735\90, 815\100), and the I/O APIC may be integrated with the
I/O chipset, such as Intel’s 82430 PCI-EISA bridge chipset.

2.1.3 System Memory

A system that complies with the MP specification uses the standard AT memory architecture. All
memory is allocated for system memory with the exception of addresses 0A_0000h through
0F_FFFFh and 0FFFE_0000h through 0FFFF_FFFFh, which are reserved for I/O devices and the
BIOS.

Compared to a uniprocessor system, a symmetric multiprocessor system imposes a high demand
for memory bus bandwidth. The demand is proportional to the number of processors on the
memory bus. To reduce memory bus bandwidth limitations, an implementation of this
specification should use a secondary cache that has high-performance features, such as a write-back
update policy and a snooping cache-consistency protocol. A secondary cache can push the
scalability limit upward by reducing bus traffic and increasing bus bandwidth.

While both secondary caches and memory bus controllers are critical components for high
performance in a symmetric multiprocessor system, this specification does not detail their
implementation, requiring only that they be totally software transparent.

2.1.4 I/O Expansion Bus

The MP specification provides a multiprocessor extension to the industry-standard PC/AT
platform. The term “industry-standard PC/AT platform” here refers to the software-visible
components of the PC/AT. The standard does not designate a specific bus architecture. All
industry-standard buses, such as ISA, EISA, MCA, VL, and PCI, can be incorporated in the
design. Systems developers can implement one or more bus types in their designs, depending on
the systems’ I/O performance or capacity requirements.

2-4 Version 1.4

System Overview

2.2 BIOS Overview

A BIOS functions as an insulator between the hardware on one hand, and the operating system and
applications software on the other. A standard uniprocessor BIOS performs the following
functions:

• Tests system components.

• Builds configuration tables to be used by the operating system.

• Initializes the processor and the rest of the system to a known state.

• Provides run-time device-oriented services.

For a multiprocessor system, the BIOS may perform the following additional functions:

• Pass configuration information to the operating system that identifies all processors and other

multiprocessing components of the system.

• Initialize all processors and the rest of the multiprocessing components to a known state.
This specification allows a wide range of capability in the BIOS. At the minimal end of the

capability scale, the system developer can simply insert an MP floating pointer structure in the
standard BIOS. The cost of this level of simplicity in the BIOS, however, is that the system
developer has less flexibility in the design of the hardware. At the maximal end of the BIOS
capability scale might be a BIOS that dynamically configures the system to provide resilience in
the face of component malfunctions.

BIOS developers should read Chapters 3, 4, 5, and Appendix A to understand the tradeoffs
between hardware and BIOS capabilities.

2.3 Operating System Overview

Enabling the creation of shrink-wrapped MP operating systems is one of the principal goals of this
specification. This goal is achieved by allowing a flexible balance between the capabilities of the
hardware and those of the BIOS. The potentially vast variety of hardware configurations is
reduced by the BIOS to a few simple scenarios that can be readily handled by the low-level boot­up phase of the operating system.

Operating-system developers should read Chapters 3, 4, and 5, and Appendixes A and B to fully
understand the interface between the BIOS and the operating system.

Version 1.4 2-5

3

Hardware Specification

This section outlines the minimal set of common hardware features necessary for the operating
system to operate on multiple hardware platforms. The MP hardware specification defines how the
components mentioned in Chapter 2 are implemented. Compliance to the specification involves
the following aspects of hardware implementation:

• System memory configuration

• System memory cacheability and shareability

• External cache implementation requirements

• Control of memory contention (locking)

• Ordering of posted memory writes

• Multiprocessor interrupt control

• Reset support

• Interval timer usage

• Support for fault-resilient booting

While the bulk of the MP hardware specification pertains to multiprocessor interrupt control, other
areas also require some attention. The following sections take up each of these topics in turn.

3.1 System Memory Configuration

The MP memory specifications are based on the standard PC/AT memory map, which currently
has a physical memory space of four gigabytes, as shown in see Figure 3-1. Physical memory
should begin at 0 and be contiguous. Memory-mapped I/O devices should be mapped at the top of
physical memory. The APIC default base memory addresses defined by this specification are
0FEC0_0000h and 0FEE0_0000h.

Version 1.4 3-1

MultiProcessor Specification

4GB

1MB

640K

FFFF_FFFF H

FFFE_0000H

FEF0_0000H

FEE0_0000H

FED0_0000H

FEC0_0000H

0010_0000H

000F_0000H

000E_0000H

000D_0000H

000C_0000H

000A_0000H

BIOS PROM

LOCAL APIC

I/O APIC

MEMORY-MAPPED

I/O SPACE

EXTENDED

MEMORY REGION

SHADOWED BIOS

SHADOWED

EXPANSION BIOS

EXPANSION ROM

ROM EXTENSIONS

VIDEO BUFFER

SYSTEM-BASED

MEMORY

0000_0000H

PART OF THIS SPECIFICATION

UNSHADED ADDRESS REGIONS ARE FOR REFERENCE ONLY AND SHOULD NOT BE CONSTRUED AS THE SOLE
DEFINITION OF A PC/AT-COMPATIBLE ADDRESS SPACE.

Figure 3-1. System Memory Address Map

3.2 System Memory Cacheability and Shareability

The cacheability and shareability of the physical memory space are defined in Table 3-1. The
address space reserved for the local APIC is used by each processor to access its own local APIC.
The address space reserved for the I/O APIC must be shareable by all processors to permit dynamic
reconfiguration.

3-2 Version 1.4

Table 3-1. Memory Cacheability Map

Hardware Specification

Addresses
(in hex) Size Description

0000_0000h –

640KB Main memory Yes Yes

0009_FFFFh
000A_0000h –

000B_FFFFh
000C_0000h –

000D_FFFFh
000E_0000h –

128KB Display buffer for

video adapters

128KB ROM BIOS for add-on

cards

128KB System ROM BIOS Yes Yes

000F_FFFFh
0010_0000h –

Main memory Yes Yes Maximum address

0FEBF_FFFFh

Not specified. Memory-mapped I/O

devices

0FEC0_0000h –
0FECF_FFFFh

0FED0_0000h –
0FEDF_FFFFh

1

APIC I/O unit Yes No Refer to the register

Reserved for
memory-mapped I/O
devices

Shared by All
Processors? Cacheable? Comment

Yes No

Yes Yes

depends on total memory
installed in the system.

2

Yes

Not

Top unused memory

specified

description in the APIC
data book.

2

Yes

Not
specified

0FEE0_0000h­0FEEF_FFFFh

1

APIC Local Unit No No Refer to the register

description in the APIC
data book.

0FEF0_0000h –
0FFFD_FFFFh

Reserved for
memory-mapped I/O

Yes

2

Not
specified

devices

0FFFE_0000h –
0FFFF_FFFFh

NOTES:

1. These addresses are part of this specification. The other address regions in this table are shown for reference only, and

should not be construed as the sole definition of a PC/AT-compatible address space format or cache.

2. Any memory-mapped device should be shareable unless the nature of the device is that there is one device per

processor.

128KB Initialization ROM Yes Not

specified

Version 1.4 3-3

MultiProcessor Specification

3.3 External Cache Subsystem

Intel-compatible processors support multiprocessing both on the processor bus and on a memory
bus, both with and without secondary cache units. Due to the high bandwidth demands of
multiprocessor systems, external caches are often employed to improve performance. The
existence and implementation details of external caches are not a part of this specification.
However, when external caches are used, they must conform to certain requirements with regard to
the following design issues:

• Maintaining cache coherency—When one processor accesses data cached in another

processor’s cache, it must not receive incorrect data. If it modifies data, all other processors
that access that data also must not receive stale data. External caches must maintain coherency
among themselves, and with the main memory, internal caches, and other bus master DMA
devices.

• Cache flushing—The processor can generate special flush and write-back bus cycles that must

be used by external caches in a manner that maintains cache coherency. The actual responses
are implementation-specific and may vary from design to design. A program can initiate
hardware cache flushing by executing a WBINVD instruction. This instruction is only
guaranteed to flush the caches of the local processor. See Appendix B for system-wide
flushing mechanisms. Given that cache coherency is maintained by hardware, there is no need
for software to issue cache flush instructions under normal circumstances.

• Reliable communication—All processors must be able to communicate with each other in a

way that eliminates interference when more than one processor accesses the same area in
memory simultaneously. The processor uses the LOCK# signal for this purpose. External
caches must ensure that all locked operations are visible to other processors.

• Write ordering—In some circumstances, it is important that memory writes be observed

externally in precisely the same order as programmed. External write buffers must maintain
the write ordering of the processor.

3.4 Locking

To protect the integrity of certain critical memory operations, Intel-compatible processors provide
an output signal called LOCK#. For any given memory access, LOCK# is asserted once, but may
remain asserted for as many memory bus cycles as required to complete the memory operation. It
is the responsibility of the system hardware designers to use this signal to control memory accesses
among processors.

A compliant system in multiprocessor mode must guarantee atomicity of locked-aligned memory
operations; however, the implementation is not specified in this specification. A compliant system
must lock at least the area of memory defined by the destination operand. A specific
implementation may lock a broader area—it may even lock the entire bus. Therefore, software
must consider this behavior.

To guarantee AT compatibility, locking of misaligned memory operations over other
AT-compatible buses in the compliant system must be strictly implemented in accordance with the
bus specifications. A compliant system may not be required to support the misaligned memory

3-4 Version 1.4

Hardware Specification

operations over its internal shared memory bus, if it is AT compatible. Operating system and
software developers must ensure

operations on misaligned data are not guaranteed to work on all platforms.

that data is aligned if locked access is required, because lock

3.5 Posted Memory Write

When controlling I/O devices, it is important that memory and I/O operations be carried out in the
order programmed. Intel-compatible processors do not buffer I/O writes; thus, strict ordering
among I/O operations is enforced by the processors.

To optimize memory performance, processors and chipsets often implement write buffers and
writeback caches. Intel-compatible processors guarantee processor ordering on all internal cache
and write buffer accesses. However, chipsets must also guarantee processor ordering on all
external memory accesses.

For systems based on the integrated APIC, posting of memory writes may result in spurious
interrupts for memory-mapped I/O devices using level-triggered interrupts. I/O device drivers
must serialize instructions to ensure that the device interrupt clear command reaches the device
before the EOI command reaches the APIC and handles the spurious interrupt in case one occurs.

3.6 Multiprocessor Interrupt Control

In an MP-compliant system, interrupts are controlled through the APIC. The following sections
describe the APIC architecture and the three interrupt modes allowed in an MP-compliant system.

3.6.1 APIC Architecture

The Intel Advanced Programmable Interrupt Controller (APIC) is based on a distributed
architecture. Interrupt control functions are distributed between two basic functional units: the
local unit and the I/O unit. The local and I/O units communicate through a bus called the ICC bus.
The I/O unit senses an interrupt input, addresses it to a local unit, and sends it over the ICC bus.
The local unit that is addressed accepts the message sent by the I/O unit.

In an MP-compliant system, one local APIC per CPU is required. Depending on the total number
of interrupt lines in an MP system, one or more I/O APICs may be used. The bus interrupt line
assignments can be implementation-specific and can be defined by the MP configuration table
described in Chapter 4.

The Intel 82489DX APIC is a “discrete APIC” implementation. The programming interface of the
82489DX APIC units serves as the base of the MP specification. Each APIC has a version register
that contains the version number of a specific APIC implementation. The version register of the
82489DX family has a version number of “0x,” where x is a four-bit hexadecimal number. Version
number “1x” refers to Pentium processors with integrated APICs, such as the Pentium 735\90 and
815\100 processors, and x is a four-bit hexadecimal number.

The integrated APIC maintains the same programming interface as the 82489DX APIC. Table 3-2
describes the features specific to the integrated APIC.

Version 1.4 3-5

MultiProcessor Specification

Table 3-2. APIC Versions

Local APIC Version

APIC Type

82489DX APIC 0x
Integrated APIC, i.e.,

Pentium processors (735\90,
815\100)

NOTE:

is a 4-bit hexadecimal number.

x

Register (hexadecimal) Integrated APIC Features

1x STARTUP IPI. See Appendix B.4.2 for details.

Programmable interrupt input polarity

To encourage future extendibility and innovation, the Intel APIC architecture definition is limited
to the programming interface of the APIC units. The ICC bus protocol and electrical specifications
are considered implementation-specific. That is, while different versions of APIC implementations
may execute the same binary software, different versions of APIC components may be
implemented with different bus protocols or electrical specifications. Care must be taken when
using different versions of the APIC in a system.

The APIC architecture is designed to be scaleable. The 82489DX APIC has an 8-bit ID register
that can address from one to 255 APIC devices. Furthermore, the Logical Destination register for
the 82489DX APIC supports 32 bits, which can address up to 32 devices. For small system
implementations, the APIC ID register can be reduced to the least significant 4 bits and the Logical
Destination register can be reduced to the most significant 8 bits.

To ensure software compatibility with all versions of APIC implementations, software developers
must follow the following programming guidelines:

1. Assign an 8-bit APIC ID starting from zero.

2. Assign logical destinations starting from the most significant byte of the 32-bit register.

3. Program the APIC spurious vector to hexadecimal “xF,” where x is a 4-bit hexadecimal

number.

The following features are only available in the integrated APIC:

1. The I/O APIC interrupt input signal polarity can be programmable.

2. A new interprocessor interrupt, STARTUP IPI is defined.
In general, the operating system must use the STARTUP IPI to wake up application processors in

systems with integrated APICs, but must use INIT IPI in systems with the 82489DX APIC. Refer
to Appendix B, Section B.4, for application processor startup.

3.6.2 Interrupt Modes

The MP specification defines three different interrupt modes as follows:

1. PIC Mode—effectively bypasses all APIC components and forces the system to operate in

single-processor mode.

2. Virtual Wire Mode—uses an APIC as a virtual wire, but otherwise operates the same as PIC

Mode.

3. Symmetric I/O Mode—enables the system to operate with more than one processor.

3-6 Version 1.4

Hardware Specification

The first two interrupt modes, PIC Mode and Virtual Wire Mode, provide PC/AT-compatibility.
At least one of these modes must be implemented in systems that comply with the MP
specification. In these modes, full DOS compatibility with the uniprocessor PC/AT is provided by
using the APICs in conjunction with standard 8259A-equivalent programmable interrupt
controllers (PICs).

The third mode, Symmetric I/O Mode, must be implemented in addition to either PIC Mode or
Virtual Wire Mode. An MP operating system is booted under either one of the two PC/AT­compatible modes. Later the operating system switches to Symmetric I/O Mode as it enters
multiprocessor mode.

The interrupt modes are implemented by a combination of hardware and software. The hardware
and programming specifications for each of these modes are further defined in the following
subsections. BIOS programmers should refer to Appendix A, which includes information about
programming the APIC for Virtual Wire Mode. Operating system programmers should refer to
Appendix B, which provides more information about initializing the APIC for Symmetric I/O
Mode.

3.6.2.1 PIC Mode

PIC Mode is software compatible with the PC/AT because it actually employs the same hardware
interrupt configuration. As Figure 3-2 illustrates, the hardware for PIC Mode bypasses the APIC
components by using an interrupt mode configuration register (IMCR). This register controls
whether the interrupt signals that reach the BSP come from the master PIC or from the local APIC.
Before entering Symmetric I/O Mode, either the BIOS or the operating system must switch out of
PIC Mode by changing the IMCR.

Version 1.4 3-7

MultiProcessor Specification

BSP AP1 AP2

IMCR

E0

REG.

MARK

LINTIN1

LINTIN0

RESET
ICC BUS

NMI

INTERRUPT INPUTS

CPU 1

NMI

INTR

LOCAL

APIC

1

LINTIN0 LINTIN1

CPU 2

NMI INTR

LOCAL

APIC

2

LINTIN0 LINTIN1 LINTIN0 LINTIN1

8259A-

EQUIVALENT

PICS

INTR

CPU 3

NMI INTR

LOCAL

APIC

APIC

3

I/O

SHADED AREAS INDICATE UNUSED CIRCUITS. DOTTED LINE SHOWS INTERRUPT PATH.

Figure 3-2. PIC Mode

The IMCR is supported by two read/writable or write-only I/O ports, 22h and 23h, which receive
address and data respectively. To access the IMCR, write a value of 70h to I/O port 22h, which
selects the IMCR. Then write the data to I/O port 23h. The power-on default value is zero, which
connects the NMI and 8259 INTR lines directly to the BSP. Writing a value of 01h forces the
NMI and 8259 INTR signals to pass through the APIC.

The IMCR must be cleared after a system-wide INIT or RESET to enable the PIC Mode as default.
(Refer to Section 3.7 for information on the INIT and RESET signals.)

The IMCR is optional if PIC Mode is not implemented. The IMCRP bit of the MP feature
information bytes (refer to Chapter 4) enables the operating system to detect whether the IMCR is
implemented.

3-8 Version 1.4

Hardware Specification

3.6.2.2 Virtual Wire Mode

Virtual Wire Mode provides a uniprocessor hardware environment capable of booting and running
all DOS shrink-wrapped software.

In Virtual Wire Mode, as shown in Figure 3-3, the 8259A-equivalent PIC fields all interrupts, and
the local APIC of the BSP becomes a virtual wire, which delivers interrupts from the PIC to the
BSP via the local APIC’s local interrupt 0 (LINTIN0). The LINTIN0 pin of the local APIC is
programmed as ExtINT, specifying to the APIC that the PIC is to serve as an external interrupt
controller. Whenever the local APIC finds that a particular interrupt is of type ExtINT, it asserts
the ExtINTA transaction along with the PINT interrupt to the processor. In this case, the I/O APIC
is not used.

BSP AP1 AP2

REG.

MARK

LINTIN1
LINTIN0
RESET
ICC BUS

NMI

INTERRUPT INPUTS

CPU 1

NMI

INTR

LOCAL

APIC

1

LINTIN0 LINTIN1

CPU 2

NMI INTR

LOCAL

APIC

2

LINTIN0 LINTIN1 LINTIN0 LINTIN1

8259A-

EQUIVALENT

PICS

INTR

CPU 3

NMI INTR

LOCAL

APIC

APIC

3

I/O

SHADED AREAS INDICATE UNUSED CIRCUITS. DOTTED LINE SHOWS INTERRUPT PATH.

Figure 3-3. Virtual Wire Mode via Local APIC

Version 1.4 3-9

MultiProcessor Specification

Figure 3-3 shows how Virtual Wire Mode can be implemented through the BSP’s local APIC. It is
also permissible to program the I/O APIC for Virtual Wire Mode, as shown in Figure 3-4. In this
case the interrupt signal passes through both the I/O APIC and the BSP’s local APIC.

BSP AP1 AP2

LINTIN1
LINTIN0

RESET

ICC BUS

NMI

INTERRUPT INPUTS

REG.

MARK

CPU 1

NMI

INTR

LOCAL

APIC

1

LINTIN0 LINTIN1

CPU 2

NMI INTR

LOCAL

APIC

2

LINTIN0 LINTIN1 LINTIN0 LINTIN1

8259A-

EQUIVALENT

PICS

INTR

CPU 3

NMI INTR

LOCAL

APIC

APIC

3

I/O

SHADED AREAS INDICATE UNUSED CIRCUITS. DOTTED LINE SHOWS INTERRUPT PATH.

Figure 3-4. Virtual Wire Mode via I/O APIC

3-10 Version 1.4