CA2229441A1 - A high availability computer system and methods related thereto - Google Patents

Abstract

A high availability computer system and methodology including a backplane, having at least one backplane communication bus (208a-d) and a diagnostic bus (206), a plurality of motherboards (202a-h), each interfacing to the diagnostic bus (206). Each motherboard (202a-h) also includes a memory system (252) including main memory distributed among the plurality of motherboards (202a-h), at least one daughterboard (250a-b) and a scan chain that electrically interconnects functionalities mounted on each motherboard (202a-h) and daughterboard (250a-b). The system including instructions and criteria to automatically test the functionalities and electrical connections, using the scan chain, to determine the presence of faulted components and to functionally remove the faulted components from the computer system.

Description

W O 97/07457 PCTrUS96/13742 A HIGH AVAILABILITY COh~L~K ~Y~l~_I
AND METHODS RELATED THERETO
This application claims the benefit of U.S. Provi-sional Application Serial No. 60/002,320 filed August 14, 1995, the teaching of which are incorporated herein by reference (see also Appendix A).

FIELD OF lN V~N'l'lON
The present invention relates to computer sys-tems and more particularly to a high availability comput-er system that automatically senses, diagnoses and de-configures/ re-configures a faulted computer system to improve availability as well as related methods for pro-viding high availability.

BACKGROUND OF THE lNv~NllON
When procuring a computer system in a business envi-ronment, an important factor considered is the availabil-ity of the computer to perform/operate. This can af~ect profitability as well as work/ job performance. There are ~our basic design concepts used alone or in combina-tion to improve availability.
One design technique is co~mo~ly referred to as "fault tolerant." A computer system employing this tech-nique is designed to withstand a hard fault that could shut down another type of computer Rystem. Such a design typically involveR replicating hardware and software so an applications program is running simultaneously in W O 97/074~7 PCTAUS96/13742 multiple processors. In this way, if a hard fault occurs in one processor or subsystem, the application program running in the other processor(s)/ subsystem(s) still provides an output. Thus, as to the user, the computer system has performed its designated task. In addition to multiple processors, a voting scheme can be implemented, whereby the outputs from the multiple processors are compared to determine the correct output.
Fault tolerant systems are complex, essentially require multiple independent processing systems and, as such, are very expensive. Further, although the system is fault tolerant, once a fault occurs it is necessary for a service representative to arrive on site, diagnosis and repair the faulted path/ sub-system. This makes maintenance expensive.
Another technique, involves designing components such that they are highly reliable and, therefore, un-likely to fail during an operational cycle. This tech-nique is common for space, military and aviation applica-tions where size and weight limitations of the intended use (e.g., a satellite) typically restrict the available design techniques. Highly reliable components are typi-cally expensive and also make maintenance activities expensive to maintain these design characteristics.
Such expenses may make a computer system commercial-ly unacceptable for a given application. In any event, once a system has a failure, a service representative W O 97/07457 PCT~US96/13742 must be dispatched to diagnosis and repair the failed system. When dealing with military/ aviation applica-~ tions, the vehicle/ item housing the failed component must be brought ~o a ~epa~ fa~ ty. However, until the sy6tem is repaired it is unavailable. As such, thisincreases maintenance costs and makes such repairs/ re-placement activities critical path issues.
A third technique involves clustering multiple inde-pendent computer systems together such that when one computer system fails, its work is performed by any one of the other systems in the cluster. This technique is limited to those applications where there are, or there is a need for, a number of independent systems. It is not usable for a stand alone system. Also, in order for this type of system to work each independent computer system must be capable of accessing the data and applica-tion program of any of the systems in the cluster. For example, a central data storage device (e.g. hard drive) is provided that can be accessed by any of the computer systems. In addition to the limited applicability, the foregoing is complex, expensive and raises data security issues.
~ A fourth technique involves providing r~lln~nt power supplies and blowers. Thus, the failure of a blow-er or power supply does not result in shutdown of the computer system. However, providing re~lln~ncy for other computer systems components is not viable because a ser-W 097/07457 PCTrUS96/13742 vice representative must be brought in to diagnosis the cause of failure so the machine can be repaired and re-turned to operability.
The fourth technique also has included providing a computer system with a mechanism to automatically re-boot the system following a system crash or hang. This tech-nique may allow recovery from transient problems, howev-er, there is no diagnosing done in connection with re-storing the system to operability. Thus, if the system is faulted a service representative must be brought in to diagnosis the cause of failure so the machine can be repaired and restored to operation.
As such, there is a need for a computer system that can automatically recover from a large percentage of the potential failure modes (i.e., recover without requiring operator/ service rep. action). In particular, there is a need for a methodology that involves self-diagnosis by a computer of its and its components' functionally, as well as a computer being capable of de-configuring/ re-configuring system hardware to isolate the failed compo-nent(s). Thus, allowing the computer to automatically continue system operation albeit in a possibly degraded condition. There also is a need for a computer system having such high availability design characteristics.

W O 97/07457 PCTrUS96/13742 SU~RY OF THE l~v~NllON
The methods and multi-processor computer system o~
the present invention result ~rom the realization that availability o~ multiple processor computer systems is improved by designing the system so it automatically senses and diagnosis any of a number o~ ~ailures that interrupt system operation. Further, upon diagnosing and isolating the failure that would prevent continued system operation, the computer system automatically de-con~igur-es the system so the isolated ~ailed component/ sub-sys-tem is ~unctionally removed ~rom system. The computer system then automatically re-boots itsel~, the operating system and applications program.
The computer system continues operation, albeit in a degraded condition, until the ~ailed component is re-placed/ repaired by a service repreRentative. However, and in contrast to other computer systems, this replace-ment can be sche~llled around normal system operation (e.g., replacement during non-production hours). In this way, repair and/or replacement activities are not criti-cal path issues to restoring system operability. In general, most systems, particularly multi-proce~sor type o~ computer systems, can be operated in a degraded condi-tion with little or no real per~ormance penalty.
The computer system also is designed so a complete system diagnosis i5 per~ormed in parallel at power-up by s~nn; ng the system (i.e, chips and boards) to identi~y any component/ sub-6ystem failures. In this way, the system is verified as being in a run condition before any code is loaded. This is preferable to finding a faulted condition after the system has gone into operation. The sc~nn;ng includes using integrated JTAG test logic, to locate open and short circuits at the chip and at the board level, as well as to determine the functionality of the application specific integrated circuits, processor and boards comprising the system. The scAnn;ng operation performed during power up also includes initializing any one of a number of the ASICs.
This yields a simple, fast and low cost diagnosis and repair strategy that can be implemented in various stages or levels commensurate with the cost and type of use for the computer system. In its broadest applica-tion, the system automatically de-configures and re-conf-igures the system around virtually all failures that can lead to a system failure while providing accurate and complete fault isolation and detection. However, it is within the scope of the instant invention to tailor the amount of system re~l~n~ncy required to allow continued system operation based on the intended use as well as to m;n;m;ze the user's cost.
In addition, to components of the processing opera-tion of the computer system, the system is configured toincluded N+1 re~l~n~nt blowers and power supplies. The power supplies are preferably grouped so each power sup-ply grouping supplies power to a discrete portion of the system. Preferably, there are N+1 power supplies provid-ed for each group. The blowers and power supplies are hot repairable so they can be replaced in the field with-out shutting the system down.
The high availability computer system further in-cludes a microcontroller mounted on each motherboard and a diagnostic bus that, in conjunction with an applica-tions program and board mounted components (e.g., scan chain, test bus controller), perform diagnostic tests to determine the integrity of the system prior to the load-ing of any applications program.
The application specific integrated circuits on each motherboard and daughter board include a gated balanced clock tree to supply the clocks to the logical flip/flops (F/F). The clock tree includes a clock trunk with a plurality of branches extending therefrom. An AND gate is disposed in at least one of the branches and is under the control of control circuitry 80 that only certain clock pulses are passed through to the logical F/Fs.
The control circuitry is configured so the AND gate automatically allows pulses to pass through the branches during a first operational condition of the computer system, i.e., normal system operation, and 80 the AND

gate blocks the pulses when the system is in a second operational condition, i.e., when the system experiences a fatal error. The control circuitry i5 also configured WO 97/07457 PCTrUS96/13742 so pulses are passed when the computer system undergo diagnostic testing.
The system also includes two re~lln~nt clock genera-tion and distribution circuits where one circuit is desi-gnated as being the source of system clocks. If a fail-ure to generate the clock pulses is identified, the clock generation and distribution circuitry automatically caus-es the system to failover to the re~nn~nt clock circuit-ry and the system returned to service. Preferably, the circuitry is mounted on each motherboard of the system.
In a specific embodiment, the clock generation and dis-tribution circuitry on the motherboard in slot O of the backplane is the normal source of system clocks and the backup source is the circuitry mounted on the motherboard in slot 1.
In sum, the above methodology of the instant inven-tion yields a high availability computer system that can be configured based on the intended use and cost of the system. Thus, the amount of system re~l~n~ncy required to allow continued system operation can be tailored to the use as well as to m;n;m;ze the user's co~t. Such systems allow the repair or replacement of failed compo-nents to be scheduled for those times where it would have the least impact on the user.

W O 97/07457 PCTrUS96/13742 DEFINITIONS
The instant invention is most clearly understood with reference to the following definitions:
"ASIC" shall be understood to mean application spe-cific integrated circuit.
"Board Master JP" shall be understood to mean the lowest-numbered functional job processor/ central pro-cessing unit on each motherboard. The board master is determined by running isolated board-level tests.
"Degrade", and the related terms, shall be under-stood to mean the 1088 or removal of a component, e.g., a FRU, from a computer system. An actual reduction in operation or performance of the computer system may or may not occur with the loss or removal of the component.
"Diagnostic Master JP (DM)" shall be understood to mean the job processor/ central processing unit in the ~ystem that coordinates all inter-board testing and in control when the system first halts into the main user interface.
"Fault detection" shall be understood to mean the ability to recognize that a failure has occurred and that current system data may be corrupted.
"Fault Isolation/ Isolation of Fault", and the re-lated terms, shall be understood to mean the ability to identify the location of the failure to some level of definition. Isolation may be defined at the system, sub-~ystem, board, component, FRU or sub-FRU-level.

WO 97/07457 PCTrUS96/13742 "FRU" shall be understood to mean field replaceable unit. The field replaceable unit is a component (e.g., board, daughter board, bus, power supply, blower) of a computer system that is designed to be removed and re-5 placed as a unit in the field by a field service repre-sentative (i.e., not repaired in field).
"Master Microcontroller" shall be understood to mean the microcontroller in the system that does basic sizing of which boards exist and tests off-board scanning, in-cluding the midplane SEEPROM.
"Sc~nn~hle Mailbox" shall be understood to mean a register accessed through the DBUS by microcontrollers as a means of communication.
"SEEPROM" shall be understood to mean Serial Elec-trically Erasable Programmable Read Only Memory. There is one of these on each FRU except for SIMMs.
"TCK" shall be understood to mean te~t clocks, the IEEE 1149.1 Test Clock signal.
"TMS" shall be understood to mean the IEEE 1149.1 Test Mode Select signal.
"TDI" shall be understood to mean the IEEE 1149.1 Test Data In signal.
"TDO" shall be understood to mean the IEEE 1149.1 Test Data Out signal.
"TRST" shall be understood to mean the IEEE 1149.1 Test Reset signal.

BRIEF DESCRIPTION OF THE DRAWING
For a fuller understanding of the nature and desired objects of the present invention, reference is made to the following detailed description taken in conjunction with the accompanying drawing figures wherein like reference character denote corresponding parts throughout the several views and wherein:
FIGS. lA-C is a high level flow chart of the high availabi-lity methodology for a multiple parallel processor;
FIG. 2 is a high level block diagram of the high availability computer system of the instant invention;
FIG. 3 is a block diagram of a motherboard for the computer system of FIG 2;
FIG. 4 is a block diagram of one daughter board for the motherboard of FIG 3;
FIG. 5 is a block diagram of the PLL gated balanced clock tree for an ASIC of the instant invention;
FIG. 6 is a hierarchial depiction of the BCT;
FIG. 7 is a block diagram of the PLL module of an ASIC according to the instant invention;
FIG. 8 is a block diagram of the circuitry/ logic for the ASIC internal reset and clock generation;
~ 25 FIG. 9 is a block diagram of the clock generation and detection circuitry/ logic of the instant invention;

WO 97/074~7 PCTAUS96/13742 FIG. 10 is a block diagram of the clock failure detection circuitry;
FIG. 11 is an exemplary block diagram of the distri-bution of clocks to the motherboard and daughter boards;
FIG. 12 i8 the scan chain/ scan logic for the mothe-rboard located at position "O" on the backplane;
FIGs. 13A, B is a tabulation of power up tests per-formed by the microcontroller;
FIGs. 14 is a tabulation of the power up tests per-formed by the job processor (JP);
FIG. 15A, B is a tabulation of the power up testsperformed by the diagnostic master JP; and FIG. 16 is a tabulation of the power tests performed by the diagnostic mater JP.

DESCRIPTION OF THE PREFERRED EMBODIMENT
Referring now to the various figures of the drawing wherein like re~erence characters refer to like parts, there is shown in FIGS. lA-C a flow chart of the high availability methodology for a multiple processor comput-er system of the instant invention. The computer system and associated methodology of the instant invention yields a system that successfully detects a failure event occurring while rl~nn;ng the users application code, iso-lates the event to at least a FRU, de-configures the FRU/
faulted component, reboot~ the operating system and rebo-ots the user application code. This process of detec-tion, i601ation, de-configuration and rebooting is per-formed automatically by the computer system without re-quiring user/service representative interaction/ input.
When the computer system is being powered-up or started, step 100, the power systems controller perform~
a number of function6. This includes bringing operating voltages to no~; n~l values, bringing the blower fans up to normal speed and appropriately clearing any diagnostic interrupts or other status under its control. While in this mode, other functions of the computer system are inhibited. It should be noted, that system operation is not limited to the illustrative ~equence of operations, but rather operations may be performed in parallel and/
or in a different sequence.
When the system is started up, a syRtem clock pro-vides the clocking signals (i.e., clocks) required for proper system operation. As described hereinafter, the clock circuitry/ logic 256 (FIG. 3) monitors the opera-tion of the normal system clock source and provides a clock fail output when the normal system clock source fails, step 102. The system clock monitoring function is begun at power-up and continues during all modes of sys-tem operation until the system is powered off.
In a preferred embodiment, the computer system in-cludes re~lln~nt or back up clock circuitry/ logic so the computer system can withstand a single system clock fail-ure. Thus, if a failure of the normal system clock WO 97/07457 PCTtUS96tl3742 source i8 identified, the computer system automatically redesignates the re~nn~nt clock circuitry as being the designated source for system clocks, step 104. It should be noted that the failure of the clock circuitry/ logic 256 on the slot 0 motherboard, does not automatically require the slot 0 motherboard to be functionally deconf-igured from the system. Because a loss of the system clock i8 an irretrievable failure, the computer system automatically returns to the beginning of the diagnostic testing process. For purposes of the present invention, such an automatic re-start of the system will be referred to as a cold reset of the system. However, this cold reset does not involve le...o~ing and re-establishing power to the system.
Assuming that no other system faults are discovered during the cold reset power up process, as described below, the operating system will be automatically booted and re-invoked. Also, as part of the system start up process, messages are sent to the user and local service center regarding the clock failure. In sum, the above described clock failover process is performed automati-cally by the system without requiring the user to inter-vene and switch the clocks.
When the above initial power-up process is complet-ed, the computer system is released to per~orm a number o~ diagnostic testing and evaluation operations. These operations check out various components (e.g., chips, W O 97/074S7 PCT~US96/13742 boards, buses) to verify the integrity and operability of the computer system prior to the loading of applications (e.g., the operating system). In a preferred embodiment, the computer system is designed to include integrated JTAG test circuitry/ logic and scan ch;7;n~ so the diag-nostic testing and evaluation can be automatically per-formed by the system.
Specifically, the computer system checks for elec-trical continuity (i.e checks for shorts and opens) at the board and chip level, step 110. In addition to the continuity checks, the computer performs functionality checks, e.g., to identify a hard stuck, step 112. All components identified as being faulted are so tagged.
If there are faulted components (YES, step 114), then the computer system attempts to de-configure the faulted components so they are not logical or operation-ally functional components of the computer system, step 116. The computer system as a practical matter should not de-configure itself below the m;n;ml7m number of com-ponents required for system operation. For example, thesystem cannot functionally remove the last system board.
If de-configurement of the computer system is not possi-ble (NO), then the system shuts itself down, step 118.
If the computer system can be de-configured (YES, step 116), then the computer system takes the necessary actions required to er.~ve the faulted component, func-tionally and operationally, from the system, step 120.

W O 97/07457 PCTrUS96/13742 The computer system preferably operates so fault isola-tion i8 at least to the FRU which failed or to the FRU on which the component believed to be faulted is located.
Isolation to the FRU makes the diagnosis and fault isola-tion process accurate, fast and complete. The systemalso updates a system memory chip (e.g., EEPROM) to iden-tify the active and de-configured components including a failed system clock.
After de-configurement is completed, the process and computer system returns to re-performing the diagnostic tests, steps 110-114. Thus, the operability and integri-ty of the de-configured system i8 verified prior to load-ing of the applications. The diagnostic testing and evaluation process is repeated until there are no faults identified or a system shutdown is required.
If no further faults are detected, or if there was no fault detected in the initial diagnostic tests (NO, step 114), then the system proceeds to load the operating system, step 122. If the operating system is successful-ly loaded (YES, step 124), then the computer system isavailable for use, namely to load and run the applica-tions program(s) of the user(s), step 126.
As provided in the following, the system outputs a message about deconfigurement or shutdown to the user and, preferably, to the appropriate service center. This includes advising of a failed system clock (i.e., 1055 of re~lln~ncy). In thi5 way, the service center and user W O 97/07457 PCT~US96/13742 are informed of the failure. Also, this allows the user and service center to schedule the replacement of the component at a time that is convenient for the user.
While the applications program(s) is being run, the computer system is monitoring for errors that may be representative of the presence of a faulted component(s).
If no errors are detected (N0, step 128), then the user continues to run the applications program. If an error is detected (YES, step 128), then the operating system and/ or the computer system determines i~ the error is fatal or non-fatal, step 130.
Non-fatal errors are those errors that do not result in an immediate shutdown of the computer system and/ or irretrievable corruption of information, data or state.
An example of a non-fatal error is a single bit error.
Fatal errors on the other hand are those errors that indicate the potential failure of component which would cause a system shutdown, those where the program running has crashed or hung or a failure in which there is a corruption o~ information, data or state. An example, o~
a fatal error is when the watch dog timer for a given operation/ component times out indicating a hung condi-tion.
If the error is determined to be fatal (YES), then the computer system is put in a condition to save comput-er system information such as that found in the system's memory and in component buffers/ registers, step 150. As discussed below, the board mounted ASICs include regis-ters ~or storing state or bus transaction in~ormation.
Also, the ASICs are in general configured to maintain their state when a fatal error is detected. However, saving this information does not require the system clocks to be frozen as is required with a number of prior art suggested techniques. In this way, the system auto-matically saves in~ormation that can be later used, e.g., in the manu~acturing/ repair ~acility, to identify the ~ailed component/ failure cause.
A~ter placing the system in its save in~ormation condition, the computer system retrieves the in~ormation concerning the state o~ the system step 152. Pre~erably, this is accomplished using the integrated JTAG test log-ic/ circuitry and scan ch~; nR to scan the information outo~ the various registers/ bu~ers and ASICs Flip/Flops (F/Fs) in the system. After retrieving the information, a warm reset is asserted, step 154.
The computer system/ operating system, where possi-ble, also take those actions that can resolve the identi-fied error 80 the computer system may be re-started by the warm reset for the purpose of retrieving the memory's contents. As such, if the warm reset is successful (YES, step 156), then the computer system causes the memory to be dumped and saved, e.g., in the system's hard drive.

After completing the memory dump, or if the system faults again after assertion of the warm reset (YES, step 156), WO 97/07457 PCT~US96/13742 then the process returns to performing the system diag-nostic evaluation process, step 102 (i.e., a cold reset asserted). Thus, before system operation is continued, computer system integrity and operability is again veri-fied.
If the error is non-fatal (N0, step 130) then the computer system/ operating system takes the necessary ac-tions to correct or resolve the initially identified error, step 132. For example, re-obt~;n;ng/ re-writing the data involved in the single bit error. For some non-fatal errors, the computer system/ operating system, as part of this process, also determines if the occurrence of non-fatal errors has exceeded a threshold criteria.
If yes, then a notation is made so the component or re-lated FRU is de-configured during the next power-up or cold reset of the system. For example, if the single bit errors for a given SIMM exceed the threshold limit, then the SIMM is tagged for later de-configurement. The oper-ation of the computer system and the running of user applications program is continued (i.e., computer system not shutdown).
There is shown in FIG. 2 a high level block diagram of the high availability multi-processor computer system 200 of the instant invention. The multi-processor com-puter system 200 of the instant invention preferablyemploys a CC-NUMA architecture as described in a co-pend-ing commonly assigned U.S. patent application Serial No.

W 097/07457 PCT~US96/13742 08/ , entitled "SYMMETRIC MULTIPROCESSING COM~I~K
WITH NON-UNIFORM MEMORY ACCESS ARCHITECTURE"
tAttorney Docket No. 46, 585, ~iled August dy, 1996) the teachings o~ which are incorporated herein by re~erence.
The multi-processor computer system 200 includes a plu-rality of board complexes or motherboards 202a-h that are each interconnected to the ~our busses 208a-d that make up the PI BUS 208. The PI BUS 208 is the bus traversing the backplane and interconnecting all the motherboards o~
the system. The ~our busses 208a-d allow in~ormation, data and instructions to be communicated among and be-tween the motherboards. Each motherboard 202 also in-cludes a diagnostic bus interface (DBI) 204 that inter-connects each motherboard to the diagnostic bus (DBUS) 206. While the illustrated computer system includes 8 motherboards 202a-h, this is not a limitation as the system can be con~igured with at least two motherboards.
Re~erring also to FIG. 3, a block diagram o~ a moth-er-board, each motherboard 202 includes two job processor (JP) daughter boards 250a,b that are plugged into the mother-board; a memory subsystem 252, an I/O subsystem 254, clock circuitry/ logic 256, a bus/ PIBUS inter~ace sub-system 258, and local resources 260. The clock circuitry 256 included with the motherboards at slot 0 normally generates and selectively provides system and test clocks W O 97/07457 PCT~US96/13742 to the motherboards in all backplane positions/ slots.
The clock circuitry/ logic 256, including its distribu-tion, is discussed further below in connection with FIGS.
9-11. Similarly, the job processor daughter boards 250a-,b will be discussed below in connection with FIG. 4.
The memory subsystem 252 on each motherboard 202 in-cludes a plurality of SIMMs 270, two Error Detection and Correction Unit ASICs (EDiiAC) 272, a directory 274 and a memory controller ASIC (MC) 276. The memory subsystem 252 is capable of providing up to 512MB of system memory on each motherboard 202 for the computer system 200.
Actual random access memory storage is provided by up to eight (8) 16Mx36 stAn~d SIMMs 270 provided on the moth-erboard 202. The motherboard 202, however, can be popu-lated with 4 or 8 SIMMs. Memory data is protected using ECC which is generated/ corrected using the two EDiiAC
ASICs 272. Each EDiiAC provides a 64 bit data path and the two are used to interleave with a cache block.
The memory subsystem 252 also includes a storage for the directory 274 which is u~ed to maintain cache coher-ency. The directory 274 includes 4Mx4 dynamic random access memories that are mounted to the motherboard 202.
The ECC codes for both the directory and the main data Rtore (i.e., the SIMMs 270) are capable of correcting all single-bit errors and detecting all double-bit errors.
The MC ASIC 276 controls the execution of physical memory operations. This involves managing both the di-rectory 274, which maintains system coherency, and the memory data store SIMMs 270. The MC ASIC 276 processes memory transaction packets that are driven onto the MCBUS
by the BAXBAR 292.
The I/O subsystem 254 on each motherboard 202 in-cludes two I/O subsystem interface ASICs (GG) 280, two Peripheral Component Interface (PCI) expansion cards 282, two Small Computer System Interfaces (SCSI) 284, and one Local Area Network (LAN) interface 286. Each of which are mounted to the mothe~ho~d 202.
The I/O subsystem 254 of each motherboard 202 com-prises two independent PCI ch~nn~ls operating at 25MHz.
Each PCI channel is interfaced to the GG bus o~ the bus/
bus interface subsystem 258 by means of a GG ASIC 280.
Each GG 280 contains an integrated cache for I/O trans-fers and also contains all the necessary logic to provide the interface between the GG bus and the PCI bus, includ-ing PCI arbitration. The GG 280 also serves as a gather-er o~ interrupts from the motherboard and connected pe-ripherals and combines these interrupts and directs themto the appropriate job processor (JP) on a JP daughter board 250a,b by means of a bus packet.
Each of the two PCI busses is connected to an inte-grated SCSI inter$ace 284 and to a single PCI expansion board 282. One of the two PCI busses also is connected to an integrated 10Mb LAN interface 286. The two SCSI in-terfaces are implemented using an NCR825 Integrated PCI--W O 97/07457 PCTrUS96/13742 SCSI controller as a pair of Wide Differential SCSI-2 interfaces. Each controller is connected through a set of differential transceivers to the 68pin High Density SCSI connector on the airdam. No onboard termination of the SCSI bus is provided allowing the connection of the motherboard 202 into Multi-initiator or other SCSI clus-ter configurations. The single LAN connection is made using the DECchip 21040 PCI-Ethernet controller. This provides a single chip integrated LAN which is connected to an RJ-45 connector on the airdam.
The bus/ PIBUS interface subsystem 258 on each moth-erboard 202 includes four PIBUS interface ASICS (PI) 290, a crossbar switch (BAXBAR) 292 that serves as the inter-connect path for the motherboard level busses, an arbi-tration ASIC (ORB) 294 and a plurality of board levelbusses. The ORB ASIC 254 controls arbitration for the motherboard level buses and controls the BAXBAR bus tran-sceiver. The BAXBAR (BB) 292 i5 implemented as four ASICS and includes a mode ~w,itch to select between the ORB ASIC functions and the BAXBAR ASIC functions.
The primary communication between job processors across the backpanel is accomplished using the PIBUS
Interface portion of the bus/ PIBUS Interface subsystem 258. A single PIBUS 208a consists of a multiplexed 72-b-it Address-CTRL/ Data bus and associated arbitration and control signals. Each motherboard 202 includes 4 identi-cal PIs 280, where each PI is interconnected to one of WO 97/07457 PCTrUS96/13742 buses 208a making up the PIBUS 208. Traffic is parti-tioned across the four busses 208a-d, so that each bus is approximately equally utilized.
As indicated above, the subsystem 258 includes a plurality of board level buses. The following lists each of the individual busses and includes a brief description for each.
RI bus: Bus that interconnects the BAXBAR 292 to the Resources Interface ASIC (RI) 306 and to the debug buf~ers/ debug connector.
GG bu~: Bus that interconnects the BAXBAR 292 to the two GG ASICs 280.
MC Bus: Bus that interconnects the BAXBAR 292 to the MC ASIC 276.
CIo Bus: Bus that interconnects the BAXBAR 292 to the Cache Interface ASIC (CI) 414 that is mounted on the ~PO daughterboard 250a.
CI1 Bus: Bus that interconnects the BAXBAR 292 to the Cache Interface ASIC (CI) 414 that is mounted on the JP1 daughterboard 25Ob.
PIX Bus: Bus that interconnects the BAXBAR 292 to the four PI ASICs 290.
MUD L, MUD H Bus: The two busses that interconnect the BAXBAR 292 to the two EDiiAC ASICs 272 of the memory subsystem.
Each motherboard 202 contains all the local resourc-es that are required of the system, with the exception of the System ID PROM 204, which is cont~;ne~ on the backpa-nel. The local resources 260 includes a microcontroller (uC) 300, EEPROMs 302, SR~M, NOVRAM, DUARTs, SCAN inter-face logic 304, M~CE l~gic, ar.d a Resources Ir.terface (RI) ASIC 306. Although the local resources 260 is du-plicated on each motherboard 202, the computer system 200 only uses the local resources section of the board in either slot 0 or slot 1 on the backplane as the system wide Global Resources. The RI ASIC 306 provides the interface between the RI Bus/ BAXBAR 292 and the devices within the local resources 260.
The microcontroller 300 performs low-level early power-up diagnostics of the system prior to de-asserting RESET to the JPs on the JP daughterboards 250a,b. It also is the controller/ engine used for all scan opera-tions. If a JP 400a,b on a daughterboard 250a needs to do a scan operation, it makes a request to the micro-con-troller 300 which then performs the required operation.
Scan also is used to configure the ASICs during power up, communicate with the power supplies and blowers, communi-cate with the various ID PROMs within the system, and to dump failure information after a hardware fatal error.
There are four 512Kx8 EEPROMs 302 that store all the JP 400a,b and microcontroller 300 firmware. The EEPROMs 302 also contain the appropriate test vectors for per-forming JTAG scan tests during power-up. A 512Kx8 SRAM

is included in the local resources 260 to be used as W O 97/07457 PCT~US96/13742 scratchpad RAM for early power-up and for microcontroll-er stack space. A 128Kx8 NOVRAM/RTC is also provided to give an extra area for storing critical information in non-volatile storage and to provide a real-time clock for the system.
The local resources 260 provides the DUARTs for implementing the three required UART ports for the sys-tem. The fourth UART port also i8 used as part of a loopback circuit to allow a JP to monitor what is being driven on the main system console.
In addition, the local resources section 260 also provides the logic 304 to do JTAG based sc~nn;ng of all the board mounted ASICs, the power supplies, the blowers, the SEEPROM and SYSID PROM. The logic is in place to allow the system to be sc~nne~ either during manufactur-ing test using an external tester or during normal opera-tion/power-up using the microcontroller 300 on the motherboard 202. This logic allows simple boundary scan testing to be used as part of the power-up system testing to detect and isolate possible faulty components (e.g., FRUs).
Additionally, the MACHs on the resource bus can be plO~ r a-u.-.ed using its JTAG interface from an external connector. Also, the microcontroller can be used with an external connector to program the EEPROMs on the resource bus. This allows manufacturing to assemble the boards with blank MACHs and EEPROMs and then "burn" them as part of the test procedure, rather than stocking "burned"
versions of the parts to be installed during assembly.
This "incircuit programmabilityll feature also makes up-dates for ECO activity as simple as plugging in the pro-gr~mm; ng connector and re-programming the parts, rather than le...~ving the old part and installing a new part in its place.
Referring back to FIG. 2, the computer system 200 also includes a group of three power supplies 210a-c for each pair of motherboards 202, three blowers 212 that cool the system, and a system ID SEEPROM 204. When there are eight motherboards 202, there is a total of 12 power supplies. The three power supplies 210a-c in each group represents N+1 re~lln~nt power supplies for the pair of motherboards. Also, the three blowers 212 represent N+1 r~lln~nt blowers for the system 200.
The power supplies 210a-c for each group also are interconnected to each motherboard 202 of the correspond-ing pair of motherboards. In this way, and as described below, each motherboard 202 has the capability to ascer-tain (e.g., scan, diagnose) the operability status of the power supplies for that pair of motherboards. The blow-ers 212 also are interconnected to the motherboards 202 in slots 0 and 1 on the backplane. In this way, and as ~ 25 described below, the motherboards 202 in these slots have the capability to ascertain (e.g., scan, diagnose) the operability status of the blowers 212.

The System ID SEEPROM 204 provides a non-volatile place to store important system in~ormation such as seri-al number and back panel configuration. Because the System ID SEEPROM 204 does not have a true JTAG inter-face, it cannot be connected directly to a IEEE 1149.1scan chain (see discussion below). Thus, a buffer is used to provide the interface between the two serial protocols.
Referring to FIG. 4, there i5 shown an exemplary block diagram for a JP processor daughter board 250.
Each JP daughter board includes two 50 MHz Motorola 88110 central processing units or job processors (JPs) 400a,b, each JP having associated therewith lMB of static random access memory (SRAM) as a level 2 cache 402a,b and a 88410 cache controller (SLCC) 404a,b. Also mounted on each daughter board 250 is 16MB o~ dynamic random access memory (DRAM), a third level cache (TLC) 406 and a third level cache controller (TLCC) ASIC 408 which controls the TLC. The third level cache 408 is shared by both JPs 400a,b. The DRAMS are protected by ECC, which is gener-ated and checked by two EDiiAC ASICS 410 that are under the control o~ the TLCC ASIC 408. The cache tags ~or the third level cache 406 are stored in a SRAM 412.
Each JP daughter board 250 also includes a Cache Interface (CI) ASIC 414. The main function of the CI
ASIC 414 is to serve as a translation/ sequencer between the packet-switched local bus protocol on the motherboard WO 97/07457 PCT~US96/13742 202 and the 88410 cache controller bus protocol on the JP
daughter board 250. All off JP daughter board communica-tions, with the exception of clocks and reset, are part of the motherboard level buses and the CI is directly connected to the CIBUS.
The two EDiiAC ASICs 410 are interconnected to the daughter board level buses via 8ix ABT16260 latching 2:1 muxes 416. For purposes of multiplexing, the 32 bit S A
Bus and the 32 bit S D Bus are multiplexed into the S AD
bus by means of four LVT162245 bus crossovers 418.
Each daughter board 250 includes a SEEPROM 420 that provides a non-volatile place to store important daughter board information such as the board number, serial number and revision history. Because the SEEPROM 420 does not have a true JTAG interface, a buffer 422 is used to pro-vide the interface between the two serial protocols.
Each of the above described ASICs uses a Phase Locked Loop (PLL) based "gated" balanced clock tree (BCT) design, as shown in FIG. 5. ASIC clock control is han-dled by the Test Access Port (TAP) module and ASIC Clock/Reset (CLK RST) module 602 of each board mounted ASIC.
The BCT circuitry/ logic 600 includes a clock trunk 604 that has a plurality of branches therefrom. All but one of these is a gated branch 606 that includes an AND gate 608 which is controlled by the enable function ASIC_CLK_-EN from the CLK RST Module. FIG. 6 illustrates how the BCT is connected in the ASIC hierarchy (GlOOO level).

WO 97/07457 PCT~US96/13742 The uncontrolled branch 610 is a "free running" leg of the clock tree and it provides a delayed clock input to the PLL 612 for clock deskewing. The clocks being inputted to the PLL 612 are passed through a delay cell 614, that adjust the feedback clocks 50 they are repre-sentative of the travel time delay through a set branch length. Preferably, the delay cell 614 i8 configured/
set so the expected travel times or branch line lengths for the longest and shortest branches lies within the acceptable range for the set branch length/ time. As such, the PLL feedback path is always active 80 the PLL
612 can stay in synch with the ASIC's reference clock input CLK. The uncontrolled branch also feeds a LOOP-BACK_CLK 616 to the Clock Reset (CLK RST) module 602 that is used for synchronization of the Test clocks (TCK).
The PLL module, as shown in FIG. 7, is a common module for all system ASICs that use the above described BCT ASIC design. The ATE_TEST input is used to control test modes for the VCO logic. The EN, TSTN and IDDTN
signals need to be controlled at the ASIC pin level dur-ing manufacturing package testing so all PLL based ASICs provide these pins discretely or share this pin with some other input pins. This saves on pin count. The ATE_TEST
input pin is used to enable shared pins pin level control of these signals. For example, the Pl shares the EN func-tion with the PI_ORDERED_OP input and the IDDTN function with the PI_MED_CUSTOMER input.
The ASIC Clock/ Reset (CLK RST) module 602 generate6 the ASIC internal reset and clock signals as well as the enable function (ASIC_CLK_EN) which causes the F/Fs in the ASIC design to see only certain clock pulses. There is shown in FIG. 8, a block diagram of the circuitry/
logic ~or ASIC internal reset and clock generation. The CLK RST module 602 ~or each ASIC operates in one o~ three modes; Normal Mode, Scan Mode and Reset Mode. If not in one o~ these three Modes, then the CLK RST module gener-ally acts so as to mask out or block the clocks ~rom reaching the F/Fs in the gated branches 606.
In the Normal Mode, the CLK RST module 602 continu-ously generates the ASIC_CLK_EN enabling ~unction. Thus, the AND gates 608 in the gated branches 606 o~ the BCT
are con~igured so the clocks pass therethrough. If a FATAL_IN_N is asserted by the computer system 200, then the gates 608 ~or all board mounted ASICs, except ~or the MC ASIC 276 (FIG. 3), are re-con~igured 50 as to block or mask out the clocks. Essentially, the ASIC_CLK is ~ree-running until the ASIC experiences an assertion of FA-TAL_IN_N. The clocks remain blocked/ masked out until the Scan Mode or Reset Mode is entered.
In the Scan Mode, the JTAG TAP controls the ASIC
clocks. ASIC_CLK is allowed to pulse once every 80 nano seconds (nsec.), but only i~ the TCK_EN enable signal is asserted ~rom the TAP. The Scan Mode overrides Reset Mode or Normal Mode. When in Reset Mode, the ASIC_CLK
also i8 allowed to pulse once every 80 nsec. This allows the board mounted ASICS to be synchronized and reset all at the same time. The Reset Mode overrides the Normal Mode.
When either the reset or scan inputs to the CLK RST
module 602 is asserted, the clocks generally are switched ~rom the normal system clocks (SYS CLK), e.g., the 50 MHz clocks, to the test clocks (TCK), e.g., the 12.5 MHz clocks. However, ~or the MC ASIC 276, only a COLD_RESET_N will switch the clocks; WARM_RESET_N will be registered on TCK mode and is expected to make setup to all ~lops in one 20nsec cycle.
The clock enable is generated such that the ASIC_CLK
will switch from SYS CLK to TCK at the end o~ the TCK
cycle in which the external reset is asserted. The inter-nal reset signals will not assert until two TCK cycles a~ter the external reset asserts. When the external reset is deasserted, the internal reset deasserts two TCK cy-cles later.
The TCK_EN is used ~or scan ~unctionality. It mustbe asserted in order to pass TCK through the clock gating logic. TCK_EN is generated in the TAP controller.
As such, there are three basic modes o~ clock opera-tion that af~ect the internal ~lip/flop (F/F) clocks.The ~irst mode iB where all oi~ the logical F/F clocks are synchronized to the CLk/CLK_N input clocks phase using the on chip Voltage Controlled Oscillator (VC0) in the CM_PLL module. The second mode i8 where the internal F/F
clocks are Rtopped on detection of an error detected via ~_FAT~T_I~.T_N input pin. Thi~ i~ done for all board moun-ted ASICs except the MC ASIC 275 (FIG. 3). The last modei5 where a particular TAP instruction i5 loaded into the TAP instruction register and results in the internal F/F
clocks being clocked in phase with the TCK input pin when the TAP controller is in the "Capture-DR" and "Shift- DR"
TAP states. This last mode of operation is used to scan initialize or scan dump the ASIC state.
In sum, the "free running" running leg of the clock tree allows the internal F/F clocks to be stopped on detection of a fatal error while keeping the system clocks running and the PLL maintained in synch with the ASIC's reference clock input CLK. As such, when an ASIC
is being scan dumped for its state, the ASIC clocks will be in sync with the clock~ of the computer system. This is in comparison to known BCT designs where such a clock stop/ restart would require the PLL to re-~ync itself and thus prevent scan dumping of the ASIC state. Such a design also allows the system clock to continue providing clock signals to the other components of the computer system (e.g., microcontroller 300- FIG. 3) particularly those that are used for Rc~nn;ng and diagnosing of the computer system following receipt of a fatal error as hereinafter described.

WO 97/07457 PCT~US96/13742 As indicated above, the internal F/F clocks of the MC ASIC 276 (FIG. 3) are not stopped by a X_FATAL_IN_N
input. This is done because stopping the internal F/F
clocks would destroy the memory image being stored on the motherboard 202 (i.e., interfere with memory refresh).
The memory image is maint~; n~A 80 the core can be dumped which is required for operating system (i.e., DG/UX) debug. The MC ASIC 276 does support a mode where the internal F/F clocks can be stopped for lab debugging.
On receipt o~ an X_FATAL_IN_N input representative of the detection of a fatal error somewhere in the sys-tem, the MC ASIC 276 will abort the current operation and will remai~ idle except for re~reshing the memory ti.e.,-DRAMs/ SIMMs). The MC ASIC's input and output queues are cleared and some internal state machines are reset to idle. The MC ASIC will not respond to any bus activity until it receives a warm reset. After the warm reset, the MC ASIC's control-space registers can be read to get error information that was saved when the fatal error was detected.
Because the MC ASIC 276 cannot be sc~nne~ to collect information about a fatal error, it freezes copies of some of its current state into shadow registers when a fatal error occurs. Shadow registers are copies only and ~reezing them does not affect normal ASIC behavior. Many of these shadow registers are control space accessible.
The information in the shadow registers rem~;n~ valid through a warm reset and will not change until after the error registers have been cleared by specific control space writes. The other ASICs on the motherboard 202 and the daughter boards 250a,b also include shadow registers for freeze copying some state information therein.
When a fatal error originates at the MC ASIC 276, the MC ASIC immediately freezes shadow copies of internal state relating to the operation currently being executed.
This allows state to be captured before it advances with-out using several levels of shadow registers. Shadowregisters cont~;n;ng less volatile state are not frozen until the MC ASIC generated fatal error is sent back to the MC ASIC as a system fatal error.
As indicated above, and for purposes of maintaining a high level of availability, the computer system 200 of the instant invention includes re~lln~nt system clock circuitry/ logic. Preferably, the clock circuitry/ logic 256 generates the system clocks and test clocks, monitors the clocks being generated to identify a failure of the clock generation and distribution circuitry and is pro-vided on each motherboard 202. Such circuitry and logic is provided on each motherboard 202 at least to simplify manufacturing and the stocking of spares. In a specific embodiment, the computer system and test clocks are sour-ced to the clock circuitry/ logic 256 on one of the moth-erboards in backplane slots 0 and 1. In this arrange-ment, circuitry/ logic on the motherboard in slot 0 nor-W O 97/07457 PCTrUS96/13742 mally sources the system and test clocks and the circuit-ry/ logic on the motherboard in slot 1 i8 the backup clock source.
There is shown in FIGS. 9-10, the clock circuitry and logic that generates the system and test clocks as well as monitoring the clocks being generated to detect a failure. The m~nn~ in which these clocks are distribut-ed to a motherboard 202 and its JP daughter boards 250a,b is shown in FIG. 11.
The clock circuitry/ logic 256 includes two oscilla-tors. The primary oscillator 500 is for the normal clocks and in a specific embodiment is a 100 MHz crystal.
The other oscillator, the margin oscillator 502, provides a 5~ fast margin feature and in a specific embodiment is a 105 MHz crystal. Each of the oscillators 500,502 drives a divide by two circuit 504 to generate a 50~ duty cycle clock (e.g., 50 MHz and 55 MHz clocks). The out-puts of the divide by two circuits 504 are inputed to a two to one mux 506, which is preferably configured so its output is the normal clocks (e.g., 50MHz clocks). Howev-er, the two to one mux 506 does allow the clocks to be sourced by the margin crystal 502.
The clocks from the two to one mux 506, the system clocks, are provided to an E111 one to nine clock fanout circuit 508. One output from the one to nine fanout circuit 508 is feed to an E431 divide by four circuit 510 that is driven to form the test clocks synchronizing W 097/07457 PCT~US96/13742 signal (test clock sync signal) which in turn i6 provided to an E111 one to nine clock fanout circuit 512. Both the system clock and test clock sync signal drive their respective E111 fanout circuits 508, 512 80 as to fanout and distribute the system and test clock sync signal (across the back panel) to all the motherboards 202.
All ~ystem clocks are fanned out with equal line lengths to all the motherboards 202 where they are routed through more E111 one to nine clock fanout circuits 514.
These in turn drive the board mounted ASICs and TTL clock buffers which drive the JPs, SLCCs, EDAC's, FPGA's, and other elements requiring TTL clocks. On each ASIC, the system and test clocks are automatically deskewed with a PLL, so the clock delay through all ASICs is compensated.
The TTL clock buffer, like the ASICs, also has a PLL to compensate for both the delay through the buffer and ~or the etch and load on the buffer~.
Similarly, the test clock sync signal i8 fanned out to all the motherboards 202. The test clocks generated on the motherboard are fanned out to all components requiring a test clock via an E111 one to nine fanout circuit 512 and H641 PECL to TTL level translator~ 516. The test clocks are used for both scan and reset control and to control the resource section of the board~.
The system clocks also are used to generate the PCI

clocks on each motherboard 202. This is accomplished by W O 97/07457 PCTrUS96/13742 inputing the system clocks to a divide by two circuit 518. The PCI clock preferably operates at 25 MHz and ~ans out to all PCI bus circuits, including the GG ASIC
280. The GG ASIC 280 synchronizes the 25 Mhz clock with ASIC_CLK (ASIC' 8 internal version o~ the System Clock) ~or use in gating the D inputs to internal F/Fs. This i8 accomplished by ~irst clocking on a ~alling edge o~ ASIC-_CLK and then clocking on a l~; ng edge o~ ASIC_CLK.
The clock circuitry/ logic 256 also includes clock ~ailure detection circuitry 210 that senses a stoppage of signal pulses ~rom the primary oscillator 500 or distri-bution logic. The detection circuitry also is capable o~
sensing other ~ailures such as a stuck high or low condi-tion. The clock ~ailure detection circuitry 520 includes ~our E431 F/Fs 522a-d, two OR gates 524 and an E431 di-vide by 2 F/F.
An output ~rom the E131 divide by two 404 for the margin clocks (e.g., 55MHz) provides clock input ~or the E431 divide by two F/F 526. The output ~rom the E431 divide by two F/F 526 provides clock input ~or two o~ the E431 F/Fs 522a-b (i.e., sets the F/Fs). The motherboard level clocks are also ~eed to the reset o~ these two E431 F/Fs 522. In this way, a ~ailure o~ the primary oscilla-tor 500 (e.g., no output) will result in the ~ailure o~
the F/F reset to be asserted.

The OR gates 524 and the r~m~;n;ng E431 F/Fs 522c-d are interconnected to the outputs ~rom the E431 divide by two 526 and the ~irst two E431 F/Fs 522a-b such that, when there i8 a Eailure to generate system clocks ~rom the primary crystal 400, the third E431 F/F 522c provides an output representative o~ the ~ailure. The ~ourth E431 F/F 522d generates an output to the back panel to ~orce selection o~ the backup motherboard's clock circuitry logic 256, i.e., the slot 1 motherboard.
The motherboard at either backplane slots O or 1 can switch or re-designate the source of the clocks ~rom the normal clock source (i.e., circuitry on motherboard in slot 0) to the backup clock source. This is accomplished by the assertion o~ a wire-or PE~L_MUX_SEL_OUT wire on the backpanel. The clock source is switched to the backup clock and the resources section on the motherboard desig-nated as the Diagnostic Master will force the system togo through COLD_RESET_N.
Once the backup clock has been selected, the clock ~ailure detection circuit 520 will continue to monitor the clocks source ~rom the backup clock circuitry/ logic.
I~ a ~ailure is identi~ied while the PECL_MUX_SEL_OUT is asserted, the microcontroller 300, on each motherboard 202 that detects the error, will activate the TN pins on all o~ the board ASICS. This tri-states the ASIC out-puts and eliminates part damage due to extended bus ~ig-hting. I~ there is only one motherboard 202 in the com-puter system 200, and the clock ~ails, the clock ~ailure detection circuitry 520 will cause the TN pins to be -activated, thereby protecting the board components.
Because the microcontroller 300 ha6 the capability to switch clock sources ~rom nominal to margin clocks, one can bring the motherboard 202 back on line at margin speed.
The computer system 200 o~ the instant invention scans the boards, board mounted chips, busses, blowers and power supplies comprising the system to verify the integrity and operability o~ the system be~ore applica-tions are loaded. Such scAnn;ng is done when the systemis powered up and also a~ter the system detects a ~atal error. The scAnn;ng operation includes automatically detecting a $ault, automatically isolating the ~ault to at least a FRU and automatically de-con~iguring the sys-tem so as to logically and ~unctionally ~~uloved the iso-lated component/ FRU. A~ter the system is decon~igured, the system automatically re-boots itsel~ and re-loads any applications program. In this way, the system has the capability to automatically recover the system a~ter it has su~ered a ~ailure event without requiring interven-tion o~ the user or service representatives.
The board mounted ASICs and the boards are designed with integrated JTAG IEEE 1149.1 test logic so test pat-terns can be shi~ted into one device, driven onto the logic board and captured at another device. In this way, board interconnect can be veri~ied. The standard also provides ~or the veri~ication o~ proper component selec-WO 97/07457 PCTrUS96/13742 tion and proper device insertion. The computer system will implement both the boundary scan and ~ull scan im-plementations o~ the IEEE st~n~d.
There is shown in FIG. 12 the scan the scan Ch~i n~
and associated logic ~or a portion o~ the computer sys-tem, speci~ically those for the motherboard 202 located in slot "0" on the backplane. The scan ch~; n~ ~or the motherboards in the other backplane slots are established based on the components which are to be scanned by that motherboard (e.g., board/ board level components and power supplies). Scan ch~;n~ will be used as a basis ~or test of the boards and board mounted ASICs. Board-level opens, adjacent pin bridging, proper component selection and insertion will be tested via 1149.1 boundary scan.
Each motherboard 202 includes a microcontroller 300 that is responsible ~or reset generation, scan-based powerup interconnect testing and ASIC testing. The micr-ocontroller 300 shares the Resources bus with the RI ASIC
306. When reset is ~irst applied, the microcontroller 300 asserts a signal to the RI ASIC 306 that ~orces the RI
ASIC o~ the Resources bus. The microcontroller 300 can then test the bus and use the Test Bus Controller (TBC) 700 ~or scan-based tests. Following powerup interconnect and ASIC scan-based tests, the microcontroller 300 deass-erts the control signal to the RI ASIC 306, allowing theRI ASIC to provide normal system access to the Resources bus and its associated devices.

Scan chA; n~ will be used to identi~y all boards and board mounted ASICs in the computer system. Speci~ical-ly, the presence of each motherboard 202 and the associ-ated JP daughter board6 250 will be detected and board serial numbers and the system ID will be read via scan.
Also, ASIC part numbers and revision numbers will be read on powerup via scan. For EEPROMS and SEEPROMS, a buffer is included so information can be extracted during the 5~nn;ng process.
Scan ~-h~; n~ also are used to communicate with the power supplies 210a-c and the cooling blowers 212. Power supply status functions, e.g., turned on, over tempera-ture, over/under voltage, and disabled, and control func-tions, e.g., fault masks and disable, are also performed through scan. Blower status, e.g, high speed, low speed, ambient over temperature, turned off and control, e.g., ~ault masks, speed up, disable) is commlln;cated through scan.
The scan interface part is the Test Bus Controller (TBC) 700, an off the-shelf part no. 74ACT8990 available from TI (see also Texas Instruments, Advanced Loqic and Bus Interface Loqic Databook", 1991). The TBC 700 func-tions as an 1149.1 controller, where parallel data is written to and read from the TBC, while serial 1149.1 data is generated and received. The TBC reside6 in the local resources 260 of each motherboard. The microcontr-W O 97/07457 PCTrUS96/13742 oller 300 use6 the registers inside the TBC to perform system scan operations.
To perform a scan operation, the PROM code loaded in the microcontroller accesses the registers inside the TBC
700. Data will be transmitted in order to specify the target board, the target device, and the operation to be performed. Operations will consist of scan chain shift operations, loopback operations, and others as they be-come known.
The TBC 700 uses the Addressable Shadow Port (ASP) to select and communicate with one or all of the board scan ch~; n~ in the system. Each ASP has hardwired node address input signals that allow the ASP to distinguish messages directed to its local chain. The node address inputs are based upon the "node ID" that is hardwired to the motherboard's backpanel connector. A "broadcast"
address can be used that allows the master TBC to commu-nicate with all board scan ~h~; n~ in the system, should this be desired. The ASP protocol is detailed in A Pro-Posed Method of Accessinq the 1149.1 in a Backplane Envi-ronment, Lee Whetsel, International Test Conference, 1992, the teachings of which are incorporated by refer-ence.
The Diagnostic Bus Interface (DBI), a PAL, interfac-es the TBC with the backpanel DBu~ 208 and the ASP. The DBI's job is to allow three modes of operation: local mode, remote mode, and manufacturing test mode.

WO 97/07457 PCT~US96/13742 In local mode, data ~rom the TBC 700 is sent direct-ly to the local scan chain and data from the local scan chain is sent directly to the TBC. The DBus 206 is not disturbed by local scan operations. As such, the mother-boards can operate in the local scan mode in parallel andsimultaneously. These simultaneous scan operations will occur during powerup testing.
In remote mode, data ~rom the TBC 700 is sent out to the DBus 206 and data on DBus is returned to the TBC.
The local scan chain is also connected to DBus 206 through the ASP part. This allows the TBC 700 to address any motherboard in the system as well as its own. When the address broadcast by the TBC 700 onto DBus 206 match-es a board's hardwired address, or a bro~c~t address, the ASP connects the local scan chain to the DBus.
In manufacturing test mode, the DBus 206 is connect-ed directly to the local scan chain and the TBC 700 and ASP are not used. This mode allows manu~acturing to directly apply test vectors to the local scan chain through the backpanel DBus signals. o~ no interest to the current scan operation may be placed in bypass mode.
The Master microcontroller uses the TBC 700 and DBI
to communicate over the backpanel diagno~tic bus DBus 206 and to the DBI/ASPs on the other motherboards. The mas-ter TBC transmits a n select" protocol to address andenable the ASP connected to the slave DBI o~ the selected board(6). Once selected, the master and slave are "con-CA 0222944l l998-02-l2 W 097/07457 PCT~US96/13742 nected" and the master TBC can perform scan operations transparently on the remote board(s), using st~n~d IEEE
1149.1 protocol.
The DBus 206 consists o~ the four standard IEEE
1149.1 signals (TCK, TDI, TD0, and TMS) and a "Diagnostic Interrupt Request" signal, DIRQ_N. During system power-up, DIRQ_N is used in determining diagnostic mastership.
During normal system operation, the DIRQ_N is used by the Power System components to interrupt the master microcon-troller.
A number o~ IEEE 1149.1 :Eeatures help to ensure that test circuitry does not inter~ere with normal system operation, should signals become stuck. Logic levels are chosen such that, in the event o~ a broken driver, sig-nals ~loat to levels that cause the test inter~aces to go into 'Test Logic Reset' mode. When the test inter~aces are reset, the test logic is placed in a benign state. A
"driving DBus" LED also is included on each motherboard to assist-in isolating some types o~ DBus faults. It will be lit whenever a board's DBI i8 driving any DBus output.
The DIRQ_N DBus signal serves a number o~ ~unctions.
As mentioned above, it is used to determine diagnostic mastership during the powerup procedure.
During normal system operation, the DIRQ_N signal is used by a power supply 210 or blower 212 to attract the attention o~ the master microcontroller. When the device detects a ~ault condition, it sets its DIRQ register bit and the interrupt i8 forwarded to the master microcon-troller by driving DIRQ_N on the backpanel DBus 206. The master microcontroller then attempts to access the TBC
700 through the normal motherboard level busses on the DM's motherboard and the RI ASIC 306.
If the moth~hoard level busses function normally, the master microcontroller can successfully access its master TBC 700. The master microcontroller begins scan-ning the slave DBIs in the computer system 200 to deter-mine which device asserted DIRQ_N. When a device's DIRQregister bit is found asserted, it is cleared and the device is serviced.
If a hard failure occurs on the motherboard level busses, the DM may not be able to access the master TBC
or the watchdog timer (WDT), which is also in the local resources 260. In this case, DIRQ_N will remain asserted on DBus 206 for an unacceptably long time and the comput-er system will hang. When the hardware WDT timer times out, NMI, warm reset, and finally cold reset will be successively invoked to break the hang. During the ensu-ing powerup testing associated with cold reset, the hard failure will be isolated and a new diagnostic master may be chosen.
As indicated above, for purposes of providing high availability, the power supply system for the computer system is designed as an (N+l) r~nn~nt configuration.
Where pairs of motherboards share three power supplies 210a-c, but require only two. A single mother board uses two supplies, but only requires one. Each power supply f~eatures a 74BCT8373 sc~nn~hle latch part that is ac-cessed by the local scan chain 702 on one of the two related motherboards. Fault status can be scanned out, and control comm~n~ can be sC~nn~ in. Power supplies 210a-c can be disabled, after a delay, and the DIRQs can be masked.
Because PIBus electrical characteristics dictate that the motherboards are populated left-to-right, an even-slot-numbered motherboard will always be present, e.g., slot 0, in an even/odd motherboard pair being pow-ered by the (up to) three power supplies 210a-c. There-fore, the even-slot-numbered motherboard's local scan chain will always be used to communicate with the power supplies for that motherboard pair.
In addition to co~nn;cating with power supplies, the local scan c~; n~ on the master microcontroller's motherboard are used to communicate with the three blow-ers 212 and the system ID buffer. The system ID isstored in a SEEPROM part 204 that is socketed on the backpanel. The SEEPROM also will be used to store system history in~ormation.
The blowers 212 of the computer syBtem 200 also are configured as (N+l) r~lln~nt. Three blowers are provid-ed, but only two are needed. The master microcontrol-ler's motherboard local scan chain 704 communicates with CA 0222944l l998-02-l2 WO 97tO7457 PCT~US96/13742 the three blowers of the computer system 200. The two main functions provided for the blowers are notification of blower faults and adjustment of blower speed.
When a blower fault occurs, circuitry on the blower interface detects the fault and sets a bit in a IEEE
1149.1 sr~nn;lhle register. The circuitry also sets the blower's individual DIRQ register bit, which causes the backpanel DIRQ_N siqnal to become asserted. This is then handled by the master microcontroller as previously de-scribed for the power supplies. The speed of the blowerscan be adjusted by means of a sc~nn~hle register part on the blower interface board.
In specific embodiments, the scan test suite of the instant invention ~or the design of the board mounted ASICs includes using the Synopsys Test Compiler. The compiler supports 6 different scan methodologies: muxed flip-flop, clocked scan, single latch LSSD (Level Sensi-tive Scan Design), double latch LSSD, clocked LSSD, and auxiliary clock LSSD (see Scan Methodoloqies, Marshall Wood, the teachings of which are incorporated herein by reference). In a preferred embodiment, the muxed flip--flop methodology is used. Because muxed flip-flop uses the normal system clock, the test access port ("TAP") controls the system clock enable within the each ASIC.
All board mounted ASICs of the instant invention will be IEEE 1149.1-compliant and include a TAP and asso-ciated pins TCK, TMS, TDI, and TDO. All ASICs and the other 1149.1-compliant components on each motherboard are incorporated into a single logical 1149.1 chain. The TDI
and TD0 signals will be cascaded from one device to an-other and the required signals TCK and TMS are buffered appropriately.
All 1149.1 compliant devices of the computer system include a boundary scan register. This provides a hard-ware-assisted means of verifying that the correct parts are properly inserted on the board. Further, this veri-fies that the I/0 drivers, boundary scan cells and bond-ing wires in the parts are functioning correctly and that board etch between parts is intact.
Each board mounted ASIC, will include a device iden-tification register, e.g., the 32-bit "Device Identifica-tion Register" of the IEEE 1149.1 st~n~d. The device ID register for each board mounted ASIC contains a unique code that identi~ies the part (e.g., as a DG part, some derivative of the DG part number) and the revision number of the ASIC. At least some of the board mounted ASICs include design-specific Test Data Registers (TDRs) that are accessed through the TAP.
The board mounted ASICS also are designed to allow full scan so that every logical ~lip-~lop in the ASIC
will be sc~nn~hle. This feature provides a number o~

bene~its, including the ability to scan-access machine state following fatal errors and improved coverage during W 097/07457 PCT~US96/13742 chip testing. The Full Scan TDR will access the ~ull scan circuitry embedded in the ASICs.
Dedicated TDRs will provide control and status ~or hardware-detected errors within the ASIC and the circuit-ry it controls (iE any). Each ASIC' 8 Error Status TDRwill also have a read/write bit that indicates, when asserted to a logic 1, that the ASIC is currently driving its FATAL_OUT_N signal. The external signal is active--low (logic 0) and is driven to the ORB ASIC, where it is combined with other ASICs' fatal signals and distributed through-out the system.
The Error Mask TDR contains one bit for each error detected by the ASIC, and one bit for FATAL_OUT_N, where a "1" masks the error out (normally register written with all 0's). Each ASIC' 8 Error Mask register will be de-signed such that the bit positions for each error's mask correspond to the bit positions in the Error Status reg-ister. The Error Eorcing TDR contains one bit ~or each error detected by the ASIC. A logic "1" in this register causes the ASIC to generate an error of that type ( nor-mally register contains all '0's).
The status bit for each error will be read/write, active high, and will indicate, when asserted, that the error has been detected. Therefore, a "0" in this bit means that no error has been detected and re~; ng "all zeros" ~rom the Error Status TDR will indicate that no errors have been detected.

W 097/07457 PCTAJS961137n The following describes how the scan ch~;nR will be used in connection with the testing of the computer sys-tem 200. Following deassertion of powerfail reset, the microcontroller 300 on each motherboard 202 takes control of the Resources bus and the associated DBI, thereafter having reign over the scan environment for that mother-board. First, the microcontroller 300 tests the Resourc-es section to ascertain its integrity. Then the micro-controller 300 initializes the scan environment by using the TBC to place all the TAP controllers in the 1149.1 devices into the Test-Logic-Reset state. TMS will be forced to logic 1 for at least 5 TCK cycles.
Next, the microcontroller 300 begins performing interconnect tests on the motherboard. This i5 done by fetching test vectors stored in on-board EEPROM 302, and applying the vectors through the DBI. Failures in these tests can be reported through console messages, and by lighting a failure LED on the board.
Finally, the microcontroller 300 performs ASIC test-ing by applying test vectors through the TBC 700 to theboard mounted ASIC. These test vectors also are stored in the EEPROM 302. When this phase of testing is com-plete, the microcontroller 300 gives up control of the Resources bus and allows normal processor requests to come through the RI.
The microcontroller for each motherboard begins sizing the computer system by again placing the TAP con-W O 97/07457 PCTrUS96/13742 trollers in the Test-Logic-Reset state. In this state, the TAP controller loads the IDCODE instruction (or BY-PASS, if IDCODE is not supported) into the 1149.1 Inst-ruction Register. The IDCODE instruction causes the device ID register to be connected between the TDI and TDO pins of each device. BYPASS selects the single bit (~0~) "bypass"register. The device ID register is 32bits long and begins with '1'. These bit definitions allow a bit stream from multiple devices to be parsed for device IDs.
The TBC 700 will use the Scan Path Linker (SPL) devices or scan bridges to multiplex the corregpon~; ng optional TDI/TDO chain into the main TDI/TDO chain for the motherboard 202. The SPL devices feature parallel input signals that can be connected to "board present"
signals. The microcontroller can read these signals and connect the appropriate JP daughterboard scan ch~;nR.
There are a number of ways to verify that the JP daugh-terboard ch~; n~ are connected. When microcontroller reads the device ID registers from all the devices on the chain, it can determine if devices known to exist only on the JP daughterboard are present, or simply count the number of devices scanned. Either method can determine that optional devices are present. Ordering of hardwired scan components and option card componentR will be chosen such that an option card's position can be determined with certainty.

WO 97/07457 PCT~US96/13742 I/O daughterboards (e.g., PCI Expansion cards 282) can be detected by polling the PCI ~hAnnels for devices.
The microcontroller can then decide if the I/O daughter-board(s) should be muxed into the TDI/TDO chain.
Each motherboard 202 sizes itself by checking for daughterboard options and checking all ASIC revision~.
The microcontroller can temporarily store the sizing information in a concise format in an easily-accessed on-board location, e.g, the RAM in the local resources 260. The DM can read this RAM later and build the over-all system configuration table.
To determine if a motherboard remote to the master microcontroller is present, the master microcontroller transmits an ASP "select" protocol to the motherboard's address. If an ASP "acknowledge" response is received, then the remote motherboard is present and the microcont-roller may begin communicating with it. If no acknowl-edge response is received, it will be assumed that the addressed motherboard is not present in the system. No further communication with this address will be attempt-ed.
Devices available only through the slot 0/slot 1 DBI
can be sized in the same m~nn~r as the motherboard i~
sized for optional daughterboards. The backpanel SEEPROM
should always be found present. If it is not present, then a break will be detected in the chain at the SEEPROM
component's position and the SEEPROM will be reported missing. Each blower or all blowers is connected to one of the slot 0/slot 1 DBI's optional TDI/TD0 ch~;n~ It i8 important to note that the sizing process at this point uses only dedicated diagnostic hardware. The "nor-mal" system logic, such as the PI arrays, does not par-ticipate in this phase of sizing.
As indicated above, the computer system 200 is de-signed to isolate the fault to at least a FRU and then deconfigure i.e., functional and logically remove the FRU
from the system. To deconfigure a section of logic, the 1149.1 optional HIGHZ instruction is invoked and when used, the instruction places all outputs in a high--impedance state. Because every board mounted ASIC imple-ments this instruction, a fine granularity of dynamically reconfigurable units t"DRUs") will be achieved.
To deconfigure a JP daughter board 250, the CI ASIC
414 and the TLCC ASIC 408 mounted thereon is placed in HIGHZ mode. Alternatively, the JP 250 also can be placed in reset through a control register in the CI 414. To deconfigure a motherboard 202, all PI ASICs are placed in HIGHZ mode. Because the motherboard also has I/0 devices on it, the I/0 bus(ses) should be disabled via a control register in the GG ASIC 280.
During the powerup process, of the board mounted ASICs (e.g., PI ASICS 290) are initialized by means of the sC~nn;ng process. At a later stage of the powerup testing proce6s, ID6 and memory ranges are loaded into the ASICs by control space writes.
As provided above, the EEPROMS 302 include the ~irm-ware ~or the microcontroller 300 and the JPs 250, partic-ularly the JP designated as the diagnostic master and themicrocontroller designated as the master microcontroller.
The ~irmware ~or the microcontroller includes powerup testing, scan testing, error handling, system snif~ing during run-time, and sc~nn;ng error state when system fatal error occurs. The JP firmware includes powerup tests, XDIAG tests, manu~acturing mode tests, and error handling.
The microcontroller 300 is the first piece of hard-ware in the computer system 200 to be used ~or firmware diagnostics. It comes out o~ cold reset and is able to test itsel~ and its resources while the rest o~ the sys-tem is being held in warm reset. It initiates JTAG boun-dary scan on all o~ the sc~nn~hle components in the sys-tem and will initialize internal scan state as well. It is responsible ~or bringing all other sections o~ the system out o~ reset in an individual m~nner~ to keep isolatability at a high level. It monitors the per-board powerup tests, controls the Watch dog timer (WDT) mecha-nism and takes care o~ EEPROM ~ h;ng During runtime, when the operating system is up, the microcontroller 300 is responsible ~or sni~ing tasks to make sure system operations are still ~unctioning properly.

W O 97/074~7 PCT~US96/13742 A tabulation of the powerup tests performed by the microcontroller 300 is provided in FIGs. 13A, B. The sequence of events executed by each microcontroller 300 on each motherboard 200 during powerup i8 a8 follows.
Each microcontroller 300 comes out of reset at ap-proximately the same time. Each microcontroller then checksums its on-chip EEPROM and tests its access to on-chip SR~M. IP as an error is detected, then the microc-ontroller executes a generic error handling function. An attempt also is made to set the resource bus LBD to pro-vide an indication of a failure. The microcontroller then execute a STOP instruction.
Each microcontroller initializes its on-chip hard-ware modules. Once its on-chip UART is initialized, code revision information and cause of microcontroller reset are printed to the individual auxiliary consoles.
Each microcontroller runs on-board diagnostics for the following sections of the Resource Bus hardware: TBC
700, SRAM, EEPROM 302, NOVRAM/RTC, DUARTs, LSM, RSM, P555, GM, and clocks. Once board O DUART testing has been performed, board O's microcontroller 300 prints a message to the system console. All microcontrollers except slot l's microcontroller will te~t DUARTs early in their re-source bus test suite. Slot l's microcontroller will test DUARTs late in its test suite. This is to prevent external loopback collisions when slot 0 and slot 1 write to the system console and feedback DUARTs.

_ Any errors or faults o~ the these components are considered ~atal for the motherboard and will result in the deconfiguration of that board. The following des-cribes the deconfiguration process which will occur on the board when the failure is detected.
As soon as an error is detected, a generic error handling ~unction will be executed. An error code is printed to the auxiliary console and an attempt is made to set the resource bus LED in order to indicate a fail-ure.
An attempt is made to write ~ailure in~ormation (error code, test/ sub- test, slot, and FRU/sub-FRU) to the NOVRAM error log. This in~ormation will not be acces-sible by any system JP since the board is held in cold reset and no Scan testing/ initialization has been per-~ormed.
A message is placed in the onboard s~nn~ble mailbox to in~orm the master microcontroller that an error was taken. The content o~ this message will simply state that a ~atal motherboard decon~iguration error occurred.
I~ the attempt to write the mailbox ~ails, this too is valid information as the de~ault master microcontroller, i.e., slot 0 microcontroller, will eventually poll every-one's scanbox. It will ~ind a RESET status in the of-~ending motherboard's mailbox which will be enough o~ anindication that the motherboard ~ailed catastrophically.
I~ the error occurs on the current de~ault master micro-WO 97/07457 PCT~US96/13742 controller, then the same steps are taken. Eventually the secondary master microcontroller will time out, hav-ing not received either a scan message or DUART loopback tra~ic and take mastership over the system.
Finally, the microcontroller will execute a STOP
instruction and the motherboard will be held in cold reset, while in this state. An attempt will be made to keep the microcontroller in an idle loop instead of exe-cuting the STOP instruction. In this m~nner~ the NOVRAM
error log can be potentially read by the diagnostic mas-ter JP later in the powerup process.
Cold reset is deasserted ~or the system and for each slot, and each microcontroller determines in-slot sizing in~ormation as well as executes in-slot scan tests. At this point the ~irst o~ three NOVRAM tables, that table describing each motherboard's physical hardware is built.
In slot sizing will bypass the SB1 and SB2 Scan Bridges ~or the Power Supply and Blowers, as these will be tested by the master microcontroller during off-board scan test.
Should any error be detected while performing in-slot scan testing follow the same steps as outlined above ~or deconfiguration with the ~ollowing exceptions.
The scan tests, unlike the on-board microcontroller diagnostic testing, can fail without the need to deconfi-gure the motherboard. For example, i~ any o~ the daugh-ter board specific tests fail, only the daughter board need be deconfigured. If any of the motherboard specific WO 97/07457 PCTrUS96/13742 tests fall, however, then the motherboard will be deconf-igured. If the motherboard/ daughter board interconnect test fails, then either the daughter board or the entire motherboard will be deconfigured depending upon the de-gree of isolation provided by that test. Once a FRU hasbeen determined bad during scan testing (e.g. JPO daughter board during TAPIT), it will be immediately deconfigured and, therefore, bypassed for the rem~;n~e~ o~ the scan testing. All specific error information, as well as on-board sizing information, will be kept locally in eachmotherboards NOVRAM.
Once the on-board scan testing has been completed or the motherboard has been determined bad at any point during the scan testing, then a message is placed in the sc~nn~hle mailbox to in~orm the master microcontroller as to the outcome of the scan testing. The content of this message will simply state that a fatal motherboard decon-~iguration error occurred or the motherboard passed test-ing. A pass/ fail message is also placed in the micro-controller's mail/ scan box for the slot O motherboard,the location of the default master microcontroller. In the case of a slot 0 motherboard failure, the slot 1 motherboard will eventually timeout, assume mastership, and then gain access to slot O motherboard's scanbox information.
The microcontrollers deassert DIRQ_N to synchronize with the master microcontroller and await further com-mands through their respective sc~nn~hle mailbox. A
master microcontroller is then chosen. The purpose o~
such a master is to allow only one microcontroller to gather system sizing in~ormation and run out-o~-slot scan tests. In addition, microcontrollers will become unsync-hronized due to previous test ~ailures as well as di~er-ences in slot hardware configurations. It is the job o~
the master microcontroller to synchronize all microcont-rollers in the system. If the microcontroller on the slot 1 motherboard is chosen as master microcontroller, then ~rom this point onward there will be no master micr-ocontrollers to de~ault.
The master microcontroller waits for the microcont-rollers on the other motherboards to ~inish in-slot scan tests. This is accomplished by ~irst having all micro-controllers drive DIRQ_N low prior to initiating scan operations. As each microcontroller ~inishes scan test-ing, they individually drive their DIRQ_N high. The master microcontroller, upon finishing its own testing, monitors the DIRQ_N line and once the DIRQ_N ]ine i8 high, the master microcontroller will know that all scan testing has been completed. All microcontrollers should be synchronized at this point.
The master microcontroller then sizes the computer 25 system to determine which boards are present. As part 0~ A
sizing, each microcontroller's sc~nn~hle mailbox is polled by the master microcontroller to determine i~

W O 97/07457 PCT~US96/13742 there has been a resource bus or on-board Scan failure.
The contents of the mailboxes are left intact and the p~n~;ng requests unacknowledged in the event master mic-rocontrollership passes and the steps have to be repeated again. The mailbox information will indicate failed motherboard status for each board, or a pass indication.
If no indication is present in the scanbox (i.e. a reset value) then the assumption is that the motherboard failed. If failures exists, the associated mother-board(s) will be excluded from off- board scan testing.
At this point the SEEPROM system backpanel ID test is performed and using data ~rom the SEEPROM, the master microcontroller constructs the second of the three NOVRAM
tables. This table contains SEEPROM sizing information indicating what the systems hardware was the last time the system was powered up. The Midplane SEEPROM is not updated with the new configuration data until after the "diagnostic master" ~P has been chosen.
The master microcontroller runs the r~;n;ng Off-board scan tests including power supply, blower, and backpanel interconnect testing. Any motherboards that were deconfigured as a result of previous testing will not be included in backpanel interconnect testing. If off-board scan testing is successful, a message to that effect is placed in the slot 1 motherboard's mailbox.
The slot 1 motherboard microcontroller, now in an idle loop, periodically checks for a special non-printable _ character printed to the system console, via the ~eedback DUART, as well as a status message in its sc~nn~hle mail-box. This DUART character will be issued by the slot 0 motherboard at the same time as the scan message is sent, i.e., right a~ter o~-board scan testing has completed.

I~ the master microcontroller is on the slot 1 mo-therboard, then the message bu~er and DUART checking described above will not be per$ormed. I~ o~-board Scan testing ~ails in a m~nne~ which is attributable to the slot 0 motherboard or in a way which is not isolatable and the current master is the slot 0 motherboard, then microcontroller mastership is passed to the slot 1 mothe-rboard via the scAnn~hle mailbox. This is the ~irst of two cases where mastership is passed directly by the slot 0 motherboard, omitting the timeout me~h~n;~m. I~ o~-board Scan testing ~alls in a m~nner which is attribut-able to another motherboard and the current master is board 0, then the o~ending motherboard is decon~igured and mastership is ret~;ne~ by board 0.
I~ off-board Scan testing fails, and the current master is board 1, then the slot 1 motherboard takes the necessary steps to deconfigure all o~fending motherboards indicated by the ~ailures. I~ the error is attributable to the slot 1 motherboard then a fatal system error re-sults. The DUART and message passing protocols provide alternate means o~ detecting scan test ~ailures ~rom the slot 0 motherboard, assuming mastership has not already been passed for any of the reasons already described. If the slot 1 motherboard fails to receive either form of commlln;cation and it is not already master, then after a specified timeout period, it will assume microcontroller mastership and perform the above described actions of the master microcontroller.
If the slot 0 motherboard does experience a self or non- attributable off-board scan test failure, the error is logged in it's NOVRAM, and mastership is passed.
However, the motherboard is not deconfigured because it has up to now proven itself to be a good board with ex-ception of being capable of off board testing. This in and of itself poses no threat to the rem-; n~e~ of the system just 80 long as it is not responsible for perform-ing system scan functions. The slot 1 motherboard, when it takes over must begin off-board scan testing from the very beginning regardless of how far the slot 0 mother-board had already gotten just to prove that it has no problem accessing all off-board resources. If the slot 1 motherboard detects a problem with the slot 0 mother-board, then the slot 0 motherboard will be deconfigured.
If no problem is found, then the slot 0 motherboard's problem is attributable to it's off-board scan hardware.
It still r~m~; n~ a healthy system board unless proven otherwise as testing continues.

WO 97/07457 PCT~US96/13742 To deconfigure, the master microcontroller will issue a command via the sc~nn~hle mailbox. The reception of a deconfiguration message causes the offending mother-board to update it's local copy of current board status kept in a NOVRAM table.
The master microcontroller, via the sc~nn~hle mail-boxes, tells every microcontroller to deassert warm reset for both the system and the slot and begin taking the JPS
out o~ reset. All microcontrollers should be synchro-nized at this point. Each motherboards' microcontrollersees to it that the decon~igured daughter boards and motherboards indicated in the NOVRAM tables, are not included in the JP testing which is about to take place.
As noted above, all of the sc~nn~hle mailboxes still contain the pass/fall indication o~ each motherboard.
This in~ormation is maintained in the event that local NOVRAMs are inaccessible by the "diagnostic master", thus maintaining at least ~ome indication of the failure.
The microcontrollers take each in-slot JP out o~
reset, one at a time. Each JP runs a Suite of "primary tests" which includes basic sanity diagnostics. The microcontrollers monitor the status of these tests by reading on-board SRAM locations updated by the in-slot JPs. In this m~nn~, microcontrollers can recognize test completions, failures, and hangs.
Each Microcontroller chooses an in-slot JP (which successfully completes its "primary tests") to run its "secondary test" suite. This test suite includes the time consuming operations of memory and I/0 initializa-tion/testing. However, allowing such testing to occur in parallel, i.e., across motherboards, will help decrease overall powerup time. As in primary testing, each micro-controller recognizes test completions, failures, and hangs. Secon~ry tests will be run until at least one in-slot JP passes all extensive tests or all in-slot JPs have been exhausted. Each microcontroller will then send "board slave" and "board master" messages to the appro-priate in-slot JPs.
The master microcontroller will await the completion of all of the JP's "secondary tests" via the DIRQ_N mech-anism already explained above. The master microcon-troller will then tell its Board Master JP to run "ter-tiary tests.~' The first part of this testing begins when the master, via the RI/Resource Bus path, reads extensive configuration information from each non-deconfigured motherboard's NOVRAM. This information is required by the master in order to conduct tertiary testing. This test suite includes all off-board JP and memory diagnos-tics.
If the testing succeeds, a "diagnostic master" mes-sage is sent to this Board Master JP. If the slot 0 motherboard fails tertiary testing, the error i8 marked in the NOVRAM and the system attempts powerup again only with the microcontroller of the slot 1 motherboard as the default master. If the slot 1 motherboard also fails tertiary testing then a fatal system error results. If either master detects problems with other motherboards during tertiary testing, then those failing motherboards will be deconfigured. Once the "diagnostic master~
finally chosen, its associated motherboard will be mapped as the global resource board.
During the entire process of selecting a diagnostic master all f~;l;ng tests and resulting motherboard/ daug-hter board deconfigurations will be logged in each boardsNOVRAM error log and the NOVRAM configuration tables.
Just as had been done for the on-board microcontroller diagnostics and scan failures. At the completion of selecting the "diagnostic master~, the diagnostic master JP will poll the various NOVRAMs to determine complete error and deconfiguration information, if any. All cur-rent system configuration information is now be written to the system ID SEEPROM 204.
A machine initiated (MI) callout is now be issued flagging all failed tests and deconfigurations and all scanboxes are cleared. This polling of the NOVRAM will be performed by the diagnostic master via RI ASIC access of the resource bus. Should this RI path access fail for any of the motherboards which have already been deconf-igured, then the scanbox information, which is stillpresent, can be used as a limited source of information.

Alternatively, the DM can request the failing motherboard W O 97/07457 PCT~US96/13742 microcontroller to access the NOVRAM and return the in-~ormation via the falling motherboards sC~nho~. In yet another way, error and deconfiguration in~ormation is copied into the local motherboard SEEPROM just be~ore a board is decon~igured. From there the master microcontr-oller can access the information via o~f-board scan.
Finally, each microcontroller enters its "idle loop". Within the loop, the microcontrollers will period-ically read on-board SRAM in order to process (global or local) requests ~rom system JPs. Additionally, watchdog processing, board sni~ing, and run-LED blinking will be periodically per~ormed in this loop.
By the time which each JP comes out o~ reset, the assumption is that the microcontroller on each board has tested everything in its local resources 260 (e.q. NOVRA-M, EEPROM, SRAM etc.), the blowers, and the power sup-plies. Also the motherboard has been scan tested. I~
there is any ~ailure in the above tests, the microcon-troller has taken the action(s) to ensure that the moth-erboard meets the m;n;mllm requirements to ~unction cor-rectly. Otherwise, none o~ the JP'S ~or that motherboard is taken out o~ reset and the motherboard will be put back into reset. In addition, JP~ will not check ~or any blower and power supply ~ailure during powerup.
The JP PROM powerup sequence is divided into three stages. Upon executing each step, each JP will send a message to the microcontroller on its board, and writes the FRUs or DRUs which is/are going to be tested in the NOVR~M. An error code also i5 written in the NOVRAM so that in case of a failure/hang, the system will know where the problem i5 and have the information to make a MI callout. I~ the test failed, the microcontroller gets a error message from the JP and in the case of a hang, the microcontroller will timeout. There is provided in FIGs. 14-16, a tabulation of the powerup tests performed by the JP and the DM JP.
JPs on each board will be taken out of reset in the order of JP0, JP1, JP2, and JP3. Three actions will be pre~ormed by each JP for each of the following steps: 1) it in~orms the microcontroller which test will be run next; 2) writes the FRU(s) name involved in the test and the error code in NOVRAM; and 3) executes the test. Upon finishing each stage, the JP waits for instruction from the microcontroller what to do next, i.e., continue to next stage or stay idle.
For primary stage testing, the microcontroller brings each JP on its motherboard out o~ reset one after another. The following tests are performed at the prima-ry stage.
- JP runs self-tests to veri~y it is functioning properly.
- JP runs the PROM Access Test to make sure it can access PROM correctly.
- JP tests local SR~M accesses.

- JP tests NOVRAM accesses.
- JP initializes the stacks, the SCM, FE. and the spads in SRAM
- ~p rung tests that will isolate faults to the daughter board (i.e., 88110/88410s C~hes~ CI, TLCC, EDAC te~ts). Some of these tests will be rerun when there are multiple JPs out of reset 80 that inter-actions among the JPs can be tested and still iso-late to the daughter board.
- JP runs RI register access (write/read) tests, including using backdoor method to write/read from itself.
- JP runs PI register access (write/read) tests.
- JP runs GG register access (write/read) tests.
- JP runs MC register access (write/read) tests.
- JP sizes and initializes the MC, PI, EDAC regis-ters and main memory on the motherboard.
- JP runs a quick main memory test, i~ there is any SIMM, on its motherboard. This test will be run on the first SIMM in the first bank o~ memory (i.e. at most 32MB will be tested) only. This test will veri~y that the JP can access memory.

The microcontroller will put the JP which failed any of the above steps back into reset. If anyone of the two JPs on a JP daughter board failed any of the tests, that JP and the daughter board shall be marked dead and decon-~igured. Also a MI call will be made later by the DM.
I~ both JP daughter boards are deconfigured, the rest o~
the associated motherboard will be tested a~ter the DM is determined. I~ all the daughter boards are bad in both slot O and 1, the system will be powered down.
A~ter each JP ~; n; ~hes the primary stage, it waits to be told by the microcontroller whether to go onto se~on~y stage or become a slave JP and go into an idle loop. The ~ollowing tests are per~ormed at the secondary stage.
- JP sizes its own board and writes the in~ormation into NOVRAM.
- JP writes a global test pattern and set a pattern-_valid ~lag in SR~M ~or later testing.
- JP sets all the MADV register(s) on its board.
- JP runs some extensive tests on the MC EDACs, the Directory SIMMs, and the on-board main memory.
- On a SIMM ~ailure, JPPROM will deconfigure the SIMM bank in error. The entire memory subsystem will be decon~igured if any o~ the tests on MC, EDACs, and the Directory ~ailed.
- JP runs the on-board caches tests using its on-board main memory (i~ any exist).
- i~ any o~ these tests failed, the daughter board on which the JP resides will be decon~igured.
- JP sizes all the PCI devices and initializes the GGs on its own board.

- JP runs extensive tests on the integrated SCSI and LAN controllers. These tests are run on each GG
ASIC on a motherboard, one a~ter another. I~ any o~
the test6 ~ailed, then that GG/PCI Bus subsystem will be decon~igured. Failure o~ both GG ASICs will cause the motherboard to be decon~igured.
- Each Board Master collect~ all on-board in~orma-tion (e.g. which ASIC iB good/bad, which JP i5 good/bad, its PROM revision number etc.) and writes it into NOVRAM.
- Each Board Master initializes whatever is le~t on the board e.g. modem.
The ~oregoing tests are run in parallel with each other on the other boards. These tasks are done before the boards are enabled on PI busses. This testing is used to determine whether a particular board can be con-sidered ~or Diagnostic Mastership.
The master microcontroller will tell the ~irst JP on its motherboard that success~ully completed the secondary stage o~ testing to be the acting diagnostic master (ADM) and to go on and per~orm the tertiary stage o~ testing, while the other JPs await instruction. The ~ollowing tests are per~ormed at the tertiary stage.
- ADM runs sel~ PIs loopback test (ThD).
- ADM runs o~-board tests to determine i~ it can access other boards in the system.

- ADM tells JPs in the system to run extensive cache tests.
- ADM collects each ACTIVE board's configuration information using the RI back-door method.
- ADM enables itself on the midplane, i.e. resets the global bits both in RIs and PIs.
- ADM initialize~ and tests other motherboards that don't have any working JPs on them, or runs I/O
tests on boards that don't have any near memory.
The DM uses memory on its board for SCSI scripts locations.
- ADM makes a MI call if the hardware configuration information among Table 1, 2, and 3 in its NOVRAM
are different.
- ADM tells the Master microcontroller to update the SEEPROM on the back- panel according to the in~orma-tion in its NOVRAM - Table 3.
- ADM does the final initialization for the whole system and sets up some control space registers for the operating system, i.e., DG/UX.
- ADM takes all good slave JPs Out of the idle loops and SCM is entered.

There are four possible outcomes from the foregoing:
1) All tests passed. The ADM becomes the DM and finishes the rem~; n; ng steps and halt to SCM.

2) A test ~ailed and the ADM knows that it is the problem on its own motherboard.

3) The ADM hangs when running these tests and its microcontroller timeouts.

4) A test ~ailed and the ADM know~ which board is the problem.
In all cases, except for case 3, the ADM tells the master microcontroller the result o~ the tests. For cases 2 and 3, the master microcontroller will mark it-Rel~ dead in its SEEPROM, and re-powerup the system. For case 4, the master microcontroller tells the ~ailing board's microcontroller to decon~igure itself from the system. A~ter the master microcontroller gets an acknow-ledgment, it will tell the ADM to restart tertiary test-~5 ing with the running o~ the cache tests.A~ter the ~orgoing microcontroller and JP tests are done, the computer system boots ~rom the default boot path. A~ter booting the operating system (DC/UX), all JPs are under control of the operating system and no ~0 JPPROM will be run except when a system call is made.
While the system is running, the microcontroll-er 300 on each motherboard 202 executes sni~fing tasks that veri~y the proper ~unctioning of certain parts o~
the computer system. The resource bus speci~ic tests include: NOVRAM Checksum, DUART Functionality, Power Supply/ Blower status via the PSSS mach, EEPROM Checksum and TBC test. The master microcontroller also is re-WO 97/07457 PCT~US96/13742 sponsible ~or statuting the power supply and blowers via a Scan chain operation. The microcontroller will need to check sem~rhore locks be~ore accessing any o~ these areas in snif~ing tasks so that no data is lost when a JP is trying to update or access an area.
As indicated above only the microcontroller and JP~
on the mothe~hoards in slot 0 and 1 on the backplane are viable candidates respectively ~or the master microcontr-oller and the diagnostic master. In de~ault, the microc-ontroller and one o~ the JP~ on the slot 0 motherboardare so designated. I~ the motherboards in both slots and/or the JPs in both slots are faulted and decon~igura-ble, then the system is shutdown.
Although a pre~erred embodiment of the invention has been described using speci~ic terms, such description is ~or illustrative purposes only, and it is to be under-stood that changes and variations may be made without departing from the spirit or scope o~ the ~ollowing claims.

Claims (33)

What is claimed is:

1. A method for automatically recovering a computer system following discovery of a fault condition, comprising the steps of:
providing a computer system having self-testing and self-diagnosing capability;
automatically testing the computer system;
automatically identifying the presence of one or more faulted components from said step of testing; and de-configuring the computer system to functionally remove the faulted component from the computer system as identified by said step of automatically identifying.

2. The method of claim 1, wherein before said step of testing the method further includes the step of recovering the content of a memory of said computer system.

3. The method of claim 1, further said step of recovering includes step of testing includes performing continuity checks on the computer system components and interconnections between the components and, wherein said identifying includes analyzing the continuity checks to identify faulted components and interconnections.

4. The method of claim 1, wherein the computer system provided includes at least one application specific integrated circuit (ASIC) having a gated balanced clock tree circuit that automatically and selectively distributes clock pulses to logical F/Fs of the ASIC, the clock tree having a plurality of branches therefrom; and wherein the method further includes the steps of:
configuring the gated balance clock tree circuit 80 clock pulses are distributed to at least some of the logical F/Fs when a computer system is in a first operational condition; and re-configuring the gated balance clock tree circuit to block the distribution of the clock pulses to the at least some of the logical F/Fs when the computer system is in a second operational condition.

5. The method of claim 1, wherein the computer system further includes two redundant system clocks, where the clocks of one system clock are provided to the computer system at a time;
wherein the method further includes the steps of monitoring the system clock providing the clocks to the computer system for a failure; and configuring the computer system to functionally remove the failed system clock and to source the clocks thereafter from the redundant system clock, when said step of monitoring identifies a failure of the system clock,

6. A gated balanced clock tree circuit that automatically and selectively supplies clock pulses to the logical flip-flops (F/Fs) of an application specific integrated circuit (ASIC) comprising:
a clock trunk in which flows the clock pulses to the logical F/Fs of the ASIC, the clock tree having a plurality of branches extending therefrom;
at least one AND gate being disposed in at least one of the branches of the clock trunk feeding at least some of the logical F/Fs;
control circuitry that controls each of the at least one AND gate so at least some of the logical F/Fs are feed only certain clock pulses; and wherein said control circuitry is configured so the AND gate automatically allows clock pulses to be feed to the at least some of logical F/Fs during a first operational condition of a computer system using the ASIC and wherein said control circuitry is also configured so clock pulses are automatically blocked by the AND gate when in a second operational condition of the computer system.

7. The gated balanced clock tree circuit of claim 6 further comprising:
a phased locked loop device (PLL) that controls the clock signals being fed into the clock trunk;

a delay cell being electrically interconnected to said PLL and a selected branch of said clock tree, wherein the delay cell delays each clock pulse from the selected branch of the clock tree by a predetermined amount for deskewing the clock pulses to the logical F/Fs; and wherein the clock pulses through the selected branch are not blocked from going to the PLL when in the second operational condition of the computer system.

8. The gated balanced clock tree circuit of claim 6, wherein said control circuitry is configured so the AND gate allows clock pulses to be feed to the logical F/Fs when in a third operational condition.

9. The gated balanced clock tree circuit of claim 8, wherein the first operational condition corresponds to normal operation of the computer system, wherein the second operational condition corresponds to a fatal error condition of the computer system and wherein the third operational condition corresponds to a system testing of the computer system during at least one of powerup and following the fatal error condition.

10. The gated balanced clock tree circuit of claim 6 further comprising a plurality of AND gates to control the clock pulses through all branches but the selected branch of the clock trunk.

11. A method for controlling clock pulses to logical flip-flops F/Fs) of an application specific integrated control circuit (ASIC) comprising the steps of:
providing an ASIC having a gated balanced clock tree circuit that automatically and selectively distributes clock pulses to the logical F/Fs, the clock tree having a plurality of branched therefrom;
configuring the gated balance clock tree circuit so clock pulses are distributed to at least some of the logical F/Fs when a computer system is in a first operational condition; and re-configuring the gated balance clock tree circuit to block the distribution of the clock pulses to the at least some of the logical F/Fs when the computer system is in a second operational condition.

12. The method for controlling clock pulses of claim 11, wherein the gated balance clock tree circuit includes at least one AND gate and wherein said step of configuring includes providing one signal to the AND gate so the clock pulses are distributed and wherein said step of re-configuring includes providing a second signal to the AND gate to block the clock pulses.

13. The method for controlling clock pulses of claim 11, wherein the gated balance clock tree circuit includes a phased locked loop device (PLL) that controls the clock signals being fed into the clock trunk and wherein said method further comprises the steps of:
feeding back clock pulses to the PLL from a selected branch of the clock tree circuit;
delaying the clock pulses being feedback a predetermined amount for deskewing the clock pulses to the logical F/Fs; and wherein the clock pulses being feed back are not blocked from going to the PLL when in the second operational condition of the computer system.

14. The method for controlling clock pulses of claim 11, wherein said step of configuring the gated balance clock tree circuit so clock pulses are distributed also is done when the computer system is in a third operational condition.

15. The method for controlling clock pulses of claim 14, wherein the first operational condition corresponds to normal operation of the computer system, wherein the second operational condition corresponds to a fatal error condition of the computer system and wherein the third operational condition corresponds to a system testing of the computer system during at least one of powerup and following the fatal error condition.

16. The method for controlling clock pulses of claim 11, wherein the gated balance clock tree circuit includes at plurality of AND gate and wherein said step of configuring includes providing one signal to the plurality of AND gates so the clock pulses are distributed and wherein said step of re-configuring includes providing a second signal to the plurality of AND gates to block the clock pulses.

17. The method for controlling clock pulses of claim 16, wherein clock pulses are blocked in all branches but a selected branch of the clock tree circuit by the second signal.

18. A system to supply clock pulses in a computer system comprising:
at least first and second clock generation and distribution devices, where the first clock generation and distribution device is set as the source for clock pulses being supplied by the system;
wherein each of the first and second clock generation and distribution devices includes:
primary oscillator circuitry being electrically configured so as to generate periodic signal pulses, detection circuitry being configured electrically to monitor the periodic pulses generated by the oscillator circuitry to detect a failure of that circuitry, and signal generating circuitry, responsive to the detection circuitry, that is configured to provide an output signal representative of the detection of a failure of the oscillator circuitry; and clock pulse failover circuitry that is configured so the clock supply system automatically redesignates the source of clock pulses being supplied as the second clock generation and distribution device responsive to a failure signal from the signal generating circuitry of the first clock generation and distribution device.

19. The clock pulse supply system of claim 18 wherein the detection circuitry includes a secondary oscillator circuitry for generating a periodic signal at a different frequency than oscillator circuitry and three flip/flops (F/Fs); and wherein the three F/Fs are electrically interconnected to each other and the primary and secondary oscillators circuitry so the F/Fs detect the failure of the primary oscillator circuitry and provide an output representative of the failure.

20. The clock pulse supply system of claim 19 wherein the output representative of the failure is a signal output from two of the three F/Fs.

21. The clock pulse supply system of claim 19 wherein the three F/Fs are respectively first, second and third F/S where the first F/F is electrically interconnected to the second and third F/Fs and the secondary oscillator circuitry and wherein the second and third F/Fs are electrically interconnected to the primary oscillator circuitry.

22. The clock pulse supply system of claim 19 wherein the primary oscillator circuitry includes a 100 MHz crystal and wherein the secondary oscillator circuitry includes a 105 MHz crystal.

23. A method for supplying clock pulses in a computer system comprising:
providing a clock supply system including two redundant signal generating devices that generate periodic signal pulses and circuitry for automatically and selectably controlling the source of the clock pulse being supplied, where one device is initially designated as being the source of the clock pulses being supplied;
monitoring the periodic signals being generated by the signal generating devices sourcing the clock pulses for the system;
automatically failing over to the redundant signal producing device in response to the detection of a failure of the first device to generate periodic signal pulses.

24. The method for supplying clock pulses of claim 23 wherein each signal generating device includes a primary and secondary oscillator circuitry that generate periodic signal pulses at different frequencies and wherein said step of monitoring includes monitoring the output of the primary and secondary oscillators circuitry to detect a failure of the primary oscillator circuitry.

25. A high availability scalable multiprocessor computer system, comprising:

a backplane, including at least one backplane communication bus and a diagnostic bus;
a plurality of motherboards, detachably connected to said backplane; each motherboard interfacing to said at least one backplane communication bus and to said diagnostic bus, each of said plurality of motherboards including:
at least one backplane communication bus interface mechanism interfacing at least one of said plurality of motherboards to said at least one backplane communication bus;
a memory system including main memory distributed among said plurality of motherboards, directory memory for maintaining main memory coherency with caches on other motherboard, and a memory controller module for accessing said main memory and directory memory and interfacing to said motherboard communication bus;
at least one daughterboard, detachably connected to said motherboard and interfacing to said motherboard communication bus, said at least one daughterboard further including:
a motherboard communication bus interface module, for interfacing said at least one daughterboard to said motherboard communication bus and a local bus on said daughterboard; and at least one cache memory system including cache memory and a cache controller module maintaining said cache memory for a processor of said scalable multiprocessor computer system;
a backplane diagnostic bus interface mechanism interfacing each of said plurality of motherboards to said backplane diagnostic bus;
a microcontroller for processing information and providing outputs; and a test bus controller mechanism including registers therein; a scan chain that electrically interconnects functionalities mounted on each motherboard and each of said at least one daughter board to said test bus controller; and an applications program for execution with said microcontroller, said applications program including instructions and criteria to automatically test the functionalities and electrical connections and interconnections, to automatically determine the presence of a faulted component and to automatically functionally remove the faulted component from the computer system.

26. The high availability scalable multiprocessor computer system of claim 25, wherein at least one of said plurality of motherboards and said at least one daughter board further includes at least one application specific integrated circuit (ASIC) having a gated balanced clock tree circuit that automatically and selectively distributes clock pulses to logical F/Fs of the ASIC, wherein said gated balanced clock tree circuit includes:
a clock trunk in which flows the clock pulses to the logical F/Fs of the ASIC, the clock tree having a plurality of branches extending therefrom;
at least one AND gate being disposed in at least one of the branches of the clock trunk feeding at least some of the logical F/Fs;
control circuitry that controls each of the at least one AND gate 80 at least some of the logical F/Fs are feed only certain clock pulses; and wherein said control circuitry is configured so the AND gate automatically allows clock pulses to be feed to the at least some of logical F/Fs during a first operational condition of the computer system and wherein said control circuitry is also configured so clock pulses are automatically blocked by the AND gate when in a second operational condition of the computer system.

27. The high availability scalable multiprocessor computer system of claim 26 wherein said gated balanced clock tree circuit further includes:
a phased locked loop device (PLL) that controls the clock signals being fed into the clock trunk;
a delay cell being electrically interconnected to said PLL and a selected branch of said clock tree, wherein the delay cell delays each clock pulse from the selected branch of the clock tree by a predetermined amount for deskewing the clock pulses to the logical F/Fs; and wherein the clock pulses through the selected branch are not blocked from going to the PLL when in the second operational condition of the computer system.

28. The high availability scalable multiprocessor computer system of claim 26 wherein said control circuitry is configured so the AND gate allows clock pulses to be feed to the logical F/Fs when in a third operational condition.

29. The high availability scalable multiprocessor computer system of claim 28, wherein the first operational condition corresponds to normal operation of the computer system, wherein the second operational condition corresponds to a fatal error condition of the computer system and wherein the third operational condition corresponds to a system testing of the computer system during at least one of powerup and following the fatal error condition.

30. The high availability scalable multiprocessor computer system of claim 26, wherein said gated balanced clock tree circuit further includes a plurality of AND

gates to control the clock pulses through all branches but the selected branch of the clock trunk.

31. The high availability scalable multiprocessor computer system of claim 26, wherein each of said at least one ASIC are interconnected to said at least one scan chain so said microcontroller can test each of said at least one ASIC.

32. The high availability scalable multiprocessor computer system of claim 25, further comprising a system to supply clock pulses in a computer system, the system including:
at least first and second clock generation and distribution devices, where the first clock generation and distribution device is set as the source for clock pulses being supplied by the system;
wherein each of the first and second clock generation and distribution devices includes:
primary oscillator circuitry being electrically configured so as to generate periodic signal pulses, detection circuitry being configured electrically to monitor the periodic pulses generated by the oscillator circuitry to detect a failure of that circuitry, and signal generating circuitry, responsive to the detection circuitry, that is configured to provide an output signal representative of the detection of a failure of the oscillator circuitry; and clock pulse failover circuitry that is configured so the clock supply system automatically redesignates the source of clock pulses being supplied as the second clock generation and distribution device responsive to a failure signal from the signal generating circuitry of the first clock generation and distribution device.

33. The high availability scalable multiprocessor computer system of claim 32, wherein the detection circuitry includes a secondary oscillator circuitry for generating a periodic signal at a different frequency than oscillator circuitry and three flip/flops (F/Fs);
and wherein the three F/Fs are electrically interconnected to each other and the primary and secondary oscillators circuitry so the F/Fs detect the failure of the primary oscillator circuitry and provide an output representative of the failure.