Copyright 2011 by Concurrent Computer Corporation. All rights reserved. This publication or any part thereof is
intended for use with Concurrent Computer Corporation products by Concurrent Computer Corporation personnel,
customers, and end–users. It may not be reproduced in any form without the written permission of the publisher.
The information contained in this document is believed to be correct at the time of publication. It is subject to change
without notice. Concurrent Computer Corporation makes no warranties, expressed or implied, concerning the
information contained in this document.
To report an error or comment on a specific portion of the manual, photocopy the page in question and mark the
correction or comment on the copy. Mail the copy (and any additional comments) to Concurrent Computer Corporation, 2881 Gateway Drive Pompano Beach, FL 3306 9. Mark the envelop e “Attentio n: Real-Time OS PublicationsDepartment.” This publication may not be reproduced for any other reason in any form without written permission
of the publisher.
Concurrent Computer Corporation and its logo are registered trademarks of Concurrent Computer Corporation. All
other Concurrent product names are trademarks of Concurrent while all other product names are trademarks or
registered trademarks of their respective owners. Linux® is used pursuant to a sublicense from the Linux Mark
Institute.
Printed in U. S. A.
Revision History:Level:Effective With:
Original Release -- August 2002000RedHawk Linux Release 1.1
Previous Release -- December 2003210RedHawk Linux Release 2.0
Current Release -- May 2005300RedHawk Linux Release 2.3
Update -- September 2005310RedHawk Linux Release 2.3-4.1
Update -- May 2007320RedHawk Linux Release 4.2
Update -- April 2008330RedHawk Linux Release 5.1
Update -- June 2008400RedHawk Linux Release 5.1
Update -- October 2010500RedHawk Linux Release 5.4
Update -- December 2011600RedHawk Linux Release 6.0
This manual is intended for users responsible for the installation and use of the Real-Time
Clock and Interrupt Module (RCIM) on Concurrent Computer Corporation’s iHawk
systems under the RedHawkTM Linux® operating system.
NOTE
Three RCIM models are described in this guide: RCIM I,
RCIM II and RCIM III. The use of the term “RCIM” refers to
functionality common to all three boards. “RCIM I”, “RCIM II”
and “RCIM III” refer to the specific boards. Refer to the section
“Specifications” on page 1-2 for specifications for each of the
boards.
This manual consists of the following:
TM
• Chapter 1, Introduction, contains a general overview and specifications for
the RCIM boards.
• Chapter 2, Hardware, Installation and Configuration, provides a
description of the RCIM boards and connectors, as well as installation and
configuration instructions.
Syntax Nota tion
• Chapter 3, Functional Description, provides the general operation, user
interfaces and configuration options for the clocks and interrupts available
on the RCIM.
• Appendix A, Registers, describes the RCIM registers.
• Appendix B, Calculating RCIM Cable Propagation Delays, provides a
formula for guarding against propagation delay when chaining RCIMs.
• The Index contains an alphabetical reference to key terms and concepts and
the pages where they occur in the text.
The following notation is used throughout this guide:
italicBooks, reference cards, and items that the user must specify appear in
italic type. Special terms may also appear in italic.
list boldUser input appears in listbold type and must be entered exactly
as shown. Names of directories, files, commands, options and man
page references also appear in list bold type.
This chapter provides an overview and specifications for the Real-Time Clock and
Interrupt Module (RCIM).
NOTE
Three RCIM models are described in this guide: RCIM I,
RCIM II and RCIM III. The use of the term “RCIM” refers to
functionality common to all three boards. “RCIM I”, “RCIM II”
and “RCIM III” refer to the specific boards. The section “Specifications” provides specifications for each of the boards.
Overview1
The Real-Time Clock and Interrupt Module (RCIM) is a PCI-based card that supports
time-critical applications that require rapid response to external events, synchronized
clocks and/or synchronized interrupts.
1
When RCIM boards of various systems are chained together, an interrupt can be
simultaneously distributed to all connected RCIMs, and from the RCIMs to all the
associated host systems.
A synchronized high-resolution clock is provided so that all the RCIMs in an RCIM chain
on multiple systems can share a common time base. It also provides a local POSIX 1003.1
compliant high resolution clock. An optional GPS module allows alignment of the clock
to GPS standard time. A high stability oscillator is standard. Optional oscillators improve
the accuracy of times measured with the RCIM.
In addition to the clocks, this multi-purpose PCI-based card has the following
functionality:
• connection of external device interrupts
• real time clock timers that can interrupt the system
• programmable interrupt generators which allow generation of an interrupt
from an application program
These functions can all generate local interrupts on the system where the RCIM card is
installed. When systems are chained together, multiple input and output interrupts can be
distributed to other RCIM-connected systems. This allows one timer or one external
interrupt or one application program to interrupt multiple RedHawk Linux systems almost
simultaneously to create synchronized actions.
This chapter provides a description of the RCIM PCI-based boards as well as installation
and configuration information.
Board Descriptions1
This section provides illustrations and descriptions of the RCIM III, RCIM II and RCIM I
boards.
RCIM II and RCIM I boards mount in a standard PCI slot and the RCIM III board mounts
into a standard PCI-e slot on a host system. A connector is mounted on each RCIM for
connection to external interrupts, and a synchronization cable is included for daisychaining a master RCIM to one or more slave RCIMs.
Figure 2-2 shows the input/output connectors and LEDs on the RCIM III board. Detailed
information on the LEDs and each of the connectors is provided in the following sections.
Figure 2-2 RCIM III Connectors and LED Locations
LED Functions1
There are two bi-colored status LEDs near the input and output connectors on the
RCIM III board. They will both glow dimly RED when the board is in reset mode
followed by brief intervals of bright RED and GREEN as a test. During normal operation
of the board the LEDs function as follows:
LEDdescriptionFunction
Output
Status
LED
Input
Status
LED
RED solid 10 MHz clock failure
RED 2/sec flashcable option installed but not synchronized or miss-
ing, POSIX clock stopped
GREEN 1/sec flashPOSIX clock running without cable option
GREEN 1/sec blinkPOSIX clock running with cable attached and syn-
chronized
RED/GREEN alternating 2/sec flash
RED 2/sec flashcable option installed but cable unsynchronized or
GREEN solidcable attached and synchronized
POSIX clock running with cable unsychronized or
missing
missing
Input and Output Cables and Connectors1
The RCIM III uses a pair of standard SFP (small form-factor pluggable) connectors
installed in cages to interface to the RCIM III cable. The cable is used to communicate
interrupts, time stamps and a reference clock between RCIM III boards. The output cable
connector is used when the RCIM is either the master or a slave in the middle of an RCIM
chain (see page 2-18 for a description of RCIM modes). The input cable connector is used
when the RCIM is acting in slave mode or in the middle of an RCIM chain. The cable part
number (HS002-3CBL-xx where xx is length in meters) includes an LC fiber optic cable
and two SFPs that are installed in the empty cages on the master and slave RCIMs. Refer
to the section “Daisy Chain Cable” for more information about the cable.
NOTE
The cable SFPs should only be installed and removed with the
system containing the RCIM III powered down. See the Installation section for ESD caution. Care should be taken to insure that
the SFP modules lock into position and that the RCIM III is not
pushed out of its PCIe slot during the installation of the SFPs. The
fiber optic cables themselves can be installed and removed at any
time without damaging the RCIM III.
Oscillators1
The temperature compensated crystal oscillator (TCXO) provided with RCIM III has an
accuracy of +/- 2.5 PPM (parts per million).
Two optional oven controlled crystal oscillators (OCXO) provide a temperature stability
of +/- 210 PPB (parts per billion) or +/- 10 PPB (parts per billion).
The GPS option on the RCIM III includes an active GPS antenna and coaxial cable.
The antenna receives the GPS satellite signals and passes them to the receiver. The GPS
signals are spread spectrum signals in the 1575 MHz range and do not penetrate
conductive or opaque surfaces. Therefore, the antenna must be located outdoors with a
clear view of the sky.
If a different antenna or cable is used, it should match the following specifications:
• 50 Ohm impedence
• 27 dB gain
• 3.3 volt DC power max 30 ma.
External Interrupt I/O Connector1
The external interrupt I/O connector on the RCIM III is a Molex LFH-60 (Low Force
Helix) that provides twelve outputs and twelve inputs.
The external outputs allow equipment to be attached and controlled by the RCIM. The
outputs are driven by a multiplexer which can select any of the programmable interrupt
generators (PIGs), real-time clock timers (RTCs), edge-triggered interrupts (ETIs) or
distributed interrupts (DIs) to drive the output. The selection is controlled by a set of
configuration registers.
See Chapter 3 for information on using external output interrupts and programmable
interrupts.
The pin-outs for the external interrupt I/O connector are shown in Figure 2-3.
Figure 2-3 RCIM III External Interrupt I/O Connector Pin-outs
2-6
The external interrupt input signals are 5 volt ttl levels. The external interrupt outputs
(labeled
EXT_PIG[0-11]) are driven using a 74ABT16240 line driver. The external interrupt
inputs are terminated with 180 ohms to +5 volts, 330 ohms and 0.1 uf to ground. To drive
this input requires a line driver that can sink at least 30 ma. The input termination limits
the speed of the external interrupt signals and helps prevent noise from causing spurious
interrupts. Since most line drivers can sink more current than they can source, the falling
edge of the signal will be faster.
The signals
EXT_CLKIN and EXT_CLKOUT are used for external 10MHz clocks in or out. An
external clock driving the RCIM III should be capable of driving a 5V ttl signal into a 50
ohm load. The RCIM III will automatically switch to using the external clock if one is
present. The external clock output from the RCIM III is driven using a
The RCIM III uses a fiber optic serial synchronization cable with SFP (small form-factor
pluggable) connectors (part no. HS002-3CBL-xx) to connect RCIM IIIs in an RCIM
chain. The serial data on the cable includes parity and framing information which allow
cable problems to be detected. Polling is done continuously and messages that report the
status of the RCIM III daisy chain cables are output when an error condition is detected.
Messages indicating problems will appear on the systems directly connected by a failing
link.
The serial cables are point to point connections. The “input” cable refers to the cable going
upstream towards the master RCIM. The “output” cable is the downstream connection
away from the master.
RCIM: Cable error on input cable.
RCIM: Cable error on output cable.
The “disconnected” and “connected” messages will only occur based on whether an SFP
is installed in the appropriate cage of the RCIM III.
They will not occur when the optical cable is inserted or removed. They should not occur
during normal operation unless the SFP is not installed correctly or it is malfunctioning.
The “not synchronized” and “unsynchronized” messages indicate that the cable is not
answering attempts to communicate. These messages will occur when the optical cable is
installed or removed. They will also occur when a connected system is powered off.
The last two messages indicate transient errors such as cable parity errors or temporary
loss of cable synchronization. If a transient error occurs, it may require a link in the cable
to resynchronize. If a distributed interrupt is being broadcast on the cable, it may be lost.
Transient errors also affect the synchronization of the tick timers since the cable clock will
not reach all of the systems. Refer to Chapter 3 for instructions for synchronizing clocks.
RCIM II1
Board Illustration1
Figure 2-4 shows the RCIM II board with optional high stability OCXO (Oven Controlled
Crystal Oscillator) and GPS modules installed.
Figure 2-4 RCIM II Board
Connectors and LEDs1
Figure 2-5 shows the input/output connectors and LEDs on the RCIM II board.
The output cable connector is used when the RCIM II is either the master, or a slave in the
middle of an RCIM chain (see page 2-18 for a description of RCIM modes). The input
cable connector is used when the RCIM is acting in slave mode.
The cable attached to the output cable connector is an RJ45 serial synchronization cable
(part no. HS002-CBL-10). Refer to the section “Daisy Chain Cable” for more information
about the cable.
Note that although RJ45 cables are used with Gigabit Ethernet, the RCIM II cables are not
Ethernet compatible.
Oscillators1
The standard crystal oscillator provided with RCIM II has an accuracy of +/- 20 PPM
(parts per million).
Two optional oven controlled crystal oscillators (OCXO) provide a temperature stability
of +/- 210 PPB (parts per billion) or +/- 10 PPB (parts per billion).
GPS Antenna1
The GPS option on the RCIM II includes an active GPS antenna and coaxial cable.
The antenna receives the GPS satellite signals and passes them to the receiver. The GPS
signals are spread spectrum signals in the 1575 MHz range and do not penetrate
conductive or opaque surfaces. Therefore, the antenna must be located outdoors with a
clear view of the sky.
If a different antenna or cable is used, it should match the following specifications:
• 50 Ohm impedence
• 27 dB gain
• 3.3 volt DC power max 30 ma.
External Interrupt I/O Connector1
The external interrupt I/O connector on the RCIM II is a Molex LFH-60 (Low Force
Helix) that provides twelve outputs and twelve inputs.
The external outputs allow equipment to be attached and controlled by the RCIM. The
outputs are driven by a multiplexer which can select any of the programmable interrupt
generators (PIGs), real-time clock timers (RTCs), edge-triggered interrupts (ETIs) or
distributed interrupts (DIs) to drive the output. The selection is controlled by a set of
configuration registers.
2-10
See Chapter 3 for information on using external output interrupts and programmable
interrupts.
The pin-outs for the external interrupt I/O connector are shown in Figure 2-6.
Figure 2-6 RCIM II External Interrupt I/O Connector Pin-outs
The external interrupt input signals are 5 volt ttl levels. The external interrupt outputs
(labeled
EXT_PIG[0-11]) are driven using a 74ABT16240 line driver. The external interrupt
inputs are terminated with 180 ohms to +5 volts, 330 ohms and 0.1 uf to ground. To drive
this input requires a line driver that can sink at least 30 ma. The input termination limits
the speed of the external interrupt signals and helps prevent noise from causing spurious
interrupts. Since most line drivers can sink more current than they can source, the falling
edge of the signal will be faster.
The signals
EXT_CLKIN and EXT_CLKOUT are used for external 10MHz clocks in or out. An
external clock driving the RCIM II should be capable of driving a 5V ttl signal into a 50
ohm load. The RCIM II will automatically switch to using the external clock if one is
present. The external clock output from the RCIM II is driven using a
EXT_RXD1, EXT_TXD1, EXT_RXD2, and EXT_TXD2 are RS-232 level signals.
They are currently used for debug purposes.
System Identification1
The following output to lspci(8) shows the PCI class, vendor and device IDs for the
RCIM II (0d:06.0 (bus:slot.function) will differ on your system):
# lspci -v | grep -i rcim
0d:06.0 System peripheral: Concurrent Computer Corp RCIM II
Realtime Clock and Interrupts Module (rev 01)
# lspci -ns 0d:06.0
0d:06.0 Class 0880: 1542:9260 (rev 01)
Daisy Chain Cable1
The RCIM II uses an RJ45 serial synchronization cable (part no. HS002-CBL-10) to
connect RCIM IIs in an RCIM chain. The serial data on the cable includes parity and
framing information which allow cable problems to be detected. Polling is done
continuously and messages that report the status of the RCIM II daisy chain cables are
output when an error condition is detected. Messages indicating problems will appear on
the systems directly connected by a failing link.
The serial cables are point to point connections. The “input” cable refers to the cable going
upstream towards the master RCIM. The “output” cable is the downstream connection
away from the master.
RCIM: Cable error on input cable.
RCIM: Cable error on output cable.
The “not synchronized” and “unsynchronized” messages indicate that the cable is
connected but not answering attempts to communicate. This would be the case if the
connected system was powered off.
The last two messages indicate transient errors such as cable parity errors or temporary
loss of cable synchronization. If a transient error occurs, it may require a link in the cable
to resynchronize. If a distributed interrupt is being broadcast on the cable, it may be lost.
Transient errors also affect the synchronization of the tick timers since the cable clock will
not reach all of the systems. Refer to Chapter 3 for instructions for synchronizing clocks.
Figure 2-8 shows the input/output connectors and LEDs on the RCIM I board.
Detailed information on the LEDs and each of the connectors is provided in the following
sections.
Figure 2-8 RCIM I Connectors and LED Locations
LED Functions1
The four LEDs are on the circuit side of the RCIM I board between the output cable
connector (P2) and the external interrupts connector (P4). They function as follows:
LED NumberFunction
Red DS1If on after reset, indicates RCIM module failed
power-up reset.
Yellow DS2If on, indicates cable clock not connected.
The output cable connector is used when the RCIM I is either the master, or a slave in the
middle of an RCIM chain (see page 2-18 for a description of RCIM modes). The cable
attached to the output cable connector is called a synchronization cable (part no. 6010178-
109). The pin-outs for the output cable connector are shown in Figure 2-9.
Figure 2-9 RCIM I Output Cable Connector (P2) Pin-outs
The input cable connector is used when the RCIM I is acting in slave mode (see page 2-18
for a description of RCIM modes). The cable attached to the input cable connector is
called a synchronization cable (part no. 6010178-109). The pin-outs for the input cable
connector are shown in Figure 2-10.
Figure 2-10 RCIM I Input Cable Connector (P3) Pin-outs
External Interrupts Connector (P4)1
The external interrupts connector on the RCIM I provides four outputs and four inputs.
The external outputs allow equipment to be attached and controlled by the RCIM. The
outputs are driven by a multiplexer which can select any of the programmable interrupt
generators (PIGs), real-time clock timers (RTCs), edge-triggered interrupts (ETIs) or
distributed interrupts (DIs) to drive the output. The selection is controlled by a set of
configuration registers.
See Chapter 3 for information on using external output interrupts and programmable
interrupts.
Pin-outs for the external interrupts connector are shown in Figure 2-11.
Figure 2-11 RCIM I External Interrupts Connector (P4) Pin-outs
Debug Visibility Connector (P5)1
The debug visibility connector is intended for use on the Concurrent Computer
Corporation manufacturing floor and should not be used outside of that environment.
In-System Programming Interface Connector (P6)1
The in-system programming interface connector is intended for use on the Concurrent
Computer Corporation manufacturing floor and should not be used outside of that
environment.
System Identification1
The following output to lspci(8) shows the PCI class, vendor and device IDs for the
RCIM I (0d:06.0 (bus:slot.function) will differ on your system):
# lspci -v | grep -i rcim
0d:06.0 System peripheral: PLX Technology, Inc. RCIM Realtime
Clock and Interrupts Module old ID (rev 01)
# lspci -ns 0d:06.0
0d:06.0 Class 0880: 10b5:8845 (rev 01)
When RCIM boards of various systems are chained together, an interrupt can be
simultaneously distributed to all connected RCIMs, and from the RCIMs to all the
associated host systems.
NOTE
All RCIMs in a chain must be the same model; for example, all
RCIM IIs.
If your system will be part of an RCIM chain, it is best to determine the desired connection
mode before installing an RCIM; it is easier to connect the cable to the cable connectors
before the RCIM is installed.
Note that to reconfigure an RCIM chain, the systems must be powered off and rebooted
after moving the cables. The driver determines if a system is the master RCIM at boot time
and configures the master system to control the cable clock and its enable. Swapping the
cables without rebooting the systems will result in problems with the cable clock.
An RCIM can be connected in one of the following modes:
Isolated modeThere are no connections to any other RCIM.
Master modeThe RCIM is at the head of a chain of RCIMs. There is no cable
connection going into this RCIM, only a cable connection going out.
The RCIM master is unique in that it controls the clocks (see
Chapter 3 for a discussion).
Pass-throughThe RCIM is connected to two other RCIMs. There is an input
Slave modecable connection coming from the previous RCIM in the chain, and
an output cable connection going to the next RCIM in the chain.
Final Slave modeThe RCIM is connected to one other RCIM. There is an input cable
connection going into a final slave RCIM but no output cable
connection coming out of it.
Unpacking the RCIM1
When unpacking the equipment from the shipping container, refer to the packing list and
verify that all items are present. Save the packing material for storing and reshipping the
equipment.
2-18
NOTE
If the shipping container is damaged upon receipt, request that the
carrier’s agent be present during unpacking and inspection of the
equipment.
Normally, installation and configuration of the card is done by Concurrent Computer
Corporation. This information is provided for those cases where an RCIM is added to a
system in a post-manufacturing environment.
In order to successfully install the RCIM, you must know if you will be using the RCIM to
accept or deliver external interrupts and the mode in which the RCIM will run (isolated,
master, pass-through slave or final slave). Refer to the section “Connection Modes” above
for details.
CAUTION
Avoid touching areas of integrated circuitry as static discharge can
damage circuits.
Concurrent Computer Corporation strongly recommends that you
use an antistatic wrist strap and a conductive foam pad when
installing or upgrading a system. Electronic components such as
disk drives, computer boards, and memory modules can be
extremely sensitive to Electrostatic Discharge (ESD). After
removing the board from the system or its protective wrapper,
place it flat on a grounded, static-free surface, component side up.
Do not slide the board over any surface.
If an ESD station is not available, you can avoid damage resulting
from ESD by wearing an antistatic strap (available at electronic
stores) that is attached to an unpainted metal part of the system
chassis.
Use the following procedure to install an RCIM in your system:
1. Ensure that your system is powered down.
2. Remove the power cable from the system.
3. Open the case of your system and identify the PCIe slot (RCIM III) or PCI
slot (RCIM II or RCIM I) where you want the RCIM to reside. In general,
it is best for the RCIM to be configured in a slot where minimal or no
contention with other devices occurs and at the highest IRQ priority
possible. Refer to the iHawk Optimization Guide, publication number
0898011, for slot configuration guidelines.
4. Install the RCIM into the desired slot, securing the card in the slot using the
mechanism provided by the case.
5. If this is to be part of an RCIM chain, attach the cable as required. See the
section “Connection Modes” to determine how to connect the cable based
on the connection mode for this system.
6. If you are installing an RCIM board equipped with the optional GPS module, attach the GPS antenna lead and mount the antenna. The antenna
should be mounted on the rooftop or in an open area. Refer to the sections
“Connectors and LEDs” and “GPS Antenna” more information about the
antenna.
The following RedHawk Linux kernel parameters are associated with the RCIM. All are
accessible through the Character Devices selection of the Kernel Configuration GUI
and are enabled by default in all the pre-built RedHawk Linux kernels.
RCIMThis parameter configures the RCIM driver in the kernel. It can be
configured as a module, if desired.
MULTI_RCIMSelects whether the multi-board or the single-board RCIM driver is
to be compiled into the kernel. The multi-board driver is used by
default.
RCIM_MASTERCLOCK This parameter enables the RCIM to be used as a master clock thta
monitors and adjusts system time, resulting in more precise system
timekeeping.
RCIM_PPSThis parameter disciplines the RCIM tick and POSIX time-of-day
registers to the pulse-per-second support of the optional GPS
receiver. This provides for time more closely aligned to the official
atomic time as defined by the GPS system. This option has no
effect if your RCIM does not have GPS capability.
RCIM_IRQ_EXTENSIONS This parameter allows other drivers to attach their own interrupt
routines to the RCIM driver. The Frequency-based Scheduler
(FBS) requires this support.
For complete information about modifying kernel tunables and building a kernel, refer to
the RedHawk Linux User’s Guide, publication number 0898004.
Driver Configuration1
This section describes two methods for configuring an RCIM board: dynamic, via the
writing of a string to a /proc file, and static, via boot time (or module load time) options.
The single-board RCIM driver supports static and dynamic, however the multi-board
driver supports only the dynamic method.
Static Configuration1
When the single-board RCIM driver initializes on a system, it looks at the value of a single
tunable (rcim=RCIMoptions) for its configuration options. For a statically linked RCIM
driver, this tunable can be specified on the GRUB boot loader command line. For an
RCIM in module form, this tunable can be specified on the insmod(8) command line,
or be placed in modprobe.conf(5) where the modprobe(8) invocation in the
startup script /etc/init.d/rcim will find it.
2-20
The following functionality is defined with the rcim tunable:
In this example for an RCIM slave system, the RCIM master system name is
server1.ccur.com, edge-triggered interrupt (eti) #1 is configured to trigger on the rising
edge, distributed interrupt (di) #3 triggers on a high value, and distributed interrupt line
(di) #6 is to be driven by real-time clock timer (rtc) #3.
The sync/nosync options affect clock synchronization across an RCIM chain. “sync”
specifies that the RCIM is to use the cable clock driven by the master RCIM’s local clock.
This is the default.“nosync” specifies that the RCIM uses its local clock.
The clock/noclock options define whether the RCIM is registered as a clocksource for the
system. The default is “clock.” Note that once the RCIM is registered as a clocksource, it
cannot be “unregistered.” Also if the RCIM is configured as a module and registered as a
clocksource, the module will be locked in (‘rmmod rcim” will fail).
On RCIM III and RCIM II systems, the timing sources for RTCs, tick and POSIX clocks
can be configured individually, if desired. For example,
rcim=nosync/r,sync/tp
selects the local clock for the RTCs (r) and the cable clock for the tick (t) and POSIX (p)
clocks. If specifiers are not provided for sync and nosync, all three clocks are set
according to the option in effect.
Dynamic Configuration1
Configurations can be modified dynamically using echo(1) to write a configuration
string, in the format used by the rcim tunable, to /proc/driver/rcimN/config
(where N is the RCIM card number starting from zero). For example:
echo eti1/f > /proc/driver/rcim0/config
changes eti #1 to trigger on a falling edge. Quotes must be used to surround a
configuration request that contains a vertical bar; for example:
echo “rtc0|di1” > /proc/driver/rcim2/config
Configuration modifications made in this way are not retained when the system is
rebooted. Making these changes requires write permission to the file and should only be
made when the RCIM is not in use.
Refer to Chapter 3 or the rcim(4) man page for complete information about
configuration defaults and selections for each interrupt type.
When configuring the RCIM systems, keep the following in mind:
• For a chain of RCIMs, the tick clock and POSIX clock in all slave RCIMs
will be synchronized with the master because the clock signal incrementing
time in the master is broadcast to all slaves. Once the clocks on all RCIMs
are initially syncronized they will remain synchronized.
To synchronize tick clocks, a working TCP/IP connection between all
systems is needed. In addition, each slave RCIM hostname configuration
must be set to the master RCIM, and each slave must be configured to run
the rcim_clocksync init script once on boot. This is only required if
your application is using the tick clock for synchronization.
ntp can be used to synchronize the POSIX clocks, however for the RCIMIII there is a better mechanism: rcimdate(8). The RCIM-III master
broadcasts its POSIX time down the RCIM cabling once per second;
rcimdate uses this to make the slave POSIX clocks exactly match the
master. This has several advantages over ntp: no TCP/IP connections
between systems are required, synchronization is faster, and
synchronization is extremely accurate.
• Interrupts, whether operating locally or distributed across an RCIM chain,
will be processed according to the values configured on each system. If you
wish them to function in a manner different from established defaults, the
desired configuration options must be specified.
• When distributing interrupts across the systems in an RCIM chain, all
systems must have a compatible configuration for the distributed interrupt
lines.
MSI Interrupt Configuration1
The latest version of RCIM III (revision 9 and later) supports MSI (message signaled
interrupts). By default, the RCIM kernel driver will initialize the hardware to use MSI
interrupts instead of PCI INTA interrupts whenever possible. By using MSI interrupts, the
RCIM III is guaranteed of having its own non-shared interrupt, thus providing more
reliable interrupt response times.
The RCIM driver has an rcim.nomsi=1 option which is independent of the rcim=
configuration option described earlier. All versions of the driver have this option. If
specified the MSI capability is disabled on all RCIM boards that support it. There is no
mechanism to pick which boards are to be MSI-disabled and which ones are not. When
this option is specified, the RCIM driver will fallback to using the PCI INTA interrupt
method. For perfomance reasons, this option should only be used if a problem with MSI
interrupts is encountered.
2-22
For a statically linked RCIM driver, this tunable can be specified on the GRUB boot
loader command line (rcim.nomsi=1). For an RCIM in module form, this tunable can
be placed in /etc/modprobe.conf as “options rcim.nomsi=1”.
This option has no effect on RCIM II or RCIM I systems.
If your system contains the optional GPS module, ntp must be installed and configured to
use the GPS receiver to synchronize the RCIM’s POSIX clock to GPS time. Follow these
steps:
1. Verify that the latest ntpd rpm is installed on the system:
# rpm -ql ccur-ntp-4.2.2p1-7*
*
includes possible updates (.1, .2, etc.) and the architecture type, i386 or
x86_64.
If it is not installed, refer to the RedHawk Linux Release Notes, publication number
0898003, for instructions to install this rpm from the RedHawk installation media.
2. The file /etc/ntp.conf supplied with the rpm contains the following
lines that are required to use the GPS.
server 127.127.8.0 mode 138 prefer #PARSE TSIP (10)+ PPS(128)
fudge 127.127.8.0 flag3 1#enable PPS signal
The following three lines define a pool of world-wide servers that are randomly
selected at poweron for time synchronization. This feature acts as the default NTP
configuration and serves as backup to GPS. You may wish to include your local
country code before “pool” in these entries for best results; e.g.,
0.us.pool.ntp.org. See www.pool.ntp.org for more information.
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
A block of commented out entries beginning with “logfile” is used to configure files
used for logging statistics. Uncomment the entries if you wish to enable them.
In addition to the log files, ntpq(1) and ntpdc(1) are used for NTP monitoring.
For more information about NTP, refer to the ntpd(1) man page and
www.ntp.org.
At system poweron, once all data is received from the GPS satellites, accurate
timekeeping is available.
Verifying ntp/GPS Operation1
To determine when the GPS is producing accurate time, use the peer listing of ntpq(1)
as shown below. This example utilizes command line options, however ntpq can also be
run interactively. Refer to the man page for complete information.
-216.56.81.86 193.131.101.50 3 u 33 64 377 49.388 -0.579 2.710
+new.localdomain .GPS. 1 u 42 64 377 0.182 0.010 0.020
*GENERIC(0) .GPS. 0 l 46 64 377 0.000 0.000 0.001
The output shows how the system time compares to other time sources. This includes the
GPS receiver and other time servers.
The column labeled remote is the hostname of the timeserver. The system
new.localdomain is a local network time server; GENERIC(0) is the GPS attached to
the RCIM. The other lines are time servers assigned by pool.ntp.org. The first column
indicates which servers are being selected for synchronization. The '*' in front of
GENERIC(0) indicates that the RCIM GPS receiver is being used as the system peer.
The columns delay, offset, and jitter are all times in milliseconds. The offset is
the difference between the local system time and the time source. In this case we are
synchronized to the GPS receiver to microsecond accuracy.
The delay field is the measured network delay to exchange the time with the remote
server.
The jitter measures the difference between offset values from the same source.
The refid indicates where the remote system gets its time.
The st column is the stratum number. A stratum zero system should have a direct
connection to an authoritative source.
The poll column shows how frequently this server is being polled. The when column is
the time in seconds since the last poll.
The reach column is a bitmap in octal which shows if recent polls have been successful.
The value 377 would indicate that the last 8 polls suceeded.
Another indicator is through the ntpq clockvar command:
The refclock_states message in this example indicates that 98.04% of the time the
GPS is receiving useful data. A higher nominal number indicates the most accuracy.
Additional information is provided by many other messages in this output. Refer to
ntpq(1) for complete details
This chapter describes the clocks and interrupt capabilities provided by the RCIM and the
user interfaces for each.
Overview1
The Real-Time Clock and Interrupt Module (RCIM) provides two non-interrupting clocks.
One of these clocks can be synchronized with all the RCIMs in an RCIM chain to provide
a common time stamp across systems. The other clock is POSIX 1003.1 compliant and,
although not synchronized across the RCIM chain, it increments in unison with the other
clock on the RCIM board and can be set to a specific time.
In addition to the clocks, the following methods for handling signal processing (interrupts)
are available:
• Edge-Triggered Interrupts (ETIs)
• Real-Time Clocks (RTCs)
• External Output Interrupts
• Programmable Interrupt Generators (PIGs)
• Distributed Interrupts (DIs)
3
These interrupts operate locally on an RCIM system or can be distributed across all RCIM
systems in an RCIM chain. The open(2), close(2) and ioctl(2) system calls are
used to manipulate the interrupts. Separate device files are associated with each interrupt.
The clocks and interrupts are described in this chapter.
Clocks1
The RCIM provides two non-interrupting clocks, which are fully explained in the sections
that follow.
tick a 64-bit non-interrupting clock that increments by one on each
tick of the common 400ns clock signal. This clock can be reset to
zero and synchronized across the RCIM chain, providing a
common time stamp.
POSIXa 64-bit non-interrupting clock encoded in POSIX 1003.1 format.
The upper 32 bits contain seconds and the lower 32 bits contain
nanoseconds. This clock is incremented on each tick of the
common clock signal. It is primarily used as a high-resolution
local clock. It can be configured to synchronize system time with
GPS standard time on boards equipped with GPS.
All clocks on all RCIMs in the chain are incremented in unison, as they are all driven by a
common clock signal emanating from the master RCIM.
The tick clock is a 64-bit non-interrupting counter that increments by one on each tick of
the common clock signal. Although it cannot be set to a specific time, it can be
incremented or set to zero. Hence the tick clock cannot be adjusted on the fly to
approximate the current time of day as would be required of a true time-of-day clock. On
RCIM I systems, the system clock is frequency-synchronized with the tick clock.
When an RCIM board is part of an RCIM chain, the tick clocks on all slave RCIMs are
incremented and cleared in synchronization with whatever incrementing and clearing is
done to the tick clock located on the master RCIM.
The tick clock can be read on any system, master or slave, using direct reads when the
device file /dev/rcimN/sclk (where N is the RCIM card number starting from zero)
is mapped into the address space of a program. See the section “Direct Access to the
Clocks” below for details.
By default, tick clock initialization (zeroing) and synchronization with other tick clocks on
the RCIM chain occur automatically whenever the RCIM master boots. Initializing and
synchronizing tick clocks is accomplished using the rcim_clocksync(1) command.
See the sections “Synchronizing the Tick Clock” and “The rcim_clocksync Utility” below
for details.
The POSIX Clock1
The POSIX clock is a 64-bit non-interrupting counter encoded in POSIX 1003.1 format.
The upper 32 bits contain seconds and the lower 32 bits contain nanoseconds. This clock
is incremented on each tick of the common clock signal. On RCIM III and RCIM II
systems, the system clock is synchronized to the POSIX clock.
For RCIM II and III, setting the system clock (for example, with clock_settime(2))
will set both the system clock and the RCIM POSIX clock to the new time. In addition, the
POSIX clock can be mapped with mmap and then read like any other RCIM register by
applications. However, modifying the RCIM POSIX clock via the mmap method is not
recommended while the system is sychronizing with the RCIM POSIX clock.
The POSIX clock can be loaded with any desired time; however, the value loaded is not
synchronized with other clocks in an RCIM chain. Only the POSIX clock of the RCIM
attached to the host is updated. See the section “Direct Access to the Clocks” below for
details about accessing the POSIX clock.
Although the RCIM I and II have no hardware support for setting the POSIX clocks of all
boards to a consistent value, it is possible to pause the operation of the RCIM board chain,
synchronize the POSIX clock values during the pause using the rcim_clocksync(1)
command on each of the systems, then restart the RCIM board chain. See the section
“Synchronizing the POSIX Clock” later in this chapter for instructions. For the RCIM III,
rcimdate can be run on each slave to synchronize the POSIX clock to that of the master.
No TCP/IP connection or other software is needed. rcimdate utilizes the POSIX
timestamp sent down the RCIM cabling once per second by the RCIM III maser.
3-2
On an RCIM system with GPS module and NTP running, the GPS receiver is used to
synchronize the POSIX clock on the RCIM on which it is attached to GPS time. See the
section “Using GPS for System Timekeeping” for details. Generally, only the master
RCIM needs a GPS; slaves use one of the software methods available for syncronizing
their POSIX clock to the master.
Direct Access to the Clocks1
The device file /dev/rcimN/sclk (where N is the RCIM card number starting from
zero) can be used to access the RCIM clocks directly using mmap(2). From the address
returned by mmap, the following offsets are used to access the clock fields:
0x0upper 32-bits of tick clock
0x8lower 32-bits of tick clock
0x10status and control (cannot be modified)
0x100POSIX clock seconds
0x108POSIX clock nanoseconds
0x110status and control (cannot be modified)
These offsets are defined in the header file /usr/include/linux.h (with names
beginning with
To set the value of the POSIX clock, the rcim_clocksync(1) utility can be used in
interactive mode using the “update” command.
RCIM_SYNCCLOCK_).
Synchronizing the Clocks1
The sections below describe the techniques and tools used to synchronize the clocks on
the RCIM.
The rcim_clocksync Utility1
The rcim_clocksync(1) utility can be used to reset the tick clocks on all connected
RCIMs to zero to provide a common time stamp across the system. This synchronization
operation occurs automatically when the RCIM master system boots. As slave systems
become available, it will be necessary to reissue this command to synchronize the tick
clocks across the RCIM chain, however, this can be automated (see the section
“Automatic Synchronization” below). rcim_clocksync can also be used to
synchronize the POSIX clocks on all connected RCIMs. This procedure is described in the
section “Synchronizing the POSIX Clock.”
Note that the system clock is synchronized with the RCIM and when rcim_clocksync
is run other than at system boot, there are consequences and should be used with caution.
On RCIM III and RCIM II, master and slave system times are synchronized with the
POSIX clock; on RCIM I, the times are synchronized with the tick clock. When system
time stops advancing, time-based functions using those clocks will stop.
Synchronization always succeeds on the RCIM master or isolated system; an error is
returned if synchronization is attempted on an RCIM slave system.
Specifying rcim_clocksync with no options on the RCIM master system synchronizes
all tick clocks in the RCIM chain.
operations are:
s - synchronize clocks
0[tp] - stop clock ([t]ick/[p]osix)
1[tp] - start clock ([t]ick/[p]osix)
w[tp] - update clock value ([t]ick/[p]osix)
i[tp] - isolate clock ([t]ick/[p]osix)
c[tp] - connect clock ([t]ick/[p]osix)
d - disable cable clock signal
e - enable cable clock signal
q - quit
enter operation:
rcim_clocksync takes the following options:
-iinteractive mode (see below)
-mprints the configured hostname where the RCIM master is located (see “Con-
figuration” in Chapter 2).
-sprints the RCIM connection state
devname The device name of the desired RCIM board. Defaults to
/dev/rcim0/rcim which is the first RCIM board that was found on boot.
/dev/rcim1/rcim, /dev/rcim2/rcim, etc. should be used to point this
tool to the second, third, etc. RCIM board found on boot.
When interactive mode is invoked, a display similar to the following example provides
configuration and status information updated every two seconds, as well as command
usage. These items are explained below.
RCIM isindicates the RCIM mode for this system: master, pass-through slave, final slave or
isolated
RCIM versionis the RCIM version number
Configured RCIM master hostname is
indicates the hostname of the RCIM master. This must be configured using the host configuration option
(see “Configuration” in Chapter 2)
cable signalis one of the following:
3-4
ENABLED/DISABLED – For the RCIM master, indicates if the cable clock signal is being propagated to
slaves. For RCIM slaves, indicates if clocks are being driven by the RCIM master or if ticking locally
(without synchronization).
CLOCK_MISSING – Error condition indicating cable clock signal is not being properly propagated to
slaves.
CLOCK_STOPPED – Error condition indicating cable clock signal has been stopped.
CABLE_SYNC – indicates that the RCIM slave clock is being driven by RCIM master cable clock signal, if
available
CABLE_ENABLE – indicates that the RCIM clock resets when RCIM master does a synchronization
LOCAL_ENABLE – indicates that the clock is enabled
NO_RESET_WHEN_DISABLED – indicates that the clock is not reset when disabled
current clock values for each clock
This section is usage information for the interactive mode. At the enter operation: prompt, supply
one of the operation codes to accomplish the task described for that operation. As noted, some operations
require a designation for the clock to be operated on: t for the tick clock; p for the POSIX clock.
Synchronizing the Tick Clock1
When the RCIM master system boots, rcim_clocksync is executed, which resets all
tick clocks in the RCIM chain to zero.
When an RCIM slave system boots after the master has booted, the tick clocks on the
systems in the RCIM chain will be out of sync, unless automatic clock synchronization is
configured (see the section “Automatic Synchronization” below). Invoking
rcim_clocksync without options on the master RCIM system synchronizes all tick
clocks in the RCIM chain. See the section “The rcim_clocksync Utility” and the
rcim_clocksync(1) man page for more information about this utility.
RCIM Masterclock Considerations1
For RCIM II and III, the RCIM POSIX clock is the system masterclock. This means that
system time continually looks at the RCIM's POSIX clock and adjusts itself to match.
In previous releases of RedHawk, a broken or stopped RCIM would confuse and often
lock up the system, however this is no longer the case. The RedHawk masterclock code
now continually tests the RCIM POSIX clock for validity and pauses clock
synchronization if problems are detected. This means that the two clocks, the RCIM
POSIX clock and the system clock, become free-running. Once the RCIM POSIX clock
again becomes valid clock synchronization is resumed.
In order for synchronization to resume, RedHawk must detect that both the system clock
and the RCIM POSIX clock are incrementing, that each clock is incrementing at the
correct rate, and that each clock has a value within approximately two seconds of the
other. This latter condition is most easily achieved by executing a clock_settime(3),
but other methods, as documented in masterclock(5), are available that may be more
suitable for specific situations.
Synchronizing the POSIX Clock1
The POSIX clocks tick synchronously but generally do not have the same clock values
and should not be used as a common time stamp. They can be synchronized with the other
POSIX clocks in the RCIM chain so that they are consistent, if desired. For the RCIM III,
run rcimdate on each slave; on return the slave’s RCIM POSIX clocks will be
synchronized with the master. For RCIM I and II use the following rcim_clocksync
interactive mode procedure defined below.
Note that this operation also synchronizes the RCIM tick clocks. Be aware that while
performing this procedure, the system time stops and time-base functions will be affected.
1. Ensure that all POSIX clocks on all RCIM slave systems are connected
using the cp command.
2. Disable the cable clock signal on the RCIM master system (using the d
command). The POSIX clocks on all systems should stop ticking.
3. Update the time values of the POSIX clocks on each system to the same
value (using the wp command).
4. Re-enable the cable clock signal on the RCIM master system (using the e
command). All clocks should begin ticking.
On RCIM systems equipped with the optional GPS module, the POSIX clock can be
synchronized to standard GPS time. Refer to the section “Using GPS for System
Timekeeping” below for information.
Automatic Synchronization1
For the RCIM III, each slave system can be configured so that its POSIX clock will be
automatically synchronized with that of the master on boot. To do this, uncomment either
the RCIMDATE=continuous or RCIMDATE=oneshot line in
/etc/sysconfig/rcim on the slave. Subsequently, each time the slave is booted it
will run rcimdate either once or continuously. If it is run once then the slave's POSIX
clock will match that of the master until such time as some application on the master
executes a clock_settime(3) (this is usually done by ntp or ptpd). If rcimdate is
run in continuous mode, then the slave will execute a clock_settime(3) within a few
seconds of each clock_settime(3) occurring on the master.
The system can also be configured to automatically synchronize the tick clocks when an
RCIM slave system is booted. This feature is disabled by default and should be used with
caution. It causes the tick clock on all systems to be reset to zero when any system in the
RCIM chain is booted, which may have an undesirable impact on processes using the tick
clock during synchronization.
To set up automatic synchronization of tick clocks when a slave system is booted, create
an empty file /etc/sysconfig/rcim_clocksync on the slave system. This
activates the /etc/init.d/rcim_clocksync startup script, asking it to reset its
RCIM board and (by default) also reset the slave RCIM boards. This uses ssh(1), which
must be set up to communicate between the slave and master systems.
Using GPS for System Timekeeping1
On RCIM systems equipped with the optional GPS module, the POSIX clock can be used
for system time synchronized to standard GPS time. The NTP daemon is utilized and
treats the GPS receiver as a time server. For information about how to configure NTP for
this support, see the section “ntp Configuration for GPS Support” in Chapter 2.
RedHawk Linux includes the RFC-2783 pulse per second (PPS) interface that
synchronizes the system time to the GPS PPS interface. The POSIX clock register is
captured on the on-time edge of the GPS PPS signal. This avoids the jitter which would be
introduced if an interrupt was used to capture the error between the system time and the
GPS time.
A dedicated serial interface, /dev/rcim_uart, is used by NTP to communicate with
the GPS receiver. This symbolic link is pointed to the last RCIM found with a GPS. If
another RCIM is to be used, the administrator must point this special device to the uart
special device file of the desired RCIM to, for example, /dev/rcim3/uart.
Likewise, the device which will receive the GPS pulse per second must be pointed to by
/dev/refclock-0. This symbolic link also points to the last RCIM found having a
GPS. If that is not the GPS that is wanted, the adminstrator must point this symbolic link
to, for example, /dev/rcim3/gps.
Tools such as ntpq(1) and ntpdc(1) can be used for monitoring NTP activity. Other
aids include various log files that can be added to /etc/ntp.conf if desired. Refer to
the ntpd(1) man page and www.ntp.org for details.
One or more of the following modules is used for interrupt processing on the RCIM:
• Edge-Triggered Interrupts (ETIs) – An ETI allows you to use an external
event to trigger an interrupt. Documentation for ETIs begins on page 3-12.
• Real-Time Clocks (RTCs) – An RTC allows you to set up a counter to
trigger an interrupt. Documentation for RTCs begins on page 3-14.
• External Output Interrupts – An external output signal allows you to use
any of the other signal processing modules as the source for an interrupt on
an external device. Documentation for external output interrupts begins on
page 3-15.
• Programmable Interrupt Generators (PIGs) – A PIG allows you to
programmatically generate a signal that can be used to trigger interrupts.
Documentation for PIGs begins on page 3-16.
• Distributed Interrupts (DIs) – Distributed interrupts allow you to
distribute signals or interrupts across all systems connected using an RCIM
chain. Documentation for DIs begins on page 3-17.
The sections that follow explain how the RCIM interrupts are processed.
Interrupt Processing Logic1
Interrupt requests, whether generated by the RCIM board or by a software request, are
handled by the edge-triggered interrupts (ETIs) and the distributed interrupts (DIs). Figure
3-1 illustrates how each interrupt request is processed.
DIs and ETIs must be armed and enabled for an interrupt to occur. Each interrupt can be
armed/disarmed and enabled/disabled individually. After power up initialization, all
interrupts are disarmed and disabled.
When the interrupt is armed, interrupt requests set a request bit. When an interrupt is
disarmed, any outstanding requests are turned off and ignored.
Requests for an enabled interrupt are allowed to enter the interrupt priority resolution
logic. The enable bit may be thought of as granting the RCIM board permission to deliver
any interrupt requests that it accepts. When the ETI or DI is disabled, it accepts interrupt
requests, but delays delivering the interrupt until it is re-enabled.
On RCIM III and RCIM II, the input to the interrupt block is used to drive the DI. On
RCIM I, the pending bit is the output of the ETI or DI. This output is routed to the host
computer to which the RCIM board is attached, and additionally to other systems in the
RCIM chain if configured to do so. The ETI or DI interrupt handler on the host clears the
pending bit each time it concludes the processing of an interrupt. This clears the way for
the RCIM board to output the next instance of this interrupt.
Arming and Enabling DIs and ETIs1
DIs and ETIs are armed using the ioctl(2) system call with the following operations,
which can be used interchangeably:
DISTRIB_INTR_ARM
ETI_ARM
DIs and ETIs are enabled using ioctl(2) with the following operations, which can be
used interchangeably:
DISTRIB_INTR_ENABLE
ETI_ENABLE
Interrupt Recognition Logic1
By default, the RCIM looks for the leading edge of an input signal to trigger an interrupt.
Once the interrupt is recognized, it must be deasserted and reasserted to cause a new
interrupt request. Optionally, an interrupt may be configured as level-sensitive. In this
mode, an interrupt is triggered when the interrupt signal is high or low. Configuration
options are included with the documentation for each type of interrupt later in this chapter.
For the RCIM to be able to reliably extract interrupts from an input signal, the equipment
generating the signal must hold any value generated for at least 1.5 microseconds before
transitioning to the reset value.
The RCIM provides the ability to share interrupts across interconnected systems using an
RCIM chain. Although distributed interrupts are covered in detail starting on page 3-17,
the figures below provide an illustration of how they operate. Guidelines for setting up
distributed interrupts based on the illustration follow.
Figure 3-2 Distributed Interrupt Operation Example
In Figure 3-2, there are three scenarios:
• A signal is generated on ETI
OUT
. The interrupt is also passed to the host system.
3
• An interrupt is received from the RCIM chain on DI
to the host system and also sent to an external device via OUT
• An interrupt is generated on PIG
OUT
.
Note that on RCIM III and RCIM II, the local interrupt does not drive the configured DI.
An
ETI_REQUEST ioctl will cause a local interrupt but will not affect the DI associated with
the ETI. For example, if ETI
input to DI
benefit of this is that a local interrupt is not issued for every distributed PIG and ETI. A
PIG should be used for a programmable software controlled interrupt and an ETI should
be used for an external interrupt output.
Please note the following in this example:
• A problem can occur when more than one interrupt module tries to drive
0
directly without passing through the local ETI0 interrupt control logic. A
0
the same signal line. In the example, the ETI
that drives an interrupt on DI0 and another on
3
. This interrupt is sent
7
that is passed to an external device via
0
is configured to drive DI0, it will connect the external ETI
and on OUT3. It is possible to configure another signal processing
DI
0
module (say RTC
) to drive the interrupts on DI0 and OUT3 at the same
0
time. In this case, the signal that drives the line will be the one with the
strongest amplifier. It is up to the administrator to avoid this condition.
• You determine which direction a distributed interrupt takes. This means
that on a system where a distributed device resides, it may generate two
interrupts: its local device interrupt (ETI, PIG or RTC) and the distributed
interrupt. There is a separate interrupt vector for each.
It may be desirable to receive both interrupts, but generally only one is
sufficient. By disarming the distributed interrupt, one can prevent that
interrupt from being generated on the local system. By default, a
distributed interrupt is in a disarmed state.
Obtaining RCIM Values1
There are several methods available for displaying or obtaining RCIM values:
/proc/driver/rcimN filesystem (where N starts from zero):
The following files in this filesystem can be viewed (read only unless noted
otherwise):
config – the RCIM configuration in a form suitable to cut and paste (read/write)
interrupts – a count of all ETI, DI and RTC interrupts per CPU and in total
status – miscellaneous RCIM board status and time synchronization
rawregs – named hex display of all readable RCIM board registers
rtc – status of the RTCs (run status, count values, etc.)
eti – status of the ETIs (armed, enabled, etc.)
di – status of the DI lines (armed, enabled, etc.)
ioctl(2) system call:
Information about a specific interrupt type can be retrieved by specifying the
appropriate operation with the appropriate device file mmap’d into the address
space of the program; for example, ETI_INFO with /dev/rcim0/eti1. Refer
to the rcim_eti(4), rcim_rtc(4) and rcim_distrib_intr(4) man
pages.
The
RCIM_GET_INFO operation with the /dev/rcim0/rcim device file mmap’d
provides the same information as /proc/driver/rcim0/config.
The
RCIM_GET_ADDR operation with the /dev/rcim0/rcim device file mmap’d
provides the virtual and physical address of the RCIM control registers.
The header file /usr/include/rcim.h describes the layout of the
information returned by
RCIM_GET_INFO and RCIM_GET_ADDR. Refer to the
rcim(4) man page.
mmap(2) system call:
mmap can be used to map in some or all of the device registers of the RCIM board.
The register layout is in /usr/include/linux/rcim_ctl.h and in
Appendix A in this guide.
Each RCIM board has incoming external interrupt lines, called ETIs or edge-triggered
interrupts, so-named after their most common mode of operation. These lines permit users
to provide their own interrupt sources. The RCIM processes and delivers these interrupts
to the host system and, if they are distributed, routes and delivers them to all other RCIMs
in the chain as distributed interrupts. RCIM III and RCIM II support twelve ETIs (0-11);
RCIM I supports four ETIs (0-3).
Each ETI is independently configurable. An ETI may treat the incoming signal as an edge
or level sensitive interrupt. If edge sensitive, it may raise an interrupt on either the rising
or the falling edge. If level sensitive, it may raise interrupts for either the high or the low
signal value. To specify how the incoming signal pattern is converted to interrupt requests,
use one of the ETI configuration options described under “ETI Configuration”.
Applications in turn arm or disarm, enable or disable each ETI on each system on the fly
as appropriate to the needs of those applications.
One requirement the RCIM imposes on external signal generating equipment is that the
signals they output must hold any low or high signal value for at least 1.5 microseconds
before changing to the next state. Pulses of shorter duration may not be recognized by the
interrupt controller. As long as the pulse is longer than 1.5 microseconds, the duration of
the pulse is not important.
The rcim_eti(4) man page provides complete information about ETIs.
ETI Configuration1
Each edge-triggered interrupt can be configured to trigger on the rising or falling edge of a
signal, or on a high or low signal value using the eti configuration option. This option
has the following syntax:
etiN /[rising|falling|high|low]
The default setting for an ETI is “falling.”
The flag words (rising, falling, high, low) can be specified using the first character
of the word. These words are not case sensitive.
Examples include:
eti0/falling
eti1/r
eti2/h
See the “Configuration” section in Chapter 2 or the rcim(4) man page for the various
methods available for specifying configuration options.
sets ETI 0 to trigger on the falling edge of its input signal
sets ETI 1 to trigger on the rising edge of its input signal
Each ETI is accessed through its own special device file:
/dev/rcimN/etiM
where N is the RCIM card number (starting from zero) and M is the ID of the ETI.
These files are created automatically on system boot by the /etc/init.d/rcim
initialization script.
User Interface to ETIs1
An ETI is controlled by open(2), close(2), and ioctl(2) system calls. Note that
this device does not support the read(2), write(2) or mmap(2) system calls.
The open call assigns a file descriptor to one edge-triggered interrupt and verifies that the
interrupt is not currently being used by another device driver. One device file exists for
each edge-triggered interrupt. A close call frees the file descriptor and removes any
attached signals. Refer to the man pages for more information.
The following commands to ioctl are used to manipulate the ETIs. These commands
can also be applied to DIs. All ioctl calls use the constants defined in
/usr/include/rcim.h. Refer to the rcim_eti(4) man page for more information.
ETI_ARMarms the ETI
ETI_DISARMdisarms the ETI
ETI_ENABLEenables the ETI
ETI_DISABLEdisables the ETI
ETI_REQUESTgenerates a software requested interrupt
ETI_INFOgets information about the ETI
ETI_WAITcauses the process to sleep
ETI_WAKEUPwakes all sleeping processes
ETI_GETICNTreturns the number of times this ETI has fired
ETI_KEEPALIVE sets or clears the keepalive state
ETI_VECTORgets the interrupt vector associated with ETI
IOCTLGETICNTreturns the number of times this ETI has fired (generic)
IOCTLKEEPALIVEsets or clears the keepalive state (generic)
IOCTLVECNUMsets the interrupt vector associated with ETI (generic)
IOCTLSIGATTACHrequests a signal when an RCIM device generates an interrupt
Note that like DIs, an ETI must be armed and enabled before an interrupt can be received.
Any or all of the ETIs on an RCIM can be distributed to all systems connected by an
RCIM chain. The source of a distributed ETI can be located on any of the RCIMs in the
chain.
To determine if a specified ETI has its interrupts sent to all connected systems, use one of
the methods described in the section entitled “Obtaining RCIM Values” on page 3-11. See
the “Distributed Interrupts” section on page 3-17 for information about setting up
distributed interrupts.
Real-Time Clocks (RTCs)1
The RCIM provides real-time clock timers. Each of these counters is accessible using a
special file and each can be used for almost any timing or frequency control function.
RCIM III and RCIM II support eight 32-bit RTCs (0-7); RCIM I supports four (0-3).
The real-time clock timers are programmable to several different resolutions which, when
combined with a clock count value, provide a variety of timing intervals. This makes them
ideal for running processes at a given frequency (e.g., 600Hz) or for timing code
segments. The timers may be one-shot or periodic; if periodic, the original load value is
automatically reloaded into the counter each time zero is reached.
In addition to being able to generate an interrupt on the host system, the output of an
RCIM real-time counter can be distributed to other RCIM boards for delivery to their
corresponding host systems, or delivered to external equipment attached to one of the
RCIM’s external output interrupt lines.
The rcim_rtc(4) man page provides complete information about RTCs.
RTC Device Files1
Each RTC is accessed through its own special device file:
/dev/rcimN/rtcM
where N is the RCIM card number (starting from zero) and M is the ID of the RTC.
These files are created automatically on system boot by the /etc/init.d/rcim
initialization script.
Distributed RTCs1
Any or all of the RTCs on an RCIM can be distributed to all systems connected by an
RCIM chain. The source of a distributed RTC may be located on any of the RCIMs in the
chain.
3-14
To determine if a specified RTC has its interrupts sent to all connected systems, use one of
the methods described in the section entitled “Obtaining RCIM Values” earlier in this
chapter. See the “Distributed Interrupts” section on page 3-17 for information about
setting up distributed interrupts.
A real-time clock timer is controlled by open(2), close(2), and ioctl(2) system
calls. The close system call, if it closes the last open to the device, stops the RTC and
clears its settings unless the
before the close.
The parameters passed through ioctl control the modes of the real-time clock timer, the
clock count value, and counting itself, as well as getting the current settings of the RTC.
This device does not support the read(2) and write(1) system calls.
The following commands to ioctl are used to manipulate the RTCs. All ioctl calls
use the constants defined in /usr/include/rcim.h. Refer to the rcim_rtc(4)
man page for further information.
RTC IOC SE TLinitializes RTC values (32-bit interface)
RTCIOCGETLretrieves RTC values (32-bit interface)
RTC IOC SE Tinitializes RTC values (16-bit interface)
RTCIOCGETretrieves RTC values (16-bit interface)
RTCIOCSETCNTsets RTC clock count
RTCIOCMODCNTmodifies RTC clock count
RTCIOCGETCNTgets RTC clock count
RTCIOCRESgets RTC clock resolution
RTC IOC STARTstarts RTC counting
RTC IOC STO Pstops RTC counting
RTC IOC WAITblocks until RTC clock count reaches zero
RTCIOCWAKEUPwakes all sleeping processes
RTC IOC IN FOgets information about RTC
IOCTLGETICNTreturns the number of times this clock has fired
IOCTLVECNUMsets interrupt vector for RTC
IOCTLKEEPALIVEdoes not destroy the timer on final close
IOCTLSIGATTACHrequests a signal when this RTC generates an interrupt
IOCTLKEEPALIVE ioctl command was issued to the RTC
External Output Interrupts1
Each RCIM provides external output signals. These signals can be used as interrupt
sources for other machines or used as signals to control external devices. RCIM III and
RCIM II support twelve external output interrupts (0-11); RCIM I supports four (0-3).
The external output interrupts can be driven from one of several sources internal to the
RCIM. The most common source is from the programmable interrupt generators (PIGs).
PIGs provide full software control for generation of the output signals.
The pin-outs for the external interrupt connectors are described in Chapter 2.
Defaults for this configuration option are a PIG as source for the corresponding output
line; i.e.:
pig0|out0
pig1|out1
etc.
See the “Configuration” section in Chapter 2 or the rcim(4) man page for the various
methods available for specifying configuration options.
sets output line 0 to be driven by real-time clock 3
sets output line 2 to be driven by distributed interrupt 5
Programmable Interrupt Generators (PIGs)1
Each RCIM provides programmable interrupt generators (PIGs). PIGs are generally used
to provide software control for the output of an external output signal. Additionally, the
PIGs can be used to drive a signal onto a distributed interrupt line, permitting user
software to generate distributed interrupts that are simultaneously delivered to all RCIMs
in an RCIM chain. RCIM III and RCIM II support twelve PIGs (0-11); RCIM I supports
four PIGs (0-3).
The rcim_pig(4) man page provides complete information about PIGs.
PIG Device File1
The device file /dev/rcimN/pig is used to access the PIG register on the local system.
This file must be mapped into the address space of a program using mmap(2). By default,
only users with root privileges have access to do this.
On RCIM III and RCIM II, the PIG register is 12 bits wide, one bit for each PIG. Two
additional registers allow the PIG bits to be set and cleared in a multiprocessor safe
manner. In the mmap’ed PIG register page, the set register (PIGS) is at offset 0x10 and the
clear register (PIGC) is at offset 0x20. Setting a PIG generates a distributed interrupt or
external output, depending on how the PIGs are connected. The required length of the
signal depends upon the requirements of the attached device. If the signal is being fed into
another RCIM, it must hold any low or high value for at least 1.5 microseconds before
changing to the next state.
On RCIM I, the PIG register is a 32-bit register, one bit for each PIG. PIG 0 is controlled
by bit 0 (the least significant bit), PIG 1 by bit 1, etc. The remaining bits are unused.
Writing a value to the bit corresponding to a PIG causes the inverted signal value to be
emitted from the PIG RCIM signal generator. This can be used to control external devices
or generate distributed interrupts, depending on how the PIGs are connected. The required
length of the signal depends upon the requirements of the attached device. If the signal is
being fed into another RCIM, it must hold any low or high value for at least 1.5
microseconds before changing to the next state.
Distributed PIGs1
Any or all of the PIGs on an RCIM can be distributed to all systems connected by an
RCIM chain. The source of a distributed PIG may be located on any of the RCIMs in the
chain.
To determine if a specified PIG has its interrupts sent to all connected systems, use one of
the methods described in the section entitled “Obtaining RCIM Values” earlier in this
chapter. See the “Distributed Interrupts” section below for information about setting up
distributed interrupts.
Distributed Interrupts1
The real heart and power of the RCIM lies in its distributed interrupt system. Each RCIM
can distribute interrupts simultaneously to all systems connected via an RCIM chain.
RCIM III and RCIM II support a total of twelve distributed interrupts (0-11); RCIM I
supports a total of eight (0-7). A diagram of this functionality and guidelines for setting up
distributed interrupts can be found in the section “Setting up Distributed Interrupts” earlier
in this chapter.
Any of the edge-triggered interrupts, real-time clock timers or programmable interrupt
generators on any of the RCIM boards in the chain can be configured to be distributed. A
distributed device file is associated with each of the distributed interrupts.
RCIM distributed interrupts must be configured on each system attached to the RCIM that
is either broadcasting or intending to receive a distributed interrupt. Distributed interrupts
can also be configured and used locally on an isolated system. Configuration details are
given in the “DI Configuration” section below. See the section “Obtaining RCIM Values”
on page 3-11 for the methods available for obtaining configuration information.
The rcim_distrib_intr(4) man page provides complete information about DIs.
It is important that all RCIM-connected systems have a compatible configuration for the
distributed interrupt lines of the RCIM.
By default, no distributed interrupts are configured.
Distributed interrupts must first have a source, and then can be configured to trigger on the
rising or falling edge of a signal, or on a high or low signal value.
To define the source for a distributed interrupt, use the following configuration option:
<source>|diN
The value specified for the source can be one of the following:
nonethe RCIM is not to drive this distributed interrupt
Examples include:
rtc3|di6sets distributed interrupt 6 to be driven by real-time clock 3
pig1|di3sets distributed interrupt 3 to be driven by programmable interrupt
generator 1
none|di0the RCIM is not to drive distributed interrupt 0
Each distributed interrupt can be configured to trigger on the rising or falling edge of a
signal, or on a high or low signal value using the di configuration option. This option has
the following syntax:
diN/[rising|falling|high|low]
The flag words (rising, falling, high, low) can be specified using the first
character of the word. These words are not case sensitive.
Examples include:
di0/falling
di1/r
See the “Configuration” section in Chapter 2 or the rcim(4) man page for the various
methods available for specifying configuration options.
sets distributed interrupt 0 to trigger on the falling edge of its input
signal
sets distributed interrupt 1 to trigger on the rising edge of its input
signal
Each distributed interrupt is accessed through its own special device file:
/dev/rcimN/diM
where N is the RCIM card number (starting from zero) and M is the ID of the distributed
interrupt.
These files are created automatically on system boot by the /etc/init.d/rcim
initialization script.
User Interface to DIs1
A distributed interrupt is controlled by open(2), close(2), and ioctl(2) system
calls. Note that this device does not support the read(2), write(2) and mmap(2)
system calls.
The open call assigns a file descriptor to one distributed interrupt. A close call frees the
file descriptor and, if it is the last close, disarms the interrupt if the
is clear. Refer to the man pages for more information.
IOCTLKEEPALIVE state
The following commands to ioctl are used to manipulate distributed interrupts. These
commands can also be applied to ETIs. All ioctl calls use the constants defined in
/usr/include/rcim.h. Refer to the rcim_distrib_intr(4) man page for
more information.
DISTRIB_INTR_ARMarms the DI
DISTRIB_INTR_DISARMdisarms the DI
DISTRIB_INTR_ENABLEenables the DI
DISTRIB_INTR_DISABLEdisables the DI
DISTRIB_INTR_REQUESTgenerates a software requested interrupt
DISTRIB_INTR_INFOgets information about the DI
DISTRIB_INTR_WAITsleeps until the next DI
DISTRIB_INTR_WAKEUPwakes up all sleeping processes
DISTRIB_INTR_KEEPALIVE sets or clears keepalive state
DISTRIB_INTR_GETICNTreturns the number of times this DI has fired
DISTRIB_INTR_VECTORgets interrupt vector for the DI
IOCTLKEEPALIVEsets or clears keepalive state (generic)
IOCTLGETICNTreturns the number of times this DI has fired (generic)
IOCTLVECNUMsets interrupt vector for the DI (generic)
IOCTLSIGATTACHrequests a signal when an RCIM device generates an interrupt
Note that like ETIs, a distributed interrupt must be armed and enabled before an interrupt
can be received.
Figure A-3 RCIM III Interrupt Enable/Request/Pending/C le ar/ Arm/L eve l/Po l arit y Re gi st ers
(IER, IRR, IPR, ICR, IAR, ISLR, ISPR)
The enable registers (IER) enable the selected interrupts.
The request registers (IRR) are software driven requests of the selected interrupts.
The pending registers (IPR) are pending requests.
The clear registers (ICR) clear the selected interrupts.
The arm registers (IAR) arm the selected interrupts for edge triggering.
The level registers (ISLR) set level (1) or edge (0) for the selected interrupts.
The polarity registers (ISPR) set polarity high (1) or low (0) for the selected interrupts.
The PPS Snapshot register contains a snapshot of the nanoseconds field and two bits of the seconds field of
the POSIX clock. The snapshot is taken every time the GPS PPS signal occurs.
Offset: 0200
The Cable Snapshot register contains a snapshot of the nanoseconds field and two bits of the seconds field of
the POSIX clock. The snapshot is taken every time the cable master time is received.
Figure A-6 RCIM III PPS Snapshot Register (PPS)
Figure A-7 RCIM III Cable Snapshot Register (CSR)
Offset: 0210
Figure A-8 RCIM III Cable Master Time Register (CMTR)
The Cable Master Time register contains the seconds field of the master RCIM POSIX clock that is
transmitted on the cable at every transition of the clock at the seconds boundary.
The clock frequency adjust register is used to control the frequency of the 10 Mhz master clock.
Offset: 1120
The initial RTC timer value is loaded in the RTC timer registers. The current value of the timer is read from
this register. NOTE: Loading this register also loads the RTC Repeat Register for compatibility with RCIM.
Figure A-19 RCIM III Clock Frequency Adjust Register (CFAR)
The GPS transmit pointers are used for communication with the optional GPS module.
Offset: 3204
Figure A-27 RCIM III GPS Transmit Pointers (GTXP)
Figure A-28 RCIM III GPS Debug Control Register (GDCR)
The GPS debug control register contains bits used during testing and debug. Setting any of these bits will
disable RCIM communication with the GPS module.
Figure A-35 RCIM II Interrupt Enable/Request/Pending/Clear/Arm/Level/Polarity Regis ters
(IER, IRR, IPR, ICR, IAR, ISLR, ISPR)
The enable registers (IER) enable the selected interrupts.
The request registers (IRR) are software driven requests of the selected interrupts.
The pending registers (IPR) are pending requests.
The clear registers (ICR) clear the selected interrupts.
The arm registers (IAR) arm the selected interrupts for edge triggering.
The level registers (ISLR) set level (1) or edge (0) for the selected interrupts.
The polarity registers (ISPR) set polarity high (1) or low (0) for the selected interrupts.
Setting a bit in a PCI interrupt routing register routes selected interrupts to the designated PCI interrupt. The
default power-up value is everything routed to PCI A. Setting bits in multiple registers will drive multiple
PCI interrupt lines.
The PPS Snapshot register contains a snapshot of the nanoseconds field and two bits of the seconds field of
the POSIX clock. The snapshot is taken every time the GPS PPS signal occurs.
Offset: 0200
Figure A-41 RCIM II GPS Transmit/Receive Register (GPS)
The GPS Transmit/Receive register is used for serial communication with the GPS module via the PCI bus
interface.
Offset: 0300
Figure A-42 RCIM II Clear Cable Errors Register (CCERR)
This is a Write Only register that clears any reported cable errors. The data field is don’t care.
The initial RTC timer value is loaded in the RTC timer registers. The current value of the timer is read from
this register. NOTE: Loading this register also loads the RTC Repeat Register for compatibility with RCIM.
Figure A-56 RCIM I Interrupt Enable/Request/Pending/Clear/ARM/Level/Polarity Registers
The enable register (IER) enables the selected interrupt.
The request register (IRR) is a software driven request of the selected interrupt.
The pending register (IPR) is a pending request.
The clear register (ICR) clears the selected interrupt.
The arm register (IAR) arms the selected interrupt for edge triggering.
The level register (ISLR) sets level (1) or edge (0) for the selected interrupt.
The polarity register (ISPR) sets polarity high (1) or low (0) for the selected interrupt.