Chapter 84 SCL (ServerNet Cluster Subsystem) Messages

Cause  The ServerNet cluster monitor process has been started by an operator or by the persistence manager process ($ZPM) after a failure of both the primary and backup ServerNet cluster monitor processes. (The ServerNet cluster monitor process has no means of distinguishing between the two cases.)

Effect  The ServerNet cluster monitor process is running.

Recovery  This is an informational message. No corrective action is required.

Cause  The ServerNet cluster monitor process terminated voluntarily, either by an operator command or because an environmental problem caused it to self-terminate. If this event is due to self-termination, an SCL 1010 message reported the environmental problem.

Effect  The ServerNet cluster monitor process is no longer running.

Recovery  If this event is due to self-termination, follow recovery instructions for the SCL 1010 message. After correcting any environmental problems, restart the ServerNet cluster monitor process with an operator command.

Cause  Either the ServerNet cluster monitor process was initialized for the first time or a backup process has become the primary process.

Effect  The ServerNet cluster monitor process is running in the indicated processor.

Recovery  This is an informational message. No corrective action is required.

Cause  The ServerNet cluster monitor process successfully created a backup process.

Effect  The ServerNet cluster monitor process is no longer vulnerable to a single failure.

Recovery  This is an informational message. No corrective action is required.

Cause  An attempt to create a ServerNet cluster monitor backup process has failed. For information on process creation errors and error detail subcodes, see the Guardian Procedures Errors and Messages Manual.

Effect  Until a backup process is started, the ServerNet cluster monitor process is vulnerable to a single failure. The ServerNet cluster monitor process attempts to start a backup process immediately if any processor in its processor list, other than that used by the primary process, is running. The ServerNet cluster monitor process makes two restart attempts in each processor eligible to contain the backup process. Each failed attempt results in an SCL 1005 message. If all attempts fail, an SCL 1007 message is generated.

Recovery  This is an informational message. Although no corrective action is required, the data in this message might provide information for recovery in the event of an SCL 1007 message.

Cause  The backup process of the ServerNet cluster monitor process pair failed.

Effect  Until a new backup process is started, the ServerNet Cluster monitor process is vulnerable to a single failure. The ServerNet cluster monitor process attempts to start a new backup process immediately if any processor in its processor list, other than that used by the primary process, is running. The ServerNet cluster monitor process makes two restart attempts in each processor eligible to contain the backup process. Each failed attempt results in an SCL 1005 event. If all of these attempts fail, an SCL 1007 event is generated.

Recovery  This is an informational message. Although no corrective action is required, the data in this message might provide information for recovery in the event of an SCL 1007 event.

Cause  Either no processor is available for running the backup process, or there have been multiple failures of the backup process or attempts to create a backup process. If this event is due to backup process creation failures, there will have been associated SCL 1005 messages generated. Other possible precursors are SCL 1006 and SCL 1008 messages.

Effect  The ServerNet cluster monitor process is running without a backup, and the ServerNet cluster subsystem is vulnerable to a single failure. Whenever a processor in the processor list is reloaded, the ServerNet cluster monitor process attempts to create a backup there.

If this event is caused by repeated backup failures or backup process creation failures, the ServerNet cluster monitor process periodically attempts to create a backup.

Recovery  Either reload the processors on the processor list for the ServerNet cluster monitor process, or use the information in the associated SCL 1005 messages to determine the cause and recovery actions for backup process creation failures.

To list the processors that are configured in the processor list for the ServerNet cluster monitor process, issue an SCF INFO command. For example:

-> INFO PROCESS $ZZKRN.#ZZSCL, DETAIL

Cause  The backup processor failed, or the backup process was terminated by the primary process (for example, if the primary process found a fatal error when checkpointing to the backup.)

Effect  Until a new backup process has been started, the ServerNet cluster monitor process is vulnerable to a single failure. The ServerNet cluster monitor process attempts to start a new backup process immediately if any processor in its processor list, other than that used by the primary process, is running. The ServerNet cluster monitor process makes two restart attempts in each processor eligible to contain the backup process. Each failed attempt results in an SCL 1005 message. If all of these attempts fail, an SCL 1007 message is generated.

Recovery  This is an informational message. Although no corrective action is required, the data in this message might provide information for recovery in the event of an SCL 1007 message.

Cause  An internal trace was initiated on the ServerNet monitor process.

Effect  Trace data is dumped into the EMS log.

Recovery  This is an informational message. No corrective action is required.

Cause  The ServerNet cluster monitor process found one or more environmental problems. The possible environmental problems are described in more detail under “Recovery.” This message is usually followed by an SCL 1002 termination message.

Effect  This is an informational message, but for certain environmental problems, the ServerNet Cluster monitor process terminates.

Recovery  Corrective action might be required to fix one or more environmental problems:

Cause  The ServerNet cluster monitor process encountered an error using the HP NonStop Kernel configuration services application program interface (API).

Effect  If this error occurs during process startup, the ServerNet cluster monitor process uses STARTSTATE STOPPED (default) instead of using the data stored in the private configuration record. If this error occurs later, the action is prompted by an SCF [ALTER | START | STOP] SUBSYS command. In this case, the command fails with an error.

Recovery  Investigate the cause of the error. Restart the process or reissue the failed SCF command.

Cause  The ServerNet cluster monitor process encountered an error using the DSM trace routines.

Effect  Any pending trace is terminated.

Recovery  Investigate the cause of the error. Reissue the SCF TRACE command. If the problem persists, contact your service provider.

Cause  The ServerNet cluster monitor process attempted to register for ServerNet SAN manager (SANMAN) notifications relative to external fabric connection status changes. However, no room remained in the Fabric Access Control (FAC) structure in system global memory, so the process could not be registered.

Probably all registration slots are being consumed by previously registered clients. Currently, the maximum number of clients allowed to register simultaneously is 32.

Effect  The ServerNet Cluster subsystem is placed in the STOPPED state. Direct ServerNet communication with processors on other servers, even when physically connected, is not possible.

Recovery  Stop at least one of the processes currently registered. Then use the SCF START SUBSYS command to restart the ServerNet Cluster subsystem.

Cause  The ServerNet cluster monitor process has determined that the physical connection to the ServerNet Cluster is not ready for ServerNet connections.

Probably the ServerNet cluster monitor process was running before the ServerNet SAN manager process could communicate with the external fabrics or program the Node Numbering Agents (NNAs).

Effect  The ServerNet Cluster subsystem remains in the STOPPED state. Direct ServerNet communication with processors on other servers is not possible.

Recovery  The ServerNet SAN manager process automatically notifies the ServerNet cluster monitor process when the external fabric connections are ready for use. The ServerNet cluster monitor process then retries the Start command.

Cause  A message monitor process found some environmental problems such as the process name being wrong (not $ZIMnn). This event is usually followed by a message monitor termination event.

Effect  This is an informational message, but for certain environmental problems, the message monitor process terminates.

Recovery  Alter the message system monitor process’s (MSGMON) startup parameters (generic configuration under SCF).

Cause  The ServerNet cluster monitor process or the message system monitor process compared its own version and that of the NonStop Kernel message system and determined that the versions were incompatible. This event will be followed by a process termination event.

Effect  The ServerNet cluster monitor process or the message system monitor process terminates.

Recovery  Ensure that the versions of the ServerNet cluster monitor process and message system monitor process (T0294) and the NonStop Kernel message system (T9050) are compatible. Check SPR requisites for T9050 and T0294.

Cause  Both the X and Y paths from the local processor to the indicated remote processor are down.

This message is preceded by one or more IPC (NonStop Kernel Message System) 110 messages generated by the local processor. The IPC 110 messages report the causes of the failures.

If the lost connection is due to the failure of the remote processor, an SCL 1102 message is generated.

	NOTE: If the ServerNet cluster monitor process receives the remote processor down information in time, the SCL 1100 message is suppressed.

Effect  All intersystem ServerNet traffic between the indicated local and remote processors is routed via the Expand-over-ServerNet line-handler process. Consequently, transmission is slower.

Recovery  This is an informational message. For recovery information, see the accompanying IPC 110 or SCL 1102 messages.

Cause  One or both of the paths between the indicated processors is restored. An IPC (NonStop Kernel Message System) 111 message with additional information is generated by the local processor.

	NOTE: This event is not generated during ServerNet cluster monitor process initialization.

Effect  Direct ServerNet communication between the processors is possible.

Recovery  This is an informational message. No corrective action is required.

Cause  A remote processor failed. This event is logged on every other ServerNet cluster member when a processor on a node fails.

Effect  ServerNet paths between all local processors and the remote processor are taken down. The local processors suppress IPC 110 messages in this case. Possibly local processors detected path failures and logged path down events before being informed of the remote processor's failure. The particular sequence of events depends on the speed with which the ServerNet cluster monitor process is informed of the remote processor's failure and the levels of message traffic from the local system to the failed remote processor.

Recovery  Reload the remote processor.

Cause  A remote processor is reloaded, and its connections with this system are restored. This event is logged on every other ServerNet cluster member when a processor on a node is reloaded.

Effect  Direct ServerNet message traffic with the indicated processor can resume.

Recovery  This is an informational message. No corrective action is required.

Cause  A local processor is reloaded.

Effect  Local connections of the processor are restored.

Recovery  This is an informational message. No corrective action is required.

Cause  A remote processor detected that its connection to the indicated ServerNet fabric failed. Possibly the other PMF CRU in the processor enclosure was removed. The indicated processor logs an IPC (NonStop Kernel Message System) 112 message on its local system.

Effect  ServerNet paths on the indicated fabric between all local processors and the remote processor are taken down. Direct ServerNet message traffic is still possible by using the other fabric. When the local processors down the paths to the remote processor on the indicated fabric, IPC 110 message logging is suppressed.

Possibly local processors detected path failures and logged path down events before being informed of the remote processor's fabric failure. The particular sequence of events depends on the speed with which the Servernet cluster monitor process is informed of the remote processor's fabric failure and the levels of message traffic from the local system to the remote processor.

Recovery  This is an informational message. For recovery information, see the IPC 112 message.

Cause  A remote processor regained connectivity to the indicated ServerNet fabric.

Effect  Paths between the local system are restored and paths between the local system and the indicated remote processor on the indicated fabric wait for automatic recovery kick in. The system on which the processor resides logs an IPC (NonStop Kernel Message System) 113 message.

Recovery  This is an informational message. No corrective action is required.

Cause  All individual processor-to-processor paths over the indicated fabric from the local system to the remote system failed. These failures are documented by IPC (NonStop Kernel Message System) 110 messages generated by the individual processors. Possibly a ServerNet router or cable failed. If both fabrics are indicated through the IPC 110 messages, the remote system itself might have failed or lost power.

Effect  All communication with the remote system over the given fabric ceases.

Recovery  If only one fabric is involved, investigate the condition of the intervening ServerNet routers and cables. If both fabrics are involved, investigate the condition of the remote system itself. In either case, the associated IPC 110 messages might provide further recovery information.

Cause  ServerNet direct connectivity to a remote system is lost for the reason stated in the message. The remote system failed, lost power, or has a duplicate system number. The local system might have lost power. Router or cable failures might have occurred on both ServerNet fabrics.

Effect  All connections with the remote system are shut down.

Recovery  Depending upon the reason indicated, correct the problem.

For example, bring up the failed processors or bring up/replace the failed cable or router.

If a protocol error occurred, there might be an associated HP Tandem Failure Data System (TFDS) (TFDS subsystem ID: DMP) failure capture event for the primary ServerNet cluster monitor process.

If the reason was Duplicate System Number, change the Expand node number of the newly connected node to a unique number (range 0 to 254) in the cluster. For more information about changing the system number, see the SCF Reference Manual for the Kernel Subsystem and contact your service provider.

Cause  A ServerNet connection is established with a remote system. This connection might be the initial one caused by starting ServerNet cluster services on either of the systems, or it might be the recovery of a failed connection.

Effect  Direct message system traffic of ServerNet between the two systems is possible.

Recovery  This is an informational message. No corrective action is required.

Cause  The service processor found a configuration error when checking each ServerNet fabric prior to starting ServerNet cluster services.

The tokens in the event contain details of the configuration error, including:

Effect  If either the X or Y fabric is correctly configured, ServerNet cluster services move to the STARTING state and connections with remote systems are established on the correctly configured fabric.

If both fabrics are incorrectly configured, ServerNet cluster services remain in the STOPPED state, and no connections with remote systems are established.

Recovery  Determine the details of the configuration error and correct it.

Cause  This system is the first in the ServerNet cluster to be started, or a ServerNet connectivity failure occurred.

Effect  The system attains the STARTED state, but there is no ServerNet connectivity with other systems.

Recovery  If this system is the first in the ServerNet cluster to be started, no corrective action is required. Otherwise, repair the ServerNet connectivity problem.

Cause  The MS Driver in the reporting processor could not register to listen to permissive packets.

Effect  Any discovery packets sent to the reporting processor are not delivered to the MS Driver. The MS Driver in the reporting processor cannot find out about any packets received in the permissive Access Validation and Translation Table Entry (AVTTE). If there are no registered processors in the target system, the target system can neither be discovered nor can it generate any SCL 1114 events.

Recovery  Stopping one or more subsystems that register for permissive packets should allow the MS Driver to register itself. MS Driver registration occurs when a processor is loaded or reloaded, when the message monitor process is initialized, and when the ServerNet cluster monitor process is initialized. The recommended long-term solution is to request a TNet Services (T8460) SPR capable of supporting a larger number of registered listeners.

Cause  A discovery failure was detected by the sender system. The ServerNet cluster monitor process could not discover the target system.

Effect  A ServerNet cluster connection to the target system is not established. The systems perform periodic retries to discover each other. Once the cause of failure is corrected, discovery should proceed automatically.

Recovery  The recovery procedure depends on the value of reason:

Cause  A discovery failure was detected by the target system. The ServerNet cluster monitor process could not discover the sender system.

Effect  A ServerNet cluster connection to the target system is not established. The systems perform periodic retries to discover each other. Once the cause of failure is corrected, discovery should proceed automatically.

Recovery  The recovery procedure depends on the value of reason:

Cause  The connection between the ServerNet cluster monitor process (SNETMON) on the local system and SNETMON on a remote system is lost due to a low-level communication problem.

Effect  The connection between the SNETMON process on the local node and the SNETMON process on the remote node is lost.

Recovery  This is an informational message. No corrective action is required; recovery is automatic.

Cause  There is at least one working path on the specified fabric from the local node to the specified remote system now, when previously there were none. The path state changes are documented by IPC 111 events generated by the individual processors.

Effect  There is at least partial connectivity with the remote system over the given fabric.

Recovery  This is an informational message only. No corrective action is required.

Cause  The ServerNet cluster monitor process (SNETMON) made a call to an SP I/O library routine, but it failed.

Effect  The ServerNet cluster monitor process (SNETMON) automatically retries the SP I/O library routine, possibly after destroying its previous session with the Service Processor and starting a new session.

Recovery  The ServerNet cluster monitor process (SNETMON) automatically retries the SP I/O library routine. The Service Processor might have to be initialized, or the Service Processor firmware might have to be upgraded. Contact your service provider for assistance.

Cause  SNETMON invoked an SMC Driver API interface function. The function returned a code other than SMC_RTN_OK.

Effect  SNETMON will automatically retry the SMC Driver API call.

Recovery  This event is primarily provided for support personnel. SNETMON will automatically retry the SMC Driver API call. If the error persists (as indicated by additional SCL 1118 events), use the error and error detail codes in the event to determine and correct the cause of the failure.

Cause  The primary SNETMON process attempted to register with the SMC Driver, but registration failed.

Effect  The primary SNETMON process will not be able to communicate directly with remote nodes via ServerNet reads and will terminate itself. The backup SNETMON process will take over and attempt to register with the SMC Driver when it becomes the new primary. The SNETMON process pair will terminate if registration with the SMC Driver does not succeed in at least one processor in SNETMON's configured CPU list.

Recovery  SNETMON will automatically attempt to register with the SMC Driver on a different processor. A ZZSA* save abend file will be created when the primary SNETMON process terminates. The ZZSA* save abend file should be provided for analysis, along with the ZZSV* service log event file containing the SCL 1119 event.

Cause  SNETMON made a comparison of its own version and that of the SMC driver that it is using. The version was determined to be incompatible.

Effect  The ServerNet Cluster subsystem monitor process terminates.

Recovery  Check SPR requisites for the SMC Driver (T2800) and SNETMON (T0294). The operator needs to ensure that the versions of SNETMON and the SMC Driver are compatible.

Cause  Statistics data is sent to the EMS service log.

Effect  Statistics counters with nonzero values are written to the EMS service log. If a request to reset the statistics counter was received, all statistics counters are reset to 0.

Recovery  This is an informational message. No corrective action is required.