MTRotator PXI Controller Reboot (Soft, Hard and Control System Restart)

Overview

This document provides detailed instructions on how to perform a soft reboot, hard reboot, and restart the control system for the MTRotator PXI controllers at the summit. These procedures are essential for troubleshooting faults encountered during routine operations but might also be necessary during maintenance and other non-standard observatory operations, such as scheduled power off.

Error diagnosis

When troubleshooting the EUI/CSC failed to connect to the control system, follow the sequential order of the procedures below to re-establish control. If a procedure proves unsuccessful, proceed to the next one:

• Restart Control system

• Soft Reboot

• Hard Reboot or Power off

Prerequisites

• Ensure you have access to LSST-WAP network.

• Obtain the necessary IP, account user, and password information from the LSST 1Password MainTel Vault.

• Familiarity with Linux commands and the use of general Power Distribution Unit (PDU) or netbooter for power cycling.

Log Collection

Before starting the reboot procedure, collect the log files for debugging purposes:

1. Write down the exact timestamps of the error.

2. To copy log messages from PXI to your local machine:

   1. Access 1Password vault for credentials

   2. SSH into rot-pxi:

   ssh admin@rot-pxi-controller.cp.lsst.org
   

   3. Create a log file with current date:

   cat /var/log/messages > log_message_YYMMDD
   

   4. Exit SSH:

   exit
   

   5. Copy the log file to your local machine:

   scp admin@rot-pxi-controller.cp.lsst.org:~/log_message_YYMMDD ./
   

3. Attach all relevant log files and timestamps to OBS-664.

Procedure Steps

Restart Control System

Warning

As the primary troubleshooting step to regain control, perform the restart of the control system if the CSC/EUI is entirely unresponsive and unable to establish a connection with the control system.

1. Before proceeding, make sure there are no active interlocks in the EUI and that no other EUI instance is running. For the MTRotator, check if runRotEui is running by following commands in the terminal:

   ps -aux | grep runRotEui
   

   If processes are already running, you may need to identify who is running them and ask permission to end one (or both) so you can run your own EUI session. If another runRotEui is running:

   sudo kill -9 {pid}
   

2. You can do the following to check the MTRotator control system status in the terminal:

   /etc/init.d/rotator status
   

   It will tell you whether the control system is running or not. To stop it, do:

   /etc/init.d/rotator stop
   

   To start it do:

   /etc/init.d/rotator start
   

1. Establish an SSH connection to the MTRotator PXI: Using the credentials and hostnames found in the LSST 1Password MainTel Vault create an ssh tunnel in the terminal:

   ssh admin@rot-pxi-controller.cp.lsst.org
   

2. To restart the control system for the MTRotator, run the command in the terminal:

   /etc/init.d/rotator restart
   

3. Allow 5-10 minutes for the system to initialize: This period is necessary for the OS and control system to set up the EtherCat and Copley drives before making a connection through the CSC/EUI.

Warning

Only proceed with Soft Reboot if EUI/CSC control is not connecting and the restart of the control system procedure proved unsuccessful to regain control.

Soft Reboot

1. Establish an SSH connection to the MTRotator PXI using the credentials and hostnames found in the LSST 1Password MainTel Vault (See above).

2. Execute the reboot command: To initiate a soft reboot of the PXI, type in the terminal:

   sudo reboot now
   

3. Allow 5-10 minutes for the system to reboot: This time is necessary for the OS and control system to configure the EtherCat and Copley drives before attempting a connection through the CSC/EUI.

Hard Reboot

Warning

Only proceed with a hard reboot, if the EUI control connection remains unsucessful after a soft reboot.

This method involves cutting power to the PXI and drives and should only be used as a last resort due to the potential risks.

If a power shutdown is scheduled, you can proceed until step 2 before the power on.

1. Login into Utilities cabinet Power Distribution Unit (PDU): While in the LSST-WAP, connect to https://tea-pdu01.cp.lsst.org/ using the credentials stored in the Operators vault of LSST 1Password as PDU Utilities Cabinet. Click on Outlets on the left hand side menu to open the outlets screen. The description of each outlet can be found here.

   

2. Power Cycle PXI and drives: To power Off the system, first power off the PXI, followed by the drive. For MTRotator (Camera Rotator in PDU), the PXI is connected to outlet 6, and drives in outlet 5.

   Note

   Scheduled Power Off

   In case a scheduled power off is intended, do not continue with power on, and stop the procedure here.

   When powering On, activate the drives first, and wait for at least 3 minutes before powering on the PXI. This delay is crucial for the Ethercat application within the PXI to establish a connection with the Copley drive.

3. Wait for an additional 5-10 minutes after powering on before using the CSC/EUI: This allows time for the OS and control system to configure the necessary drives.

4. Reset GIS interlocks that were triggered during the power cycle.

Warning

When the MTRotator is in the OFFLINE Publish only state in the runRotEui you need to go to the Offline Substate Commands and then send the System Ready command. The DDS can then be enabled from OFFLINE state. Refer to MTRotator Recovery procedure for more details.

Post-Condition

• The PXI controller and its associated drives are correctly rebooted, allowing for successful connection and operation through the CSC/EUI.

• The EtherCat and Copley drives are properly set up and functional.