MTHexapod PXI Controller Reboot (Soft, Hard and Control System Restart)#

Overview#

This document provides detailed instructions on how to perform a soft reboot, hard reboot, and restart the control system for the MTHexapod PXI controllers at the summit. These procedures are essential for troubleshooting faults encountered during routine operations but might also be necessary during maintenance and other non-standard observatory operations, such as scheduled power off.

Error diagnosis#

When troubleshooting the EUI/CSC failed to connect to the control system, follow the sequential order of the procedures below to re-establish control. If a procedure proves unsuccessful, proceed to the next one:

Prerequisites#

  • Ensure you have access to LSST-WAP network.

  • Obtain the necessary IP, account user, and password information from the LSST 1Password MainTel Vault.

  • Familiarity with Linux commands and the use of general Power Distribution Unit (PDU) or netbooter for power cycling.

Procedure Steps#

Restart Control System#

Warning

As the primary troubleshooting step to regain control, perform the restart of the control system if the CSC/EUI is entirely unresponsive and unable to establish a connection with the control system.

  1. Before proceeding, make sure there are no active interlocks in the EUI and that no other EUI instance is running. For the MTCamHexapod, check if runCamHexEui is running by following commands in the terminal:

    ps -aux | grep runCamHexEui

    If processes are already running, you may need to identify who is running them and ask permission to end one (or both) so you can run your own EUI session. If another runCamHexEui is running

    sudo kill -9 {pid}

    Replace runCamHexEui with runM2HexEui for the MTM2Hexapod.

  2. You can do the following to check the MTCamHexapod control system status in the terminal:

    /etc/init.d/cameraHexapod status

    It will tell you whether the control system is running or not. To stop it, do:

    /etc/init.d/cameraHexapod stop

    To start it do:

    /etc/init.d/cameraHexapod start

    For MTM2Hexapod, replace in the command above cameraHexapod with m2Hexapod.

  1. Establish an SSH connection to the MTHexapod PXI: Using the credentials and hostnames found in the LSST 1Password MainTel Vault create an ssh tunnel in the terminal.

    For the MTCamHexapod, the command would look like:

    ssh admin@camhex-pxi-controller.cp.lsst.org

    For the MTM2Hexapod, the command would look like:

    ssh pbalucan@m2-hexapod-pxi.cp.lsst.org

  2. To restart the control system for the MTCamHexapod, run the command in the terminal:

    /etc/init.d/cameraHexapod restart

  3. To restart the control system for the MTM2Hexapod, run the command in the terminal:

    /etc/init.d/m2Hexapod restart

  4. Allow 5-10 minutes for the system to initialize: This period is necessary for the OS and control system to set up the EtherCat and Copley drives before making a connection through the CSC/EUI.

Warning

Only proceed with Soft Reboot if EUI/CSC control is not connecting and the restart of the control system procedure proved unsuccessful to regain control.

Soft Reboot#

  1. Establish an SSH connection to the MTCamHexapod/MTM2Hexapod PXI using the credentials and hostnames found in the LSST 1Password MainTel Vault (See above).

  2. Execute the reboot command: To initiate a soft reboot of the PXI, type in the terminal:

    sudo reboot now

  3. Allow 5-10 minutes for the system to reboot: This time is necessary for the OS and control system to configure the EtherCat and Copley drives before attempting a connection through the CSC/EUI.

Hard Reboot#

Warning

Only proceed with a hard reboot, if the EUI control connection remains unsuccessful after a soft reboot.

This method involves cutting power to the PXI and drives and should only be used as a last resort due to the potential risks.

If a power shutdown is scheduled, you can proceed until step 2 before the power on.

  1. Login into Utilities cabinet Power Distribution Unit (PDU) or MTM2Hexapod netbooter: Depending on the which subsystem you are rebooting, you need to follow either A or B.

    1. For MTCamHexapod, while in the LSST-WAP, connect to https://tea-pdu01.cp.lsst.org/ using the credentials stored in the Operators vault of LSST 1Password as PDU Utilities Cabinet. Click on Outlets on the left hand side menu to open the outlets screen. The description of each outlet can be found here.

    ../../../../_images/mtrot-controller-pxi-reboot-1.jpeg

    1. For MTM2Hexapod the netbooter is required to power cycle the PXI and drives: Connect to the M2 Hex Netbooter m2-hexapod-netbooter.cp.lsst.org using the credential stored in the 1Password MainTel Vault under M2 Hexapod PXI/Drive NetBooter.

  2. Power Cycle PXI and drives: To power Off the system, first power off the PXI, followed by the drive.

    1. For MTCamHexapod, PXI is energized through outlet 8, while the drives correspond to outlet 4 in the PDU.

    2. For MTM2Hexapod, power off PXI first , then the drive from the M2 Hexapod PXI/Drive NetBooter.

    Note

    Scheduled Power Off

    In case a scheduled power off is intended, do not continue with power on, and stop the procedure here.

    When powering On, activate the drives first, and wait for at least 3 minutes before powering on the PXI. This delay is crucial for the Ethercat application within the PXI to establish a connection with the Copley drive.

  3. Wait for an additional 5-10 minutes after powering on before using the CSC/EUI: This allows time for the OS and control system to configure the necessary drives.

  4. Reset GIS interlocks that were triggered during the power cycle.

Post-Condition#

  • The PXI controller and its associated drives are correctly rebooted, allowing for successful connection and operation through the CSC/EUI.

  • The EtherCat and Copley drives are properly set up and functional.