MTHexapod PXI Controller Reboot (Soft, Hard and Control System Restart)#

Overview#

This document provides detailed instructions on how to perform a soft reboot, hard reboot, and restart the control system for the MTHexapod PXI controllers at the summit. These procedures are essential for troubleshooting faults encountered during routine operations but might also be necessary during maintenance and other non-standard observatory operations, such as scheduled power off.

Error diagnosis#

When troubleshooting the EUI/CSC failed to connect to the control system, follow the sequential order of the procedures below to re-establish control. If a procedure proves unsuccessful, proceed to the next one:

Prerequisites#

  • Ensure you have access to LSST-WAP network.

  • Obtain the necessary IP, account user, and password information from the LSST 1Password MainTel Vault.

  • Familiarity with Linux commands and the use of general Power Distribution Unit (PDU) for power cycling.

Procedure Steps#

Restart Control System#

Warning

As the primary troubleshooting step to regain control, perform the restart of the control system if the CSC/EUI is entirely unresponsive and unable to establish a connection with the control system.

Before proceeding, make sure there are no active interlocks in the EUI and that no other EUI instance is running. For the MTCamHexapod, check if runCamHexEui is running by following commands in the terminal:

ps -aux | grep runCamHexEui

If processes are already running, you may need to identify who is running them and ask permission to end one (or both) so you can run your own EUI session. If another runCamHexEui is running

sudo kill -9 {pid}

Replace runCamHexEui with runM2HexEui for the MTM2Hexapod.

  1. Establish an SSH connection to the MTHexapod PXI: Using the credentials and hostnames found in the LSST 1Password MainTel Vault create an ssh tunnel from the terminal in the LSST-WAP network.

    For the MTCamHexapod, the command would look like:

    ssh admin@camhex-pxi-controller.cp.lsst.org

    For the MTM2Hexapod, the command would look like:

    ssh admin@m2-hexapod-pxi.cp.lsst.org

  2. To restart the control system for the MTCamHexapod, run the command in the terminal:

    /etc/init.d/hexapod restart

    To restart the control system for the MTM2Hexapod, run the command in the terminal:

    /etc/init.d/hexapod restart

    Information

    To check the MTCamHexapod control system status in the terminal:

    /etc/init.d/hexapod status

    It will tell you whether the control system is running or not. To stop it, do:

    /etc/init.d/hexapod stop

    To start it do:

    /etc/init.d/hexapod start

  3. Allow 5-10 minutes for the system to initialize: This period is necessary for the OS and control system to set up the EtherCat and Copley drives before making a connection through the CSC/EUI.

Warning

Only proceed with Soft Reboot if EUI/CSC control is not connecting and the restart of the control system procedure proved unsuccessful to regain control.

Soft Reboot#

  1. Establish an SSH connection to the MTCamHexapod/MTM2Hexapod PXI: Using the credentials and hostnames found in the LSST 1Password MainTel Vault create an ssh tunnel from your terminal in the LSST-WAP network. (See command explicitly described in above).

  2. Execute the reboot command: To initiate a soft reboot of the PXI, type in the terminal:

    sudo reboot

  3. Allow 5-10 minutes for the system to reboot: This time is necessary for the OS and control system to configure the EtherCat and Copley drives before attempting a connection through the CSC/EUI.

Hard Reboot#

Warning

Only proceed with a hard reboot, if the EUI control connection remains unsuccessful after a soft reboot.

This method involves cutting power to the PXI and drives and should only be used as a last resort due to the potential risks.

If a power shutdown is scheduled, you can proceed until step 2 before the power on.

  1. Login into Utilities cabinet Power Distribution Unit (PDU) or MTM2Hexapod PDU: Depending on which hexapod you are rebooting, you need to follow either A or B.

    1. For MTCamHexapod, while in the LSST-WAP, connect to https://tea-pdu01.cp.lsst.org/ using the credentials stored in the Operators vault of LSST 1Password as PDU Utilities Cabinet. Click on Outlets on the left hand side menu to open the outlets screen. The description of each outlet can be found here.

    ../../../../_images/mtrot-controller-pxi-reboot-1.jpeg

    MTCamHexapod https://tea-pdu01.cp.lsst.org PDU outlets.#

    1. For MTM2Hexapod while in the LSST-WAP, connect to https://pdu1-tea-as02.cp.lsst.org using the credentials stored in the MainTel vault of LSST 1Password as pdu1-tea-as02.cp.lsst.org. Click on Outlets on the left hand side menu to open the outlets screen.

    ../../../../_images/MTM2Hex-PDU.png

    MTM2Hexapod https://pdu1-tea-as02.cp.lsst.org PDU outlets.#

  2. Power Cycle PXI and drives: To power Off the system, first power off the PXI, followed by the drive.

    1. For MTCamHexapod, PXI is energized through Outlet 8, while the drives correspond to Outlet 4 in the https://tea-pdu01.cp.lsst.org PDU.

    2. For MTM2Hexapod, power off the PXI which is Outlet 2, then turn off the drives in Outlet 1` in the https://pdu1-tea-as02.cp.lsst.org PDU.

    Note

    Scheduled Power Off

    In case a scheduled power off is intended, do not continue with power on, and stop the procedure here.

    When powering On, power on the cabinet first and wait for 1-3 min to let the EtherCAT slaves finish the setup on the drives. Power on the PXI controller and wait for 5 min to let the EtherCAT master finish the setup. This delay is crucial for the Ethercat application within the PXI to establish a connection with the Copley drive.

  3. Wait for an additional 5-10 minutes after powering on before using the CSC/EUI: This allows time for the OS and control system to configure the necessary drives.

  4. Reset GIS interlocks that were triggered during the power cycle.

Post-Condition#

  • The PXI controller and its associated drives are correctly rebooted, allowing for successful connection and operation through the CSC/EUI.

  • The EtherCat and Copley drives are properly set up and functional.