Power Cycle MTM2 cabinet

Overview

This procedure is the final option for resolving an MTM2 issue during the night. It’s utilized when the M2 GUI consistently displays alarms, and the MTM2 cannot transition to a closed-loop status following the steps outlined in the M2 recovery procedure.

This document mirrors the Power-Cycle section under M2 handling errors, albeit presented in the format of Observatory Operations Documentation.

Precondition

Note

Ensure you have access to 1Password and are a member of the Operators vault. If not, request access for the 1password Vault Operators via an IT Jira ticket.

  • Consult with the hardware or system engineers to obtain approval before proceeding with this procedure.

  • Ensure you are connected to the control room network as VPN cannot reach it.

  • You should only do this procedure if:

    • Following M2 recovery procedure doesn’t fix the issue, this means re-starting the control system fails to clear alarms and you are unable to set MTM2 to closed-loop.

    • You cannot ping or ssh the M2 controller m2-crio-controller01.cp.lsst.org (on TMA) m2-crio-controller02.cp.lsst.org (on level 3).

    • You see errors codes 6051 and 6088, or 6051 and 6052, and in some cases error 6056.

  • Know the location of the MTM2 cabinet according its installation, which is either in TMA or on level 3.

Post-Condition

  • MTM2 GUI displays no alarms, and it can transition to a closed-loop status.

Procedure Steps

Depending on the location of MTM2 the corresponding steps are:

If MTM2 is on the TMA

  1. Access https://tea-pdu01.cp.lsst.org/, the internet interface of the general power distribution (PDU).

    ../../../../_images/pdu-m2.png

    General Power Distribution outlets

  2. Use the credentials for PDU Utilities Cabinet found in the 1Password Operators Vault.

  3. Select Outlets in the left panel of the window.

  4. Select and power Off M2 Cabinet.

  5. Wait 30 seconds.

  6. Turn the power On the M2 Cabinet.

If MTM2 is on level 3

  1. Go to level 3.

  2. Locate the power inverter

  3. Turn off the switch

  4. Wait 30 seconds.

  5. Turn the switch back on.

    ../../../../_images/MTM2-power-inverter-switch.png

    MTM2 power inverter at level 3

Final steps:

Connect the MTM2 control system to MTM2 CSC. Bare in mind that before requesting a connection you must wait for 5 minutes.

Troubleshooting

You can verify if the control system is running and waiting for the TCS/IP connection, by referring to the Get information block in the M2 recovery page.

In the event that connecting fails try re-starting the M2 EUI.

Note about TCP/IP connection:

If, even after performing a power-cycle, you’re still unable to ping the controller, you must contact IT support.