MTRotator Clear Safety Interlocks¶

Overview¶

This troubleshooting procedures serves as a guide to clear low-level interlocks when recovering the MTRotator CSC from a FAULT.

The procedure may require access to the MTRotator GUI run_rotgui (check access instructions).

Error Diagnosis¶

MTRotator CSC goes into FAULT state, but it cannot recover or cycle through CSC states from LOVE.
When trying to re-enable the MTRotator CSC in ASummary State View, the following message is reported in the ERROR CODE or MESSAGE LOG:
```
Lower level interlock is activated
```

Procedure Steps¶

Note

Most MTRotator low-level interlocks can be cleared through the GIS GUI panel on level 2.

Before manually clearing the low-level interlocks, first try to reset the MTrotator GIS, and then attempt to re-enable the MTRotator CSC.

Transition MTRotator CSC to STANDBY.
1. This can be done in ASummary State View or in a ScriptQueue.
  set_summary_state.py¶
  data: - [MTRotator, STANDBY]
Access the MTRotator GUI.
1. Enter the virtual machine that controls the rotator (hexrot-vm01.cp.lsst.org) with your IPA account credentials.

Accessing the Rotator GUI

Remote Access: The How to Access MT M2/Rotator/Hexapods/Dome EUI has a detailed procedure for accessing all the GUIs in the virtual machine.

Summit Access: If you are logged into a linux machine at the summit, you can enter the virtual machine using an SSH command.

Open a terminal from the ‘Activities’ tab on top left, and type the following command:
```
ssh -Y hexrot-vm01.cp.lsst.org
```

Once in the virtual machine, access the rotator using the run_rotgui command:
```
[(username)@hexrot-vm01 ~]$ run_rotgui
```

../../../../_images/mtrot_recovery_8.png — MTRotator Python GUI¶

Change the Command Source from CSC to GUI mode.
1. Once in the Rotator Control GUI, Connect to the low-level controller (top-left).
2. In the Command section of the GUI, select Switch command source.
3. Under the Command Parameters go to Command Source and select GUI.
4. Execute the command by clicking Send Command at the bottom of the GUI.

Clear Simulink/Interlock errors in MTRotator GUI.
1. In the Main tab, go to the Commands tab.
2. Under Command, select State command.
3. Under Command Parameters, go to State Trigger and select ClearError.
4. Send the command by clicking on the Send Command button.

../../../../_images/mtrot_recovery_7.png — Sending ClearError Command¶

Telemetry Verification

Double-click the Telemetry tab at the bottom of the GUI (see image above).

If the Safety Interlock is activated, you can find a red light next to the

Safety Interlock fault with the Application Status tab.
In case of a Simulink Error, there will be red light next to the Simulink fault on the same page.

../../../../_images/mtrot_recovery_6.png — Rotator Telemetry with Faults (GUI Command Source)¶

When both errors are cleared, the telemetry no longer have red fault lights:

../../../../_images/mtrot_recovery_9.png — Rotator Telemetry without Faults (CSC Command Source)¶

Change the Command Source from GUI to CSC mode and exit the GUI.
1. In the Command section of the GUI, select Switch command source.
2. Under the Command Parameters go to Command Source and select CSC.
3. Execute the command by clicking Send Command at the bottom of the GUI.
4. Remember to Disconnect the low-level controller before you Exit the GUI.

Reset MTRotator in GIS GUI on Level 2.
1. Select the M2Cam tab and then click Overview (default). Below the
  
  CAM. ROTATOR section, press and hold the Reset button. A green “X” mark
  
  should appear next the the Reset label. If it does not show, press and hold the
  
  Reset button, again.

../../../../_images/mtrot_recovery_3.png

IMPORTANT: Interlock Due to CCW Limits

If the interlocks activated due to the rotator being out of range with the CCW, make sure to follow the steps on MTRotator CCW Limit Switches and Pull Cord Alarms before clearing the interlocks on the GIS.

Transition MTRotator CSC back to ENABLED.
set_summary_state.py¶
```
data:
  - [MTRotator, ENABLED]
```

Post-Condition¶

MTRotator is operational and can be re-enabled from LOVE to safely continue operations.

Contingency¶

If the above procedure was not successful, announce in the #summit_simonyi channel.

MTRotator PXI Controller Reboot¶

In case EUI/CSC control is not connecting, you may need to proceed with the control system restart procedure of the MTRotator and MTHexapods PXI controller Reboot (Soft, Hard and Control System Restart).

MTRotator Locking Pin Engaged¶

On rare occasions, the Safety Interlock fault may remain when attempting to clear the low-level interlocks. If the safety interlock does not clear, it might be due to the rotator pin not being disengaged properly. Instructions on removing the rotator locking pin are located in the MTRotator Unlock Locking Pin page on Confluence.