MTRotator Recovery

Overview

In the event of a fault occurrence within the MTRotator, following the proper recovery procedure is essential to maintaining operational safety. This document will outline the necessary step-by-step actions to return to normal MTRotator functioning.

Error diagnosis

• The difference between the MTRotator and the Camera Cable Wrap (CCW) is too large. Bulkhead limit switch activated; this is also called Pull cord + or Pull cord -.

  

  TMA EUI screenshot indicating the pull cord +/- alert highlighted by the red box

• Safety Interlock active in the MTRotator GUI

• Simulink fault in the MTRotator GUI

• CCW Interlock D-8 activated on the GIS (link here to the GIS interlock screenshot)

• MTRotator CSC goes into FAULT, but can not be recovered/cycled through CSC from LOVE.

• WD (watchdog) box in safety system of TMA EUI is red and GIS TMA IS is triggered.

  

  

Procedure Steps

Warning

Only proceed after an accurate assessment of the MTRotator fault is done and details have been documented in a ticket.

1. Transition MTRotator CSC to STANDBY status.

2. Clear the error from TMA EUI first.

3. Access the MTRotator EUI/GUI:

   1. Enter hexrot-vm02.cp.lsst.org with your IPA account credentials.

   2. Once in the virtual machine, choose your user profile and enter your IPA password.

   3. open a terminal from the ‘Activities’ tab on top left and go to the following:

      cd /rubin/
      cd rotator/
      cd build
      ./runRotEui
      

4. Change from DDS Command Source to GUI mode:

   Click the Parameters tab in the MTRotator EUI, select GUI under Command Source, and press Set Command Source. In case the GUI control is not possible consult the Contingency section below for further guidance.

   

5. Clear Simulink error in MTRotator GUI:

   1. In the MTRotator EUI Main tab, select State Cmd under Commands to Send. In State Triggers, select ClearError and click on the Send Command button. The Simulink Error light should be cleared now.

   

   2. When the Safety Interlock fault is activated.

   

   3. When the Safety Interlock fault is deactivated.

   

6. Reset MTRotator in GIS GUI at Level 2:

   1. Press Bypass by the D-8 (CCW Safety Device Actuated).

   

   2. Click M2Cam and then Overview (Default). Note that you should see a x mark on the square of Reset. If not, click the Reset button again.

   

7. Back to MTRotator GUI, clearError command to reset Safety Interlock:

   Following a similar process to Step 4.a, the ClearError command will remove the safety interlock.

8. Enable the MTRotator, then move it to zero degrees:

   To enable, State Cmd is selected, StateTriggers menu shows Enable under and then click Send Command button. To move the MTRotator, go to the Commands to Send section and in Enabled Substate Triggers, choose Move. Then, input 0 degrees in the Position Cmd field and execute the movement by clicking on the Send Command button.

9. Reset alarms in TMA GUI:

   1. In the Safety System menu, reset the Pull Cord + or Pull Cord - alarm.

   2. Exit the Safety System`and enter the :guilabel:`Camera Cable Wrap tab. Click on Reset alarm.

   3. In the Camera Cable Wrap tab, press the ON button. Everything should be shown as green now.

10. Release the bypass to the CCW in GIS GUI in Level 2 (Refer to Step 5.a)

11. Revert Command Source from EUI to DDS (opposite to the Step 3)

Post-Condition

• MTRotator is operational and can be re-enabled from LOVE to safely continue operations.

Contingency

If the above procedure was not successful, inform in the #summit-simonyi channel.

In case EUI/CSC control is not connecting, you could proceed with the control system restart procedure of the MTRotator and MTHexapods PXI controller Reboot (Soft, Hard and Control System Restart).