MTDome Recover and clear Faults

Overview

If issues arise with moving the MTDome directly, or if you observe that the MTDomeTrajectory following mode is ENABLED, but the dome fails to follow the telescope, it is likely that the MTDome is having an internal subsystem fault and requires recovery.

Under specific circumstances, the dome may ignore move commands, such as when its position is identical to the previous moveAz command and the velocity is 0.0 [deg/sec]. To confirm this, inspect the MTDome logs in LOVE. The CSC would have received the duplicated move command, and a warning message will be displayed in the logs, indicating that it was ignored.

In other instances, the dome may reject move commands, when there is a subsystem FAULT.

Error diagnosis

  • If a FAULT is indicated in the Azimuth Control Software Status (ACS status) window, the dome has an internal fault condition.

  • Look at the Chronograf dashboard MTDome Status to verify the system status.

Note

The dome has several subsystems that continue to operate when a specific subsystem goes to FAULT, the CSC itself won’t go to FAULT, unless it is an unrecoverable issue. A subsystem going to FAULT is recoverable.

Procedure Steps

  1. Disable the dome following mode in MTDomeTrajectory.

run_command.py
  component: MTDomeTrajectory
  cmd: setFollowingMode
  parameters:
      enable: false

2. To clear the subsystem FAULT, use the MTDome exitFault configuration command via the run_command.py SAL script. In most cases, running the command once will clear the fault, but you may need to repeat the process a few times.

run_command.py
  component: MTDome
  cmd: exitFault

  1. After executing the command, verify its success by checking the ACS status in the MTDome Status dashboard.

Note

When the exitFault command is sent, the CSC executes the resetDrivesAz command before exitFault command.

Command sent by exitFault:

Configuration for resetDrivesAz

run_command.py
  component: MTDome
  cmd: resetDrivesAz
  paramaters:
   reset: [1,1,1,1,1]

  1. Now, at this point, you can try to move the dome again.

  • Confirm the dome moves by slewing to a nearby position; in the example below, 45 degrees azimuth.

  • Make sure that the dome moves before enabling the MTDomeTrajectory mode.

run_command.py
  component: MTDome
  cmd: moveAz
  parameters:
      position: 45

5. Check the MTDome logs when trying to move it the first time after recovering it. If you see a warning message:

Warning

Ignoring moveAz command for position=300.0 and velocity=0.0 because it is a duplicate command

The error means the CSC is ignoring the move command, regardless of whether the position mentioned in the message is the current position of the Dome. In this case, try to move the dome to a different position.

Post-Condition

MTDome faults are clear and operations can continue.

Contingency

If the above procedure was not successful, report the issue in #summit-simonyi and #rubinobs-mtdome