AT camera recovery

Overview

This procedure describes the steps to recover the ATCamera from a FAULT state. A fault occurs whenever a parameter, such as temperature, voltage, current, etc., surpasses a specified limit (note that there is typically a warning before a hard error occurs). A fault can also be triggered by an unexpected failure during camera operation.

This document outlines the general recovery procedure, including diagnosing the issue, fixing it, and restoring the camera to ENABLED state.

This article was triggered by OBS-97, when the camera experienced a timeout from REB IN PROGRESS on February 28, 2023. However, these instructions are applicable to a broader range of incidents beyond this specific case.

Note

The instructions below assume:

1. The ability to login to the AuxTel CCS computers.

2. Some familiarity with basic CCS commands/functionality.

Error diagnosis

• ATCamera goes to FAULT state.

LATISS state view, ATCamera in FAULT state

Important

• The CCS commands included in this procedure have not yet been verified to solve this failure.

• All CCS commands have the prompt ccs>

Track down a CCS problem using the ccs-shell command line tool. Note that you can also use the ccs-console graphical interface.

1. Identify which CCS subsystem triggered the problem.

   ats-mcm getRaisedAlertSummary
   

2. Review the raised alerts and/or log files,

   ats-fp getRaisedAlertSummary
   

3. Determine if this was a transitory problem which can be documented (via JIRA ticket) and reset, or something which requires a camera expert to diagnose.

Procedure Steps

1. Clear the raised alerts in both the CCS subsystem which triggered the problem, and the Master Control Module (MCM) which tracks the overall camera state.

   ats-fp clearAllAlerts
   ats-fp getRaisedAlertSummary
   ats-mcm clearAllAlerts
   ats-mcm getRaisedAlertSummary
   

2. Clear the fault in the ocs-bridge,

   ats-ocs-bridge clearFault
   

3. Switch it back to the OFFLINE_AVAILABLE mode.

   ats-ocs-bridge setAvailable
   

4. Transition the camera to full-integrated functionality to the OCS.

   enterControl
   

5. Change the camera state to ENABLED.

Note

• When you switch the camera state to OFFLINE_AVAILABLE from the CCS control console, control is handed over to the OCS. From the OCS, you can proceed to transition the camera to full-integrated functionality or the CCS can regain control. For more details, refer to section 7.1 of the Camera Operations Manual.

• It is important you create a OBS ticket, so we can track how often specific problems occur, and whether software or hardware changes are needed to prevent future occurrences.

Post-Condition

• AT Camera is set to the ENABLED state .

LATISS state view, ATCamera in ENABLED state

Contingency

If the procedure was not successful, report the issue in #summit_auxtel and/or activate the Out of hours support.