ATHexapod Position Out of Limits

Overview

When running scripts that require ATAOS corrections (e.g., atpneumatics_checkout.py, latiss_wep_align.py, or enable_ataos_corrections.py, ATHexapod may fail to apply corrections due to its position being outside of operational limits.

Error diagnosis

  1. ATHexapod CSC goes into FAULT, and produces the following error in its message log:

    ATHexapod CSC Error Message
    Position out of limits (7)

  2. Should the ATAOS CSC go into FAULT, the following error will display in the current SAL script execution log in ATQueue:

    ATAOS Script Failure Message
        File
"/opt/lsst/software/stack/conda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-
packages/lsst/ts/salobj/topics/remote_command.py", line 191, in next_ackcmd
    raise
base.AckError(msg="Command failed", ackcmd=ackcmd) lsst.ts.salobj.base.AckError: msg='Command
failed', ackcmd=(ackcmd private_seqNum=493863606, ack=<SalRetCode.CMD_FAILED: -302>, error=1,
result="Failed: disableCorrection not allowed in Fault state: errorCode=8103,
errorReport='Correction loop died.'")

    And the ATAOS CSC produces the following error in its message log:

    ATAOS CSC Error Message
    Correction loop died.

Procedure Steps

Case 1: Only ATHexapod Faults

  1. Reset the position of the ATHexapod loading a run_command.py SAL script into ATQueue with the following configuration:

    component: ATHexapod
cmd: moveToPosition
parameters:
  x: 0
  y: 0
  z: 0
  u: 0
  v: 0
  w: 0

  2. Load enable_ataos_corrections.py to re-enable the ATAOS corrections.

    • Run this script only at high elevations \((\geq 70^\circ)\).

  3. Verify ATHexapod is operating properly by running atpneumatics_checkout.py.

Case 2: ATHexapod & ATAOS Fail

  1. Send ATAOS CSC to OFFLINE using the set_summary_state.py script with the following configuration:

    data:
  - [ATAOS, OFFLINE]

  2. Reset the ATAOS CSC on ArgoCD by following the instructions for Restarting a CSC Component in ArgoCD.

  3. Send ATAOS CSC back to ENABLED with another set_summary_state.py script:

    data:
  - [ATAOS, ENABLED]

  4. Load enable_ataos_corrections.py to re-enable the ATAOS corrections.

    • Run this script only at high elevations \((\geq 70^\circ)\).

  5. Verify ATHexapod is operating properly by running atpneumatics_checkout.py.

Post-Condition

  • The ATHexapod and ATAOS CSCs are successfully in the ENABLED state.

  • ATAOS corrections are enabled without causing faults to the CSCs.

  • The ATCS system is fully operational and ready to proceed with observations or further procedures.

Contingency

If the procedure was not successful, report the issue in #summit-auxtel and/or activate the Out of hours support.