ATSpectrograph failed - grating stage position and timed out

Overview

This procedure outlines the steps to recover the AT Spectrograph when it fails during script execution due to the loss of grating stage position after a reset. This issue typically arises following a reset, caused by UPS failure due to power outage or a hard reset.

Some relevant notes:

  • The grating stage is also known as linear stage.

  • The most frequent grating stage position is close to 67 mm during operations.

  • The grating position ranges between 67 mm and 140 mm from the rotator flange. The hard limit is 65 mm.

You can find more info on the ATSpectrograph in the LSST Auxiliary Telescope Spectrograph AS_BUILT document.

Error diagnosis

  1. You get a timeout after running a script like auxtel/standard_scripts/daytime_checkout/latiss_checkout.py or another. Check the error message (similar to the error message shown below).

  2. There is a difference between the grating stage position shown in the EUI and the previous position (approximately 67 mm). You can find the Grating Stage position (mm) field in the ACE spectrograph tab, shown in the step 2.5 image.

This means the grating stage encoder was not correctly initialized, so the read encoder position is not the actual value, causing the next move command to fail.

This issue first occurred on February 23, 2023. The stage position read 324 mm after a reset, and afterwards a move command to +67 mm failed with a timeout (OBS-272).

Error message
2023/11/03 17:17:52 TAIError in runTraceback (most recent call last):

File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/tasks.py", line 500, in wait_for return fut.result() ^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",
line 239, in _get_next_ackcmd await self._next_ack_event.wait()

File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/locks.py",

line 213, in wait await fut asyncio.exceptions.CancelledError The above exception was the direct cause of the following
exception: Traceback (most recent call last): File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 189, in next_ackcmd ackcmd = await self._wait_task

^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",

line 214, in _basic_next_ackcmd ackcmd = await asyncio.wait_for
( ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/tasks.py", line 502,
in wait_for raise exceptions.TimeoutError() from exc TimeoutError During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/base_script.py",
line 603, in do_run await self._run_task File "/net/obs-env/auto_base_packages/ts_standardscripts/python/lsst/ts/standardscripts/auxtel/daytime_checkout/latiss_checkout.py", line 110, in run await self.latiss.setup_instrument( File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/latiss.py", line 176, in setup_instrument await self.setup_atspec( File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/latiss.py", line 242, in setup_atspec await asyncio.gather(*setup_coroutines) File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 416, in set_start return await self.start(timeout=timeout, wait_done=wait_done) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 487, in start return await cmd_info.next_ackcmd(timeout=timeout)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",
line 205,
in next_ackcmd raise base.AckTimeoutError( lsst.ts.salobj.base.AckTimeoutError: msg='Timed out waiting for command acknowledgement',
ackcmd=(ackcmd private_seqNum=1137560160, ack=<SalRetCode.CMD_NOACK: -301>, error=0, result='No command acknowledgement seen')

Procedure Steps

You should follow these steps to recover the ATSpectrograph:

  1. Power cycle the ATSpectrograph.

    1. Check ATSpectrograph CSC is in STANDBY Status.

    2. Connect to http://aux-pdu-spectrograph.cp.lsst.org/ (only accessible from the summit).

    3. Log in with the username and password available in the AuxTel 1Password vault.

    4. Click On/Off button for Outlet 2 only.

    ../../../_images/power-cycle-ATSpec.png

    Netbooter to power On/Off AuxTel spectrograph.

  2. Connect to AuxTel EUI desktop ACE spectrograph (139.229.170.44:8000/Spectrograph.html).

    ../../../_images/ACE-spec-EUI-Labview.png

    AuxTel Spectrograph EUI

  3. Click Re-init Axes button on the EUI.

Post-Condition

  • The AT Spectrograph’s encoder has been cleared and is now in the home position, near 0 mm.

Contingency

If the procedure was not successful, report the issue in #summit_auxtel and/or activate the Out of hours support.