ATSpectrograph failed - grating stage position and timed out#

Overview#

ATspectrograph failed during latiss_checkout after the power cut off due to the UPS failure issue at AuxTel on November 2023. We have seen this error after hard resets of the ATSpectrograph.

Error diagnosis#

  1. Check the error message from latiss_checkout.py or another script.

  2. It appears that the encoder position is lost on a reset, as the linear stage reporte being at position -324mm but it had not moved since the reset, suggesting the encoder value position was incorrectly initialized. When the ATSpectrograph is first initialized after a power reset and a move is commanded (such as during the latiss_checkout when it tried to position the stage at +67mm), since the encoder position is incorrect the stage cannot arrive in position and times out during the move.

Error message#
2023/11/03 17:17:52 TAIError in runTraceback (most recent call last):

File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/tasks.py", line 500, in wait_for return fut.result() ^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",
line 239, in _get_next_ackcmd await self._next_ack_event.wait()

File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/locks.py",

line 213, in wait await fut asyncio.exceptions.CancelledError The above exception was the direct cause of the following
exception: Traceback (most recent call last): File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 189, in next_ackcmd ackcmd = await self._wait_task

^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",

line 214, in _basic_next_ackcmd ackcmd = await asyncio.wait_for
( ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/tasks.py", line 502,
in wait_for raise exceptions.TimeoutError() from exc TimeoutError During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/base_script.py",
line 603, in do_run await self._run_task File "/net/obs-env/auto_base_packages/ts_standardscripts/python/lsst/ts/standardscripts/auxtel/daytime_checkout/latiss_checkout.py", line 110, in run await self.latiss.setup_instrument( File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/latiss.py", line 176, in setup_instrument await self.setup_atspec( File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/latiss.py", line 242, in setup_atspec await asyncio.gather(*setup_coroutines) File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 416, in set_start return await self.start(timeout=timeout, wait_done=wait_done) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 487, in start return await cmd_info.next_ackcmd(timeout=timeout)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",
line 205,
in next_ackcmd raise base.AckTimeoutError( lsst.ts.salobj.base.AckTimeoutError: msg='Timed out waiting for command acknowledgement',
ackcmd=(ackcmd private_seqNum=1137560160, ack=<SalRetCode.CMD_NOACK: -301>, error=0, result='No command acknowledgement seen')

Procedure Steps#

  1. Power cycle the ATSpectrograph

    1. Check ATSpectrograph CSC is in Standby Status.

    2. Connect to http://aux-pdu-spectrograph.cp.lsst.org/

    3. Log in with Username/PW (it is in the AuxTel 1Password vault).

    4. Click On/Off button for Outlet 2 only.

    ../../../_images/1_power_cycle_ATSpec.png
  2. Connect to AuxTel EUI desktop

    1. Install Microsoft Remote Desktop

    2. Setup to connect to AuxTel EUI desktop.

      1. Click ‘+’ button on the top menu.

      2. Select “Add PC” from drop-down menu.

      3. Click the drop-down menu “Gateway” > “Add Gateway”

        ../../../_images/2_connect_auxtel_EUI_part1.png

      4. Put aux-brick01.cp.lsst.org on the Gateway name

        ../../../_images/3_connect_auxtel_EUI_part2.png

    3. Login with Username/PW of ATMCS/ATSpectrograph/ATDome EUI access on 1Password Vault.

    4. Open the new tab/window of the web browser on the remote desktop.

    5. Connect to 139.229.170.44:8000/Spectrograph.html

      ../../../_images/4_ACE_spec_EUI_Labview.png

    6. Click “Re-init Axes” Button on the EUI

Post-Condition#

  • The AT Spectrograph’s encoder has been cleared and is now in the home position, near 0 nm.

Contingency#

If the procedure was not successful, report the issue in #summit_auxtel and/or activate the Out of hours support.