ATSpectrograph Recovery

Overview

This procedure outlines the steps to recover the ATSpectrograph when it fails during script execution. These issues typically arise after a reset in the ATSpectrograph cRIO, or while enabling the ATSpectrograph CSC. Potential root causes of the failure might include mechanism timeouts, miscommunication, and a range of other contributing factors.

Some relevant notes:

  • LATISS has three moving mechanisms, apart from the Bonn shutter: filter wheel, grating wheel and grating linear stage.

  • In this document the grating linear stage will be also named as linear stage or grating stage.

  • The most frequent position during operations is close to 67 mm. This position is set every time the script auxtel/standard_scripts/daytime_checkout/latiss_checkout.py is run. If by any reason the grating stage is moved to a different position during daytime, please do not forget to run the script afterwards.

You can find more information on the ATSpectrograph in the LSST Auxiliary Telescope Spectrograph AS_BUILT document.

Error diagnosis

There are a few different common error modes which can result in an ATSpectrograph failure. If the ATSpectrograph goes in fault state and cannot be re-enabled, it is likely that you will need to follow this procedure to recover.

Some examples include:

  • You get a timeout error related to the self.latiss.setup_instrument method when running a script like auxtel/standard_scripts/daytime_checkout/latiss_checkout.py or another. Check the error message (similar to the error message shown below).

  • There is an error with the linear stage position while enabling the ATSpectrograph CSC, such as the error message shown below.

Error message after timeout
2023/11/03 17:17:52 TAIError in runTraceback (most recent call last):

File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/tasks.py", line 500, in wait_for return fut.result() ^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",
line 239, in _get_next_ackcmd await self._next_ack_event.wait()

File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/locks.py",

line 213, in wait await fut asyncio.exceptions.CancelledError The above exception was the direct cause of the following
exception: Traceback (most recent call last): File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 189, in next_ackcmd ackcmd = await self._wait_task

^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",

line 214, in _basic_next_ackcmd ackcmd = await asyncio.wait_for
( ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/tasks.py", line 502,
in wait_for raise exceptions.TimeoutError() from exc TimeoutError During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/base_script.py",
line 603, in do_run await self._run_task File "/net/obs-env/auto_base_packages/ts_standardscripts/python/lsst/ts/standardscripts/auxtel/daytime_checkout/latiss_checkout.py", line 110, in run await self.latiss.setup_instrument( File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/latiss.py", line 176, in setup_instrument await self.setup_atspec( File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/latiss.py", line 242, in setup_atspec await asyncio.gather(*setup_coroutines) File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 416, in set_start return await self.start(timeout=timeout, wait_done=wait_done) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 487, in start return await cmd_info.next_ackcmd(timeout=timeout)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",
line 205,
in next_ackcmd raise base.AckTimeoutError( lsst.ts.salobj.base.AckTimeoutError: msg='Timed out waiting for command acknowledgement',
ackcmd=(ackcmd private_seqNum=1137560160, ack=<SalRetCode.CMD_NOACK: -301>, error=0, result='No command acknowledgement seen')
Error message while enabling ATSpectrograph CSC
2025/01/28 17:40:46 TAI
Cannot get information from model for linear stage.
Traceback (most recent call last):
File "/opt/lsst/software/stack/miniconda/lib/python3.11/site-packages/lsst/ts/atspectrograph/atspec_csc.py", line 234, in end_enable
state = await self.model.query_gs_status(self.want_connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/miniconda/lib/python3.11/site-packages/lsst/ts/atspectrograph/model.py", line 306, in query_gs_status
ret_val = await self.run_command("?LSS\r\n", want_connection=want_connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/miniconda/lib/python3.11/site-packages/lsst/ts/atspectrograph/model.py", line 577, in run_command
raise e
File "/opt/lsst/software/stack/miniconda/lib/python3.11/site-packages/lsst/ts/atspectrograph/model.py", line 571, in run_command
read_bytes = await asyncio.wait_for(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/miniconda/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
return fut.result()
^^^^^^^^^^^^
File "/opt/lsst/software/stack/miniconda/lib/python3.11/asyncio/streams.py", line 655, in readuntil
raise exceptions.IncompleteReadError(chunk, None)
asyncio.exceptions.IncompleteReadError: 1 bytes read on a total of undefined expected bytes

Procedure Steps

The procedure will require the user to open the AuxTel EUI, to use telnet commands to directly command the ATSpectrograph cRIO, and to use the script queue.

A healthy status in ATSpectrograph will look as the following screenshot, in which the linear stage, grating and filter wheels are homed/initialized, and the linear stage position is 67mm.

../../../_images/ATSpectrograph-ok.png

ATSpectrograph EUI healthy status.

After a fault in the ATSpectrograh, you may see that one of the status lights on the left-hand side of the EUI is red, such as the FWTimeOutErr light indicating that the filter wheel did not reach it’s commanded position.

To recover the ATSpectrograph we need to clear these faults and ensure the stages are in their correct state and position as shown in the screenshot above:

  1. Open the ATSpectrograph EUI, connecting to AuxTel EUI desktop computer ACE spectrograph (139.229.170.44:8000/Spectrograph.html).

  2. The first thing to try, if the fault is due to a timeout error or one of the status lights on the left-hand side of the EUI is illuminated red, is to click the Re-init Axes button on the EUI. If the fault is cleared and the status lights are returned to normal, the axes are homed/initialized, and the linear stage position is 67mm, you can re-enable the ATSpectrograph CSC and continue observing, skipping the rest of the procedure.

  3. If you were unable to clear the fault with the Re-init axes button, a restart of the cRIO will be required. Before restarting, check the grating stage position in the EUI. To ensure it initializes properly after the reboot, we will need to move the linear stage to its home position before rebooting the cRIO using telnet commands. If the cRIO is rebooted without first homing the stage, it can lose its position and show an incorrect position, often -324mm which is well beyond its operational range.

    ../../../_images/EUI-1reboot.png

    ATSpectrograph EUI with bad linear stage position.

  4. Use telnet commands to move the linear stage to its negative limit:

    1. Check ATSpectrograph CSC is in STANDBY Status.

    2. Open a terminal on your local machine. If the cRIO was recently rebooted, make sure that at least 60 seconds have passed since the cRIO EUI is accesible to give the application time to complete its setup.

    3. Execute the command telnet auxtel-latiss-crio.cp.lsst.org 9999 to connect directly to the cRIO.

    4. Note that the port you are using needs to remain clear in order for the cRIO to connect to the CSC, so the EUI is setup to boot users from this port after 5 seconds if no commands are sent. You may need to reconnect via telnet several times during this process if you get booted.

    5. Execute !LSI. This command will move the linear stage to its negative limit.

    6. Execute !LSL. This command will display the status of the limit switches of the linear stage. It should return a - sign, indicating that the linear stage reached the negative limit and the switch is pressed.

    7. In the EUI, the green indicator for the grating stage negative limit should have been activated.

    8. The position of the linear stage may still read -324mm after this move has been commanded, and that is okay. To recover the position of the linear stage, we now need to reboot the cRIO.

  5. With the linear stage in its negative limit position, power cycle the ATSpectrograph cRIO:

    1. Check ATSpectrograph CSC is in STANDBY Status.

    2. Connect to http://aux-pdu-spectrograph.cp.lsst.org/ (only accessible from the summit).

    3. Log in with the username and password available in the AuxTel 1Password AuxTel vault.

    4. For outlet 2 (power and cRIO) click Off, wait 10 seconds, and then click On.

    ../../../_images/power-cycle-ATSpec.png

    PDU webpage to power On/Off ATSpectrograph.

  6. When the cRIO is rebooted, it might take a few minutes to see the EUI again in the webpage. If the EUI does not come up on its own after 10 minutes, then a second cRIO reboot is necessary.

  7. Once the EUI is accessible, the mechanisms should be homed/initialized, the negative limit green indicator should be active for the three mechanisms, and the linear stage position should be close to 0mm.

    ../../../_images/EUI-2reboot.png

    ATSpectrograph EUI after successful reboot.

  8. At this point, you can re-enable the ATSpectrograph CSC.

  9. Finally, in order ensure everything is working and the linear stage is in the correct position, execute the auxtel/standard_scripts/daytime_checkout/latiss_checkout.py script. If it finishes without error, check again on the EUI that the linear stage position is 67mm.

Post-Condition

Errors should have been cleared from the EUI, and the grating linear stage should have been set to 67mm. All mechanisms are homed, and LATISS is ready for operations.

Contingency

If the procedure was not successful, report the issue in #summit_auxtel and/or activate the Out of hours support.