ATSpectrograph Recovery

Overview

This procedure outlines the steps to recover the ATSpectrograph when it fails during script execution. These issues typically arise after a reset in the ATSpectrograph cRIO, or while enabling the ATSpectrograph CSC. Potential root causes of the failure might include mechanism timeouts, miscommunication, and a range of other contributing factors.

Some relevant notes:

  • LATISS has three moving mechanisms, apart from the Bonn shutter: filter wheel, grating wheel and grating linear stage.

  • In this document the grating linear stage will be also named as linear stage or grating stage.

  • The most frequent position during operations is close to 67 mm. This position is set every time the script auxtel/standard_scripts/daytime_checkout/latiss_checkout.py is run. If by any reason the grating stage is moved to a different position during daytime, please do not forget to run the script afterwards.

You can find more information on the ATSpectrograph in the LSST Auxiliary Telescope Spectrograph AS_BUILT document.

Part A: Linear Stage Recovery

Error Diagnosis

There are a few different common error modes which can result in an ATSpectrograph failure. If the ATSpectrograph goes in fault state and cannot be re-enabled, it is likely that you will need to follow this procedure to recover.

Some examples include:

  • You get a timeout error related to the self.latiss.setup_instrument method when running a script like auxtel/standard_scripts/daytime_checkout/latiss_checkout.py or another. Check the error message (similar to the error message shown below).

  • There is an error with the linear stage position while enabling the ATSpectrograph CSC, such as the error message shown below.

Error message after timeout
2023/11/03 17:17:52 TAIError in runTraceback (most recent call last):

File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/tasks.py", line 500, in wait_for return fut.result() ^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",
line 239, in _get_next_ackcmd await self._next_ack_event.wait()

File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/locks.py",

line 213, in wait await fut asyncio.exceptions.CancelledError The above exception was the direct cause of the following
exception: Traceback (most recent call last): File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 189, in next_ackcmd ackcmd = await self._wait_task

^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",

line 214, in _basic_next_ackcmd ackcmd = await asyncio.wait_for
( ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/asyncio/tasks.py", line 502,
in wait_for raise exceptions.TimeoutError() from exc TimeoutError During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/base_script.py",
line 603, in do_run await self._run_task File "/net/obs-env/auto_base_packages/ts_standardscripts/python/lsst/ts/standardscripts/auxtel/daytime_checkout/latiss_checkout.py", line 110, in run await self.latiss.setup_instrument( File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/latiss.py", line 176, in setup_instrument await self.setup_atspec( File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/latiss.py", line 242, in setup_atspec await asyncio.gather(*setup_coroutines) File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 416, in set_start return await self.start(timeout=timeout, wait_done=wait_done) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py", line 487, in start return await cmd_info.next_ackcmd(timeout=timeout)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/lsst/ts/salobj/topics/remote_command.py",
line 205,
in next_ackcmd raise base.AckTimeoutError( lsst.ts.salobj.base.AckTimeoutError: msg='Timed out waiting for command acknowledgement',
ackcmd=(ackcmd private_seqNum=1137560160, ack=<SalRetCode.CMD_NOACK: -301>, error=0, result='No command acknowledgement seen')
Error message while enabling ATSpectrograph CSC
2025/01/28 17:40:46 TAI
Cannot get information from model for linear stage.
Traceback (most recent call last):
File "/opt/lsst/software/stack/miniconda/lib/python3.11/site-packages/lsst/ts/atspectrograph/atspec_csc.py", line 234, in end_enable
state = await self.model.query_gs_status(self.want_connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/miniconda/lib/python3.11/site-packages/lsst/ts/atspectrograph/model.py", line 306, in query_gs_status
ret_val = await self.run_command("?LSS\r\n", want_connection=want_connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/miniconda/lib/python3.11/site-packages/lsst/ts/atspectrograph/model.py", line 577, in run_command
raise e
File "/opt/lsst/software/stack/miniconda/lib/python3.11/site-packages/lsst/ts/atspectrograph/model.py", line 571, in run_command
read_bytes = await asyncio.wait_for(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/lsst/software/stack/miniconda/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
return fut.result()
^^^^^^^^^^^^
File "/opt/lsst/software/stack/miniconda/lib/python3.11/asyncio/streams.py", line 655, in readuntil
raise exceptions.IncompleteReadError(chunk, None)
asyncio.exceptions.IncompleteReadError: 1 bytes read on a total of undefined expected bytes

Procedure Steps

Warning

The procedure will require the user to open the AuxTel EUI, to use telnet commands to directly command the ATSpectrograph cRIO, and to use the script queue. Exercise these steps with extreme caution.

A healthy status in ATSpectrograph will look as the following screenshot, in which the linear stage, grating and filter wheels are homed/initialized, and the linear stage position is 67mm.

../../../_images/ATSpectrograph-ok.png

ATSpectrograph EUI healthy status

After a fault in the ATSpectrograh, you may see that one of the status lights on the left-hand side of the EUI is red, such as the FWTimeOutErr light indicating that the filter wheel did not reach it’s commanded position.

To recover the ATSpectrograph we need to clear these faults and ensure the stages are in their correct state and position as shown in the screenshot above:

  1. Open the ATSpectrograph EUI, connecting to AuxTel EUI desktop computer ACE spectrograph (139.229.170.44:8000/Spectrograph.html).

  2. The first thing to try, if the fault is due to a timeout error or one of the status lights on the left-hand side of the EUI is illuminated red, is to click the Re-init Axes button on the EUI. If the fault is cleared and the status lights are returned to normal, the axes are homed/initialized, and the linear stage position is 67mm, you can re-enable the ATSpectrograph CSC and continue observing, skipping the rest of the procedure.

  3. If you were unable to clear the fault with the Re-init axes button, a restart of the cRIO will be required. Before restarting, check the grating stage position in the EUI. To ensure it initializes properly after the reboot, we will need to move the linear stage to its home position before rebooting the cRIO using telnet commands. If the cRIO is rebooted without first homing the stage, it can lose its position and show an incorrect position, often -324mm which is well beyond its operational range.

    ../../../_images/EUI-1reboot.png

    ATSpectrograph EUI with bad linear stage position.

  1. Use telnet commands to move the linear stage to its negative limit:

    1. Check ATSpectrograph CSC is in STANDBY Status.

    2. Open a terminal on your local machine. If the cRIO was recently rebooted, make sure that at least 60 seconds have passed since the cRIO EUI is accesible to give the application time to complete its setup.

    3. Execute the command telnet auxtel-latiss-crio.cp.lsst.org 9999 to connect directly to the cRIO.

    4. Note that the port you are using needs to remain clear in order for the cRIO to connect to the CSC, so the EUI is setup to boot users from this port after 5 seconds if no commands are sent. You may need to reconnect via telnet several times during this process if you get booted.

    5. Execute !LSI. This command will move the linear stage to its negative limit.

    6. Execute !LSL. This command will display the status of the limit switches of the linear stage. It should return a - sign, indicating that the linear stage reached the negative limit and the switch is pressed.

    7. In the EUI, the green indicator for the grating stage negative limit should have been activated.

    8. The position of the linear stage may still read -324mm after this move has been commanded, and that is okay. To recover the position of the linear stage, we now need to reboot the cRIO.

  2. With the linear stage in its negative limit position, power cycle the ATSpectrograph cRIO:

    1. Check ATSpectrograph CSC is in STANDBY Status.

    2. Connect to http://aux-pdu-spectrograph.cp.lsst.org/ (only accessible from the summit).

    3. Log in with the username and password available in the AuxTel 1Password AuxTel vault.

    4. For outlet 2 (power and cRIO) click Off, wait 10 seconds, and then click On.

    ../../../_images/power-cycle-ATSpec.png

    PDU webpage to power On/Off ATSpectrograph.

  3. When the cRIO is rebooted, it might take a few minutes to see the EUI again in the webpage. If the EUI does not come up on its own after 10 minutes, then a second cRIO reboot is necessary.

  4. Once the EUI is accessible, the mechanisms should be homed/initialized, the negative limit green indicator should be active for the three mechanisms, and the linear stage position should be close to 0mm.

    ../../../_images/EUI-2reboot.png

    ATSpectrograph EUI after successful reboot.

  5. At this point, you can re-enable the ATSpectrograph CSC.

  6. Finally, in order ensure everything is working and the linear stage is in the correct position, execute the auxtel/daytime_checkout/latiss_checkout.py script. If it finishes without error, check again on the EUI that the linear stage position is 67mm.

Part B: Re-Initialize Filter and Grating Wheels

Error Diagnosis

Re-initializing the filter and grating wheels is a procedure that is sometimes necessary when recovering the ATSpectrograph system, particularly after the motor control sequence has been interrupted or lost its positional reference. It is crucial to stop and perform a thorough diagnosis to determine the underlying cause before proceeding. The first step is always to diagnose whether a re-initialization is truly required.

If an initialization is required, the ATSpectrograph will not display FilterHome and/or GratingHome as enabled (green), and instead will produce a red FWNotInitErr and/or GRNotInitErr (as seen in the ATSpectrograph EUI above).

Do Not Attempt Multiple Initializations!

It is not recommended to repeat the initialization process multiple times if the first attempt fails. The root cause of the failure may be due to hardware issues, and repeating the indexing procedure can potentially worsen the problem.
Therefore, if an initial attempt to re-initialize is unsuccessful, submit a ticket right away and alert the day crew.

Procedure Steps

Warning

The procedure will require the user to open the AuxTel EUI, to use telnet commands to directly command the ATSpectrograph cRIO, and to use the script queue. Exercise these steps with extreme caution.

  1. Open the ATSpectrograph EUI, connecting to AuxTel EUI desktop computer ACE spectrograph (139.229.170.44:8000/Spectrograph.html).

  2. The first thing to try, if the fault is due to a timeout error or one of the status lights on the left-hand side of the EUI is illuminated red, is to click the Re-init Axes button on the EUI. If the fault is cleared and the status lights are returned to normal, the axes are homed/initialized, and the linear stage position is 67mm, you can re-enable the ATSpectrograph CSC and continue observing, skipping the rest of the procedure.

    Note

    If the Re-init Axes is not working, continue with the rest of this procedure.

  3. Use telnet commands to move the linear stage to its negative limit:

    1. Check ATSpectrograph CSC is in STANDBY Status.

    2. Open a terminal on your local machine. If the cRIO was recently rebooted, make sure that at least 60 seconds have passed since the cRIO EUI is accesible to give the application time to complete its setup.

    3. Execute the command telnet auxtel-latiss-crio.cp.lsst.org 9999 to connect directly to the cRIO.

    4. Note that the port you are using needs to remain clear in order for the cRIO to connect to the CSC, so the EUI is setup to boot users from this port after 5 seconds if no commands are sent. You may need to reconnect via telnet several times during this process if you get booted.

    5. Execute the commands ?FWS and ?GRS. These commands will display the current filter wheel and grating wheel statuses, respectively.

      • Each command will return an array of three characters: X # Y:

      X: Current State

      #: Current Position

      Y: Error Code
      I - Initializing/Homing
      M - Moving
      S - Stationary/Not Moving
      X - Not at Filter Position
      0
      1
      2
      3
      N - None
      B - Busy
      I - Not Initialized
      T - Move Timed Out

    6. If either of the error codes report Not Initialized, then the filter and/or grating wheel needs to be initialized:

      • Initialize Filter Wheel \(\rightarrow\) !FWS

      • Initialize Grating Wheel \(\rightarrow\) !GRS

    7. Check the statuses again to verify that the filter and gratings wheels are initialized.

      ../../../_images/ATSpectrograph_FilterGratingStatus.png

      Filter and Grating Status - Initialized

  4. At this point, you can re-enable the ATSpectrograph CSC.

  5. Finally, in order ensure everything is working and the linear stage is in the correct position, execute the auxtel/daytime_checkout/latiss_checkout.py script. If it finishes without error, check again on the EUI that the Filter and Grating Wheels are homed and there are no errors.

Post-Condition

  • Errors should have been cleared from the EUI.

  • The grating linear stage should have been set to 67mm.

  • Both the filter and grating wheels are initialized.

  • All mechanisms are homed, and LATISS is ready for operations.

Contingency

If the procedure was not successful, report the issue in #summit_auxtel and/or activate the Out of hours support.