ATDome Lost Communication with the Top-End

Overview

When running a script that commands the dome shutter (e.g., prepare_for/vent , prepare_for/onsky or auxtel/shutdown), the script fails, showing a failure to communicate with the dome top-end where the shutter control resides.

Error diagnosis

When the command to the shutter is run within the script, an error is received in LOVE, showing a communication error:

ack=<SalRetCode.CMD_FAILED: -302>

result="Failed: Command SC returned 1 lines instead of 0; read: ' Top Comm Error\\r

Or the full traceback may be displayed as follows:

 12023/06/29 23:13:03 TAI
 2Error in run
 3Traceback (most recent call last):
 4File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/lsst/ts/salobj/base_script.py", line 603, in do_run
 5await self._run_task
 6File "/net/obs-env/auto_base_packages/ts_standardscripts/python/lsst/ts/standardscripts/auxtel/shutdown.py", line 62, in run
 7await self.attcs.shutdown()
 8File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/atcs.py", line 738, in shutdown
 9raise e
10File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/atcs.py", line 732, in shutdown
11await self.close_dome()
12File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/atcs.py", line 946, in close_dome
13await self.rem.atdome.cmd_closeShutter.set_start(
14File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/lsst/ts/salobj/topics/remote_command.py", line 416, in set_start
15return await self.start(timeout=timeout, wait_done=wait_done)
16File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/lsst/ts/salobj/topics/remote_command.py", line 487, in start
17return await cmd_info.next_ackcmd(timeout=timeout)
18File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/lsst/ts/salobj/topics/remote_command.py", line 191, in next_ackcmd
19raise base.AckError(msg="Command failed", ackcmd=ackcmd)
20lsst.ts.salobj.base.AckError: msg='Command failed', ackcmd=(ackcmd private_seqNum=71336703, ack=<SalRetCode.CMD_FAILED: -302>, error=1,
21result="Failed: Command SC returned 1 lines instead of 0; read: ' Top Comm Error\\r n>'")

Procedure Steps

This is a hardware issue related to a loss in communication between the two cRIOs that are used to control the dome:

  • The Top Box cRIO, which sits on the rotating part of the dome on the second floor.

  • The Main Control Box, which is on the first floor near the ventilation fan.

This issue can occur after an incorrect power sequence or a network communication loss.

To manually reset the dome cRIOs:

  1. Check that the ATDome CSC is in the STANDBY state.

  2. Access the Main and Top Box GUIs in any browser using the following link: http://139.229.170.190/. You must be within Rubin facilities (or have VPN access) to connect to the GUIs.

  3. Head to AuxTel and reset the cRIOs manually using the ATDome Recovery procedures on the AuxTel Recovery after Shutdown page.

Note

Make sure you have a small, pointed object (e.g., a paper clip or pen cap) to press the reset button on the cRIOs.

  1. After the cRIOs are rebooted, confirm there is connectivity with the shutter and dome by verifying that the GUIs are operational (see Fig 1). If the GUIs do not come up on their own after 10 minutes, then a second reboot of the cRIOs is necessary.

  2. Once the GUIs come back online, place the ATDome CSC back into the ENABLED state.

Fig 1: Reset Auxtel Dome and Shutter GUIs
../../../_images/ATDome_gui.png

AuxTel Dome GUI
../../../_images/ATDomeShutter_gui.png

AuxTel Shutter GUI

Post-Condition

  • Communication is back online and scripts that command the dome shutter will work without issue.

Contingency

If the procedure was not successful, report the issue in #summit-auxtel and/or activate the Out of hours support.