AuxTel Rotator Out Of Range

Overview

latiss_acquire (Spectroscopic survey) or track_target_and_take_image (Imaging survey) script failed with the error message:

Rejected: Rotator out of range. Target in rotator limit (-170 to 170 degrees) but out of slew limit margin (1 degs).

and ATPtg might goes into FAULT.

Error diagnosis

During the imaging or spectroscopic surveys, latiss_acquire (Spectroscopic survey) or track_target_and_take_image (Imaging survey) causing ATPtg to fault. The full traceback error received is:

Error in run Traceback (most recent call last):
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/lsst/ts/salobj/base_script.py", line 603, in do_run await self._run_task
File "/net/obs-env/auto_base_packages/ts_standardscripts/python/lsst/ts/standardscripts/base_track_target_and_take_image.py", line 237, in run await self.track_target_and_setup_instrument()
File "/net/obs-env/auto_base_packages/ts_standardscripts/python/lsst/ts/standardscripts/auxtel/track_target_and_take_image.py", line 141,  in track_target_and_setup_instrument await self.atcs.slew_icrs(
File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/base_tcs.py", line 647, in slew_icrs raise ack_error
File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/base_tcs.py", line 615, in slew_icrs await self.slew(
File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/base_tcs.py", line 814, in slew raise ack_err
File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/base_tcs.py", line 803, in slew await self._slew_to(
File "/net/obs-env/auto_base_packages/ts_observatory_control/python/lsst/ts/observatory/control/auxtel/atcs.py", line 1413, in _slew_to await slew_cmd.start(timeout=slew_timeout)
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/lsst/ts/salobj/topics/remote_command.py", line 487, in start return await cmd_info.next_ackcmd(timeout=timeout)
File "/opt/lsst/software/stack/conda/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/lsst/ts/salobj/topics/remote_command.py", line 191, in next_ackcmd raise base.AckError(msg="Command failed", ackcmd=ackcmd)
lsst.ts.salobj.base.AckError: msg='Command failed', ackcmd=(ackcmd private_seqNum=1630455441, ack=<SalRetCode.CMD_FAILED: -302>, error=6611,
result='Rejected : Rotator out of range. Target in rotator limit (-170 to 170 degrees) but out of slew limit margin (1 degs)')

Procedure Steps

The interim solution requires recovering the ATPtg back to ENABLED in the case it went to FAULT, and skipping to the next target in the ATQueue by pressing the PLAY button. If this script also fails with the same error, wait three minutes and try again.

  1. Recover ATPtg from FAULT state: Transition the ATPtg CSC from LOVE ASummaryState through the states, FAULT → STANDBY → START → ENABLED.

  2. Add a auxtel/correct_pointing.py script to the top of the queue to recover the pointing offsets from the beginning of the night. No configuration is needed, you can use the default configuration.

  3. Press PLAY button in the ATQueue: This will skip to the next target waiting in the queue.

  4. If this “new” script fails as well, wait three minutes and repeat 1-2 again.

  5. Log in relevant information about this failure in the ticket OBS-52.

Alternative Procedure

In case the previous steps did not work, rotator continues failing and ATPtg never went to FAULT state, there is an alternative solution as follows.

  1. Track a new target placing the standard script auxtel/track_target.py at the top of the queue. Use the following configuration:

    track_target.py
    az: 90.0
el: 60.0
mag_limit: 8.0

  2. Continue adding auxtel/correct_pointing.py as in step 2 of previous procedure with the following configuration:

    correct_pointing.py
    az: 90.0
el: 60.0

  3. Press PLAY button in the ATQueue.

  4. Log in relevant information about this failure in the ticket OBS-52.

Post-Condition

  • ATPtg is back to the ENABLED state.

  • The rotator is within the acceptable range for operations.

  • No further “Rotator out of range” errors occur after executing the procedure.

Contingency

If the procedure was not successful, report the issue in #summit-auxtel and/or activate the Out of hours support.