Troubleshooting Template

Note

This is a procedure template file that is associated with a template directory structure. This note should be deleted when the section is properly populated.

Overview

The AuxTel Calibration System FiberSpectrograph.3 goes to FAULT, claims to have lost connection to USB and it can’t be recovered (cycled to ENABLED) from LOVE. This may occur when running AuxTel daytime calibrations.

Error diagnosis

  • When FiberSpectrograph.3 goes to FAULT, the following error will display within the CSC:

'Failed: Failed to connect to fiber spectrograph.'
Fault! errorCode=1, errorReport="Failed to connect to
fiber spectrograph.: RuntimeError('No attached USB Avantes devices found.')"

Procedure Steps

Important

Try to restart the ATSpectrograph CSC 1-2 times before committing power cycles and reboots. If restarting the CSC does not fix the issue, then continue with the following procedure.

  1. First, you may consider checking the connections in the USB at both ends of the Avantes spectrometer in case any of the cables is loose.

  2. Power cycle the fiber spectrograph with the rocker switch. See Figure 2 in TSTN-032: AuxTel Illumination System Handbook

Note

The power cycle of the Fiber Spectrograph could also be done via Outlet 7 of the Auxtel Illumination System PDU http://auxtel-illpdu.cp.lsst.org/. The password can be located in the 1Password AuxTel Vault.

  1. Power cycle auxtel-ill-control.cp.lsst.org computer via the green and orange power connector (see Figure 7 in AuxTel Recovery Page).

Note

It is also possible reboot by remote ssh login into the auxtel-ill-control.cp.lsst.org using your IPA.

  1. SSH into auxtel-ill-control.cp.lsst.org with your IPA, and issue the following commands:

# Initial shutdown of spectrograph:
>> sudo -iu dco

# Make sure to wait for the previous command it gets through saying,
# "The service 'durability' is now operational."
# Then continue with launching the spectrograph:
>> ./launch_AuxTel_Illumination_containers.sh

# Once lauched, verify correct status with:
>> docker ps -a

Post-Condition

  • Fiber spectrograph can be ENABLED again, and calibrations may continue.

Contingency

If the procedure was not successful, report the issue in #summit-auxtel and/or activate the Out of hours support.