AuxTel Known Faults

A comprehensive list of common faults will be displayed here along their troubleshooting procedures.

Known Faults

OCS	Issue	Common in / Error traceback	Troubleshooting procedure
ATCS/ATHexapod	ATHexapod fails to enable - No connection to athexapod controller	Daytime checkout raise OSError(err, f’Connect call failed {address}’)OSError: [Errno 113] Connect call failed (‘139.229.170.48’, 50000)	ATHexapod fails to enable with the rest of ATCS
	ATHexapod cannot issue AOS corrections - position out of limits	Daytime checkout, Nightime Operations raise base.AckError(msg=”Command failed”, ackcmd=ackcmd) lsst.ts.salobj.base.AckError: msg=’Command failed’, ackcmd=(ackcmd private_seqNum=493863606, ack=<SalRetCode.CMD_FAILED: -302>, error=1, result=”Failed: disableCorrection not allowed in Fault state: errorCode=8103, errorReport=’Correction loop died.’”)	ATHexapod Position Out of Limits
	ATMCS fails to enable - e-stop is engaged	Daytime checkout Fault event in ATMCS while in enable state (port=. Nasmyth 1 drive fault bit is ON)	AuxTel Mount Control System Fails to Enable / E-Stop is Engaged
LATISS/ATSpectrograph	ATSpectrograph failed after power down	Daytime checkout raise base.AckTimeoutError( lsst.ts.salobj.base.AckTimeoutError: msg=’Timed out waiting for command acknowledgement’, ackcmd=(ackcmd private_seqNum=1137560160, ack=<SalRetCode.CMD_NOACK: -301>, error=0, result=’No command acknowledgement seen’)	ATSpectrograph failed - grating stage position and timed out
LATISS/ATCamera	ATCamera Recovering from Fault State	Daytime checkout, Calibrations , Nightime Operations	AT camera recovery
ATCalSys/ATWhiteLight	ATWhiteLight failed to turn on	Calibrations RuntimeError: White Light Lamp failed to turn on after 900 s.	TO DO
ATCS - M1 Cover	AuxTel M1 Cover Fails to Open.	Daytime checkout, Calibrations, Prepare for on-sky result=’ERROR: Command OPENM1COVER rejected while M1 covers controller in StandbyState state.’	AuxTel M1 Cover Fails to Open.
Scheduler Driven Observations	Rotator out of range error and ATPtg might go into FAULT state.	Nightime Operations error=6611, result=’Rejected : Rotator out of range. Target in rotator limit (-170 to 170 degrees) but out of slew limit margin (1 degs)’)	AuxTel Rotator Out Of Range
	Lost pointing: Targets do not appear centered in the detector in the first acquisition	Nightime Operations RuntimeError(“Centroid finding algorithm was unsuccessful.”)	AuxTel Lost Pointing Accuracy
	AT out of focus after a ‘WEP’ failure	Nightime Operations	AuxTel Image out of focus
	Correct Pointing fails	Nightime Operations Rejected: elevation out-of-range.	AuxTel Elevation Out of range
ATCS/ATMCS	ATMount fails to move and times out	Nightime Operations RuntimeError: Telescope timed out getting in position.	AuxTel Mount Fails to Move and Times Out
	ATMount fails to move and Azimuth Max Velocity error exceeded	Nightime Operations Fault event in ATMCS while in tracking enable state. Azimuth drive #2 fault bit is ON. Azimuth drive #1 fault bit is ON. Azimuth max velocity error exceeded.	AuxTel Mount Fails to Move
ATCS/ATDome	ATDome shutter fails to close	Shutdown	AuxTel Emergency Shutdown

Per component

• ATCS
  • ATDome Fails to Arrive in Position
  • ATDome Lost Communication with the Top-End
  • ATHexapod Position Out of Limits
  • ATHexapod fails to enable with the rest of ATCS
  • AuxTel Mount Control System Fails to Enable / E-Stop is Engaged
  • AuxTel Mount Fails to Move
  • AuxTel Mount Fails to Move and Times Out
  • AuxTel M1 Cover Fails to Open

• LATISS
  • ATDewar Alarm - LATISS CCD overcooling
  • AT camera recovery
  • ATSpectrograph Recovery

• AT Calibration System
  • Troubleshooting Template
  • ATMonochromator Fault

• General Troubleshooting
  • AuxTel Elevation Out of range
  • AuxTel Image out of focus
  • AuxTel Lost Pointing Accuracy
  • AuxTel Rotator Out Of Range