AuxTel Known Faults#

A comprehensive list of common faults will be displayed here along their troubleshooting procedures.

Known Faults#

OCS

Issue

Common in / Error traceback

Troubleshooting procedure

ATCS

ATHexapod fails to enable - No connection to athexapod controller

Daytime checkout

raise OSError(err, f’Connect call failed {address}’)OSError: [Errno 113] Connect call failed (‘139.229.170.48’, 50000)

ATHexapod fails to enable with the rest of ATCS

ATMCS fails to enable - e-stop is engaged

Daytime checkout

Fault event in ATMCS while in enable state (port=. Nasmyth 1 drive fault bit is ON)

AuxTel Mount Control System Fails to Enable / E-Stop is Engaged

ATSpectrograph

ATSpectrograph failed after power down

Daytime checkout

raise base.AckTimeoutError( lsst.ts.salobj.base.AckTimeoutError: msg=’Timed out waiting for command acknowledgement’, ackcmd=(ackcmd private_seqNum=1137560160, ack=<SalRetCode.CMD_NOACK: -301>, error=0, result=’No command acknowledgement seen’)

ATSpectrograph failed - grating stage position and timed out

ATWhiteLight

ATWhiteLight failed to turn on

Calibrations

RuntimeError: White Light Lamp failed to turn on after 900 s.

TO DO

ATCS - M1 Cover

AuxTel M1 Cover Fails to Open.

Daytime checkout, Calibrations Prepare for on-sky

result=’ERROR: Command OPENM1COVER rejected while M1 covers controller in StandbyState state.’

AuxTel M1 Cover Fails to Open. Explanation video

Scheduler Driven Observations

Rotator out of range error and ATPtg might go into FAULT state.

Nightime Operations

error=6611, result=’Rejected : Rotator out of range. Target in rotator limit (-170 to 170 degrees) but out of slew limit margin (1 degs)’)

AuxTel Rotator Out Of Range

Lost pointing: Targets do not appear centered in the detector in the first acquisition

Nightime Operations

RuntimeError(“Centroid finding algorithm was unsuccessful.”)

AuxTel Lost Pointing Accuracy

AT out of focus after a ‘WEP’ failure

Nightime Operations

AuxTel Image out of focus

Correct Pointing fails

Nightime Operations

Rejected: elevation out-of-range.

AuxTel Elevation Out of range

AT Mount

ATMount fails to move and times out

Nightime Operations

RuntimeError: Telescope timed out getting in position.

AuxTel Mount Fails to Move and Times Out

ATMount fails to move and Azimuth Max Velocity error exceeded

Nightime Operations

Fault event in ATMCS while in tracking enable state. Azimuth drive #2 fault bit is ON. Azimuth drive #1 fault bit is ON. Azimuth max velocity error exceeded.

AuxTel Mount Fails to Move

AT Dome

ATDome shutter fails to close

Shutdown

AuxTel Emergency Shutdown

LATISS

ATCamera Recovering from Fault State

Daytime checkout, Calibrations , Nightime Operations

AT camera recovery

Per component#