AuxTel Known Faults

A comprehensive list of common faults will be displayed here along their troubleshooting procedures.

Known Faults

OCS

Issue

Common in / Error traceback

Troubleshooting procedure

ATCS/ATHexapod

ATHexapod fails to enable - No connection to athexapod controller

Daytime checkout

raise OSError(err, f’Connect call failed {address}’)OSError: [Errno 113] Connect call failed (‘139.229.170.48’, 50000)

ATHexapod fails to enable with the rest of ATCS

ATMCS fails to enable - e-stop is engaged

Daytime checkout

Fault event in ATMCS while in enable state (port=. Nasmyth 1 drive fault bit is ON)

AuxTel Mount Control System Fails to Enable / E-Stop is Engaged

LATISS/ATSpectrograph

ATSpectrograph failed after power down

Daytime checkout

raise base.AckTimeoutError( lsst.ts.salobj.base.AckTimeoutError: msg=’Timed out waiting for command acknowledgement’, ackcmd=(ackcmd private_seqNum=1137560160, ack=<SalRetCode.CMD_NOACK: -301>, error=0, result=’No command acknowledgement seen’)

ATSpectrograph failed - grating stage position and timed out

LATISS/ATCamera

ATCamera Recovering from Fault State

Daytime checkout, Calibrations , Nightime Operations

AT camera recovery

ATCalSys/ATWhiteLight

ATWhiteLight failed to turn on

Calibrations

RuntimeError: White Light Lamp failed to turn on after 900 s.

TO DO

ATCS - M1 Cover

AuxTel M1 Cover Fails to Open.

Daytime checkout, Calibrations, Prepare for on-sky

result=’ERROR: Command OPENM1COVER rejected while M1 covers controller in StandbyState state.’

AuxTel M1 Cover Fails to Open.

Scheduler Driven Observations

Rotator out of range error and ATPtg might go into FAULT state.

Nightime Operations

error=6611, result=’Rejected : Rotator out of range. Target in rotator limit (-170 to 170 degrees) but out of slew limit margin (1 degs)’)

AuxTel Rotator Out Of Range

Lost pointing: Targets do not appear centered in the detector in the first acquisition

Nightime Operations

RuntimeError(“Centroid finding algorithm was unsuccessful.”)

AuxTel Lost Pointing Accuracy

AT out of focus after a ‘WEP’ failure

Nightime Operations

AuxTel Image out of focus

Correct Pointing fails

Nightime Operations

Rejected: elevation out-of-range.

AuxTel Elevation Out of range

ATCS/ATMCS

ATMount fails to move and times out

Nightime Operations

RuntimeError: Telescope timed out getting in position.

AuxTel Mount Fails to Move and Times Out

ATMount fails to move and Azimuth Max Velocity error exceeded

Nightime Operations

Fault event in ATMCS while in tracking enable state. Azimuth drive #2 fault bit is ON. Azimuth drive #1 fault bit is ON. Azimuth max velocity error exceeded.

AuxTel Mount Fails to Move

ATCS/ATDome

ATDome shutter fails to close

Shutdown

AuxTel Emergency Shutdown

Per component