Rubin Observatory Operations Documentation

Important

This documentation area is under heavy development and only in the very early stages of design. It should not be used nor consulted for a place of information at this time.

Observatory Operations consists of multiple levels of interactions and procedures occurring simultaneously in many places and timezones. This section of documentation focuses on the resources needed for observers, commissioning personnel, and support staff to facilitate night-time operations.

The scope of this area of documentation is to provide a centralized, version controlled space where resources are either located, or linked appropriately. The content is provided as reference only. All aspects of the project that go through strict change control (e.g. mirror procedures) should only be linked to these pages.

Safety: Quick Reference for Nighttime Personnel

Warning

Safety of personnel first and equipment second is the utmost priority of Rubin Observatory. The safety information here is meant to serve as a quick look for nighttime support staff in case of an emergency on the summit.

It does not replace the required safety training to be on the summit at night. This includes:

• Procedures on who (and how) to call when there is an issue

• Location of keys, passwords etc

• What to do in case of vehicle failure, power-outage, fire etc.

• Procedures to enter telescope spaces during operations (e.g. AuxTel)

• Emergency meeting locations etc.

• Safety
  • Emergency Response Guide
  • Out of Hours Support
  • Summit Contact Numbers

Daytime Operations

This section encapsulates procedures for normal operations, such as calibrations routines and standard telescope checkout. Any non-standard procedures, such as shutdown recovery and non-standard operations are featured in a different section (link TBD).

• Daytime
  • Introduction to Combined Calibrations Generation Procedure
  • Safety
    • Control Safety Systems
  • Auxiliary Telescope
    • AuxTel Calibrations
      • Enable LATISS
      • Prepare ATCS For Flat Fields Procedure
      • Latiss Combined Calibrations Generation Procedure
    • AuxTel Daytime Checkout
      • Daytime Checkout SAL Scripts
    • AuxTel E-Stop Procedure
      • E-Stop Procedure
    • AuxTel LOTO Procedure
    • AuxTel Daytime Venting
      • Prepare for vent
  • Main Telescope
    • MT Calibrations
      • ComCam Combined Calibrations Generation Procedure
      • Enable ComCam Procedure
      • Enable OCPS ComCam Procedure
    • MT Procedures
      • ComCam Cooldown (CCS-specific)
      • Refrigeration System - Pathfinder

Daytime/Nighttime Interactions

This section describes the procedures for documenting events and work completed during the day and night. Daytime work should be summarized concisely to minimize the time for the night staff to prepare for on sky observations. Night events should be well-documented so day time staff can resolve issues as efficiently as possible.

• Daytime/Nighttime Interactions
  • Announcing Component Usage
  • Daytime and Nighttime Communication
  • Fault Handling Workflow
  • Fault Reporting
  • Nighttime Logging
  • Slack Channel Communication with Summit Personnel

Nighttime Operations

Standard nighttime operations are listed in this section. This includes opening the telescopes for data collection and end-of-night procedures. Special operations or engineering tasks are described in a different section (link TBD).

• Nighttime
  • Weather Constraints
  • Scheduler Procedures
    • Scheduler Operational Scripts
    • Initializing the Scheduler CSC
    • Scheduler Night-time Operation
    • Additional Scheduler Operations
  • Observatory procedures
    • DIMM
      • DIMM Startup, Shutdown and Monitoring
  • Auxiliary Telescope
    • AuxTel Preparation
      • Prepare for On-Sky Operations
    • Auxtel On Sky
      • Beginning of the night: Center, absorb pointing offsets, mirror alignment and focus
    • AuxTel Shutdown
      • Shutdown
  • Main Telescope
    • MainTel Preparation
    • MainTel On Sky
    • MainTel Shutdown

Non-Standard Operations

This includes any non-standard operations often requiring unique protocols or manual interventions, such as shutdown and recovery of the computer room, response to incidents, etc.

• Non-Standard Operations
  • Computer Room Shutdown and Recovery
  • Summit Devices Shutdown and Recovery

Troubleshooting

This section includes information on current known issues that are not addressed in other areas of the documentation. Content includes temporary issues and potential work-arounds. The section is continually updated as the observatory system evolves and develops. If the problem persists after exhausting all documented procedures, ensure safety of the equipment, stop work, and call for help.

• Trobleshooting
  • Troubleshooting Observatory
    • Restoring CSC from OFFLINE state (and no heartbeats)
    • No Historical Data
    • Remote login to a CSC container
  • Troubleshooting Scheduler
    • Troubleshooting the Scheduling Algorithm
  • Troubleshooting AuxTel
  • Troubleshooting MainTel

Project Documentation Information

Describes information regarding this documentation tree, including how to contribute.

• Contribute
  • Contributing to Observatory Operations Documentation
  • Implementation documentation
  • Documentation license
  • Template Folder Index

Observing Interface Setup

Contains detailed instructions on how to set the required environment for operations and/or interaction with the system during testing and commissioning.

• Setup
  • Debugging DDS
  • EFD SAL Kafka
  • Operational Environments
  • Getting Started With Argo CD
  • Getting Started With Chronograf
  • Getting Started With Kubernetes
  • Getting Started With LOVE
  • Getting Started With Nublado
  • Update Notebook Environment in Nublado
  • Setup Common Observing Environment in Nublado
  • Setup DDS Daemon in Nublado
  • Summit and Simulation Environments

System Testing, Deployments, and Upgrades

Content moved to https://obs-controls.lsst.io/.