Summit

This section contains site specific variations for the summit.

Resources

(1) Need to get kubeconfig file from here. File a Jira ticket with Chilean IT for access. Once able to log into Rancher:

  1. Select the andes cluster.
  2. Click the Kubeconfig File button in top-right.
  3. Near bottom of dialog, click the download link.

Non-Production Systems

The non-production domain systems list is kept here: Summit deployment.

Bare Metal Machines

  • Main OSPL Daemon: azar1.cp.lsst.org
  • T&S CSCs: azar2.cp.lsst.org
  • LOVE: amor01.cp.lsst.org
  • LOVE2: amor02.cp.lsst.org
  • Kubernetes: Can be done from own machine, just need kubeconfig file and kubectl installed.
  • ATCamera (Tony Johnson): atsccs1.cp.lsst.org
  • CCCamera(Tony Johnson): comcam-mcm.cp.lsst.org
  • ATArchiver (Steve Pietrowicz): atarchiver.cp.lsst.org
  • CCArchiver (Steve Pietrowicz): comcam-arctl01.cp.lsst.org
  • M1M3 Dev (Petr Kubánek): m1m3-dev.cp.lsst.org
  • M1M3 Test (Petr Kubánek): m1m3-test.cp.lsst.org
  • M1M3 Support cRIO (Petr Kubánek): 139.229.178.182
  • M1M3 VMS cRIO (Petr Kubánek): 139.229.178.183
  • M2 VMS cRIO (Petr Kubánek): 139.229.178.193
  • ATMCS/ATPneumatics cRIO (Tiago Ribeiro): 139.229.170.47
  • Hexapod/Rotator (Tiago Ribeiro): hexrot.cp.lsst.org
  • AT PMD (Eric Coughlin): at-keener.cp.lsst.org
  • M2 Control (Te-Wei Tsai): m2-control.cp.lsst.org

LOVE Summary View

The overall system summary state view is called ASummary State View.

Shutdown DM and Camera Services

  • Shutdown/Cleanup daemon on Archiver machines:
    • docker stop ospl-daemon
    • docker rm ospl-daemon
  • Shutdown Camera OCS Bridges:
    • ATCamera: sudo systemctl stop ocs-bridge-atcamera.service
    • CCCamera: sudo systemctl stop ocs-bridge-comcam.service
  • Shutdown Camera Daemons
    • sudo systemctl stop opensplice.service
    • Command is the same everywhere.

Shutdown LOVE

This needs to be done from amor01.

  • Uses the docker-compose-admin scripts in summit/amor01 directory.
    • ./shutdown_love
    • ./shutdown_daemon

If LOVE2 is operating, go to amor02.

  • Uses the docker-compose-admin scripts in summit/amor02 directory.
    • ./shutdown_love
    • ./shutdown_daemon

Shutdown T&S Bare Metal Services

Handle azar2:

  • Uses the docker-compose-admin scripts in summit/azar2 directory.
    • ./shutdown_eas
    • ./shutdown_daemon

Handle AT systems (ATMCS and ATPneumatics):

  • ssh admin@139.229.170.47
  • vim /usr/ts_ddsconfig/config/ospl-shmem.xml
  • Line 4: replace 0 with 2 in the <Id> tag.
  • reboot && exit

Handle M1M3 cRIO:

  • ssh admin@139.229.178.182
  • /etc/init.d/ts-M1M3support stop

Handle M1M3 VMS cRIO:

  • ssh admin@139.229.178.183
  • /etc/init.d/ts-VMS stop

Handle M2 VMS cRIO:

  • ssh admin@139.229.178.193
  • /etc/init.d/ts-VMS stop

Handle M1M3 Dev & Test:

  • ssh to those machines.
  • ps wuax | grep splice
  • sudo kill <PID> on any processes turned up by the previous command.

Handle Hexpod/Rotator (hexrot):

  • Uses the docker-compose-admin scripts in summit/hexrot directory.
    • ./shutdown_hexrot
    • ./shutdown_daemon

AT PMD (at-keener):

  • Uses docker-compose-ops, so should be similar to hexrot (just doesn’t have docker-compose-admin scripts).

M2 Control:

  • ssh to that machine.
  • ps wuax | grep splice
  • sudo kill <PID> on any processes turned up by the previous command.

Interacting with Kubernetes

Commands can be executed from your own machine with kubectl and the proper kubeconfig file.

Shutdown Main Daemon

This needs to be done from azar1.

  • Uses the docker-compose-admin scripts in summit/azar1 directory.
    • ./shutdown_daemon

Update Configuration

  • Gather the branch for the configurations and version number for ts_ddsconfig.
  • Uses the docker-compose-admin scripts in summit directory.
  • Directories to update:
    • /deploy-lsstts/docker-compose-ops (azar1, azar2, amor01, amor02)
    • /deploy-lsstts/ts_ddsconfig (azar1, azar2, amor01, amor02)
    • /deploy-lsstts/LOVE-integration-tools (amor01, amor02)
    • sudo ./update_repo <repo path> <branch or version>
  • This will fail if the branch has local modifications. At that point you may as well just do the job manually. Here is one way to do that:
    • cd /deploy-lsstts/<problem directory>
    • git status
    • sudo git reset –hard origin/<current ticket branch>
    • Return to the docker-compose-admin scripts and run the update_repo command again.

Startup Main Daemon

This needs to be done from azar1.

  • Uses the docker-compose-admin scripts in summit/azar1 directory.
    • ./launch_daemon

Startup LOVE

This needs to be done from amor01.

  • Uses the docker-compose-admin scripts in summit/amor01 directory.
    • ./launch_daemon
    • Ensure daemon is ready before proceeding.
    • ./launch_love

If LOVE2 is operating, go to amor02.

  • Uses the docker-compose-admin scripts in summit/amor02 directory.
    • ./launch_daemon
    • Ensure daemon is ready before proceeding.
    • ./launch_love

Startup T&S Bare Metal Services

Handle azar2:

  • Uses the docker-compose-admin scripts in summit/azar2 directory.
    • ./launch_daemon
    • Ensure daemon is ready before proceeding.
    • ./launch_eas

Handle Hexapod/Rotator (hexrot):

  • Uses the docker-compose-admin scripts in summit/hexrot directory.
    • ./launch_daemon
    • Ensure daemon is ready before proceeding
    • ./launch_hexrot

Enabled CSCs

There are a few CSCs that must be put into ENABLED state before declaring an end to the deployment. These are:

  • Watcher
  • ScriptQueue:1
  • ScriptQueue:2
  • WeatherStation:1

The ScriptQueues can be started in the LOVE Queue view. Click the cog wheel next to the Summary State display to chose Disable. Click the cog wheel next to the Summary State display to chose Enable.

The Watcher and WeatherStation:1 can be started by using the set_summary_state..py script once the ScriptQueues are ENABLED. Both systems require specific configuration settings for optimal operation. They are: * Watcher - summit * WeatherStation:1 - default