On call primer

Welcome!

First: Thank you.

covidtest.usa.gov was an urgent call with accelerated timelines. Much of the focus has been on development and delivery in time of launch; so we acknowledge that there is work to be done to support on call staff. This is definitely "building the plane as we fly it" situations.

Onboarding checklist

The following is an issue template that can be created for each new member of the on call team. When onboarding a new teammate, create an issue with the on call template.

Tasks for existing team member

• Review the rest of the checklist below before starting to work through it. Any step that does not seem relevant or necessary for the kind of work that the person onboarding will be doing should have a - in the checkbox and ~ around the line so it's clear that we will not be granting that particular kind of access for this person.
• Add team member to TTS Slack
  • #test-website (main)
  • #covidtest-app (shared with USDS)
  • #tts-covidtest-devops (platform & devops)
  • #tts-covidtest-rails (app development)
  • #tts-covidtest-rails-alerts (development-related alerts, like Github)
  • #tts-covidtest-production-alerts (prod-related alerts, like New Relic)
  • #tts-covidtest-situation (active incident resolution)
• Add team member to Google Drive
• Add team member to Service Now
• Add team member to New Relic
• [If cloud.gov access needed] Invite team member to all CF foundations
  • fr.wb.cloud.gov
  • fr.wc.cloud.gov
  • fr.ea.cloud.gov
  • fr.eb.cloud.gov
• [If cloud.gov access needed] After they've accepted all CF invites, grant team member Space Developer access to all relevant CF spaces via bin/cf-onboard.

  bin/cf-onboard <EMAIL> SpaceDeveloper prod,staging tah-ea,tah-eb,tah-wb,tah-wc
  

  This requires spaces to be configured by the names above. To set these, install cf targets and configure with:

  cf api api.fr.wb.cloud.gov; cf login --sso; \
      cf t -o gsa-tts-test-kits; \
      cf save-target -f tah-wb
  
  cf api api.fr.wc.cloud.gov; cf login --sso; \
      cf t -o gsa-tts-test-kits; \
      cf save-target -f tah-wc
  
  cf api api.fr.ea.cloud.gov; cf login --sso; \
      cf t -o gsa-tts-test-kits; \
      cf save-target -f tah-ea
  
  cf api api.fr.eb.cloud.gov; cf login --sso; \
      cf t -o gsa-tts-test-kits; \
      cf save-target -f tah-eb
  
  # Verify:
  cf targets
  

Tasks for new team member

• Accept invite to New Relic (need to be on VPN) and confirm access to dashboards
• Review Service Now documentation
  • Confirm you have set your SMS notification correctly
• [If cloud.gov access needed] Accept invite to each CF foundation via email and configure CF targets Note: OTP code doesn't seem to work for Secure Auth; use email. Recommend being on VPN for less flakiness. Initial invites expire after 24 hours, so ask for new one if expired.
  • Accept wb foundation email invite
  • Accept wc foundation email invite
  • Accept ea foundation email invite
  • Accept eb foundation email invite
  • Let admin know you have accepted invites so they can grant you access to spaces
  • After admin grants you space access, configure local CF targets for easy switching:

    # Install [cf-targets plugin](https://github.com/guidowb/cf-targets-plugin)
    cf add-plugin-repo CF-Community https://plugins.cloudfoundry.org/
    cf install-plugin Targets -r CF-Community
    
    # Set targets
    cf api api.fr.wb.cloud.gov; cf login --sso; \
        cf t -o gsa-tts-test-kits; \
        cf save-target -f tah-wb
    
    cf api api.fr.wc.cloud.gov; cf login --sso; \
        cf t -o gsa-tts-test-kits; \
        cf save-target -f tah-wc
    
    cf api api.fr.ea.cloud.gov; cf login --sso; \
        cf t -o gsa-tts-test-kits; \
        cf save-target -f tah-ea
    
    cf api api.fr.eb.cloud.gov; cf login --sso; \
        cf t -o gsa-tts-test-kits; \
        cf save-target -f tah-eb
    

• Familiarize yourself with our incident response checklist
• Familiarize yourself with our application in staging

Resources

Quick links

• Staging application
• Production application
• usagov/test-at-home repository
• Google drive

Documentation

• New Relic configuration
• Getting started with ServiceNow

Runbooks