SRE_INFO.md

SRE Info

Infra Access

To access Kubernetes Masters, you need to be on the MDC1 or MDC2 VPN. The Kubeconfig is in the private secrets repo, there is one for each of the three clusters.

SRE aws-vault setup

SRE account guide

SRE AWS accounts

Secrets

Secrets are stored in sumo-private, a private repo using git-crypt

Private repo with git-crypt guide

Source Repos

Application repo kitsune

Infrastructure repo sumo-infra

Secrets repo sumo-private which includes the yar bot that blocks malicious IPs

Block-aws repo block-aws

Monitoring

New Relic APM

New Relic Synthetics

SSL Certificates

SUMO uses certs from AWS ACM

SSL Cert Monitoring

Cloud Account

AWS account mozilla-sumo 095732026120

Observability, Disaster Recovery, and Alerting for SRE Properties

Service	Front End	Back End	Database
Synthetic Monitoring	Synthetics, Synthetics - Search, SUMO	SUMO Prod TP, SUMO Prod Oregon A, SUMO Prod Oregon B, SUMO Prod Frankfurt	N/A
Application Monitoring	SUMO Prod Oregon A, SUMO Prod Oregon B, SUMO Prod Frankfurt	N/A	InfluxDB SUMO Dashboard
Infrastructure Monitoring	InfluxDB SUMO Dashboard	CloudWatch Oregon-B k8s node ASG size Slack Alarm, CloudWatch Oregon-A k8s node ASG size Slack Alarm, CloudWatch Oregon-B k8s master ASG size Slack Alarm, CloudWatch Oregon-A k8s master ASG size Slack Alarm, CloudWatch RDS Replication Slack Alarm, CloudWatch Frankfurt k8s master ASG size Slack Alarm, CloudWatch Frankfurt k8s nodes ASG size Slack Alarm, CloudWatch Frankfurt VPN Slack Alarm, CloudWatch Oregon VPN Slack Alarm	N/A
Cert Monitoring	InfluxDB Cert Monitoring	N/A	N/A
Status Dashboard	Mozilla Websites StatusPage	Mozilla Support StatusPage	N/A
Logging	Papertrail SUMO	N/A	N/A
Pager Duty	PagerDuty support.mozilla.org-itsre Service	PagerDuty support.mozilla.org-itsre Escalation Policy	N/A
Backups	GitHub Kitsune repo, GitHub sumo-infra repo, GitHub sumo-private repo, GitHub block-aws repo	Link to RDS / S3 / AWS backups configuration	RDS prod snapshots, RDS dev snapshots