Auto‐Quarantined Tests

How it works

We're making use of the Quarantine gem. When a test fails, Quarantine will initiate Rspec::Retry to give it another 3 chances to pass.

Here are the possible outcomes of tests:

Test Result	Outcome	Fails Suite?
passes the first time	consecutive passes count is increased; if it is quarantined and the consecutive passes number reaches the specified limit (see Settings below), then it is removed from the quarantine	NO
fails the first time but passes in one of the retries	quarantined as a flaky test	NO
fails the first time and all retries	considered a failing test and not quarantined	YES

An example:

Settings

All settings are changed in rails_helper.rb.

To set the number of retries, replace the value of X below:

config.around(:each) do |example|
  example.run_with_retry(retry: X)
end

To set the number of consecutive passes to remove a test from quarantine, replace the value of X below:

config.quarantine_release_at_consecutive_passes = X

To change when to record tests, change the logic here:

config.quarantine_record_tests = ENV["CI"]

Setup

Google Cloud

In the Google Cloud Project console, do the following.

Enable APIs for a New Project

Create a new project. For that project, enable the following APIs:

1. Google Drive API
2. Google Sheets API

Add a Service Account

Go to APIs & Services --> Credentials, and click +Create Credentials. Choose Service Account. Add an appropriate name and skip the optional settings.

Once created, click the new account's Keys tab. Click Add Key --> Create new key and choose the JSON key type. The key will automatically be downloaded to your computer with the name service-account.json.

Google Drive

Docker Workflow

Create a new Google sheet that will serve as the "database" for quarantined tests. Give the sheet an appropriate name (ie. CASA Docker Quarantine).

Share edit permissions on the spreadsheet with the service account by pasting the email address into the share modal.

Set the name of the worksheet (tab) to test_statuses.

Copy the following titles into the first row of the worksheet (see above).

A	B	C	D	E	F	G
id	full_description	updated_at	last_status	location	extra_attributes	consecutive_passes

Copy the id of the Google sheet (the id of a Google sheet is the alpha-numeric hash between spreadsheets/d/ and /edit in the url).

Paste this in the repo's file .github/workflows/docker.yml as the id on the line containing: QUARANTINE_SHEET_ID. Set the CI line to true.

RSpec Workflow

Do the steps above.

• Set the name of the spreadsheet to something like CASA RSpec Quarantine.
• Paste the sheet ID in the file .github/workflows/rspec.yml as the id on the line containing: QUARANTINE_SHEET_ID.
• Set the CI line to true.

Repository

Encrypt the Service Account File

Encrypt the service-account.json file with a passphrase. Save the passphrase. You'll need to input it in Github later.

gpg --symmetric --cipher-algo AES256 service-account.json

Move the encrypted file into the spec/quarantine/ directory.

Github

Store the passphrase

Click the Settings tab for the project. In the sidebar, click Security --> Secrets and variables --> Actions. Click the Secrets tab. Click the New repository secret button. Set the Name as QUARANTINE_SERVICE_ACCOUNT_PASSPHRASE and paste the passphrase you used to encrypt the service-account.json file. Click Add secret.

Test

You can test this locally. First put the unencrypted service-account.json into the spec/quarantine directory (it's ignored by git). Then run the following from the root of the repository.

CI=true bundle exec rspec spec/quarantine/quarantine_spec.rb

After the test is done, you should see a record for the quarantined spec in the spreadsheet.