Skip to content

DataBiosphere/terra-landing-zone-service

Repository files navigation

Landing Zones

Quality Gate Status

Overview

A Landing Zone is a set of cloud resources that serve as the underlying infrastructure where workspaces or other Terra applications can be deployed. The resources in a Landing Zone define and implement constraints, provide cross-cutting features, or can be shared. These resources have a different lifecycle than resources in workspaces.

Current state

The Landing Zone "Service" is currently being used as a library attached to Workspace Manager. It is in the process of being stood up as a standalone service. See README.md for directions on running the service.

Landing Zone Manager

The Landing Zone Manager is the high-level component that lists the available Landing Zone Step Definition factories, deploys Landing Zone and lists resources per purpose.

The Landing Zone Manager requires a TokenCredential, AzureProfile and resourceGroupName.

landingZoneManager=
        LandingZoneManager.createLandingZoneManager(
        credential,
        azureProfile,
        resourceGroupName);

Reading Landing Zone Resources

You can list resources by purpose using the Landing Zone Manager:

List<DeployedResource> resources=landingZoneManager.reader().listResourcesByPurpose(landingZoneId, ResourcePurpose.SHARED_RESOURCE);

Virtual Networks can be listed by subnet purpose:

    List<DeployedVNet> vNets=
        landingZoneManager.reader().listVNetWithSubnetPurpose(landingZoneId, SubnetResourcePurpose.WORKSPACE_COMPUTE_SUBNET);

Getting the available Landing Zone Factories

You can get all available Landing Zone Step Factories:

List<FactoryInfo> factories=LandingZoneManager.listDefinitionFactories();

Landing Zone Service

The Landing Zone Service is a Spring service component that wraps the Landing Zone Manager, provides a job based/async API to deploy Landing Zones, and persists Landing Zone deployments state in a DB.

Landing Zone Definitions

The table below describes the current Landing Zone Definitions available in the library.

Factory Description Versions Shared Resources Parameters
CromwellBaseResourcesFactory Deploys required resources to run Cromwell on Azure. V1
  • AKS Cluster
  • Batch Account
  • Storage Account
  • PostgreSQL
  • Azure Relay Namespace
  • VNet with subnets for PostgreSQL, AKS Node pool, PostgreSQL databases and Compute resources
POSTGRES_DB_ADMIN: Username of the DB admin
Default value: db_admin

POSTGRES_DB_PASSWORD: DB admin password
Default value: UUID.randomUUID().toString()

POSTGRES_SERVER_SKU: PostgreSQL Server SKU
Default value: Standard_D2ds_v5

POSTGRES_SERVER_SKU_TIER: PostgreSQL Server Compute Tier
Default value: General Purpose

VNET_ADDRESS_SPACE: Virtual network address space
Default value: 10.1.0.0/27

AKS_SUBNET: AKS subnet address space
Default value: 10.1.0.0/29

BATCH_SUBNET: Batch subnet address space
Default value: 10.1.0.8/29

POSTGRESQL_SUBNET: PostgreSQL subnet address space
Default value: 10.1.0.16/29

COMPUTE_SUBNET: Compute resources subnet address space
Default value: 10.1.0.24/29

AKS_NODE_COUNT: Number of nodes in AKS Nodepool
Default value: 1

AKS_MACHINE_TYPE: Machine type used for AKS hosts
Default value: ContainerServiceVMSizeTypes.STANDARD_A2_V2

AKS_AUTOSCALING_ENABLED: Flag to enabled autoscaling for the AKS nodepool
Default value: false

AKS_AUTOSCALING_MIN: Minimum number of nodes in nodepool when autoscaling is enabled
Default value: 1

AKS_AUTOSCALING_MAX: Maximum number of nodes in nodepool when autoscaling is enabled
Default value: 3

AKS_COST_SAVING_SPOT_NODES_ENABLED: Enable Spot Node usage on AKS
Default value: false

Azure storage account overview: documentation

Azure storage CORS configuration: documentation

STORAGE_ACCOUNT_BLOB_CORS_ALLOWED_ORIGINS: The origin domains that are permitted to make a request against the storage service via CORS
Default value: *

STORAGE_ACCOUNT_BLOB_CORS_ALLOWED_METHODS: The methods (HTTP request verbs) that the origin domain may use for a CORS request
Default value: GET,HEAD,OPTIONS,PUT,PATCH,POST,MERGE,DELETE

STORAGE_ACCOUNT_BLOB_CORS_ALLOWED_HEADERS: The request headers that the origin domain may specify on the CORS request
Default value: authorization,content-type,x-app-id,Referer,x-ms-blob-type,x-ms-copy-source,content-length

STORAGE_ACCOUNT_BLOB_CORS_EXPOSED_HEADERS: The response headers that may be sent in the response to the CORS request and exposed by the browser to the request issuer
Default value: Empty string

STORAGE_ACCOUNT_BLOB_CORS_MAX_AGE: The maximum amount time that a browser should cache the preflight OPTIONS request (in seconds)
Default value: 0

Azure storage SKU types: documentation

STORAGE_ACCOUNT_SKU_TYPE: Type of storage account
Default value: Standard_LRS;
Accepted values: Standard_LRS, Standard_GRS, Standard_RAGRS, Standard_ZRS, Premium_LRS; Please see StorageAccountSkuType;

ENABLE_PGBOUNCER: Whether to have pgbouncer enabled on postgresql server
Default value: true;
ProtectedDataResourcesFactory Deploy additional resources to the landing zone for additional security monitoring (using Azure Sentinel) and exporting logs to centralized long-term storage for retention. V1
  • All resources as in CromwellBaseResourcesFactory
  • Long term storage account
  • Additional AKS log configuration
Same as for CromwellBaseResourcesFactory

Testing

Contract Testing

We use pact for contract testing between the landing zone service and other services.

Consumer Contract Tests

Contract tests are excluded from the base "tests" gradle task. To run the consumer tests, use the following gradle invocation from the repository root:

./gradlew :service:pactConsumerTests

Provider Contract Tests

Provider contract tests require a running postgres:

./service/local-dev/run_postgres.sh start

They also require input pacts, either pulled from a broker or stored locally. To run against a broker, use this invocation:

PACT_BROKER_USERNAME="<brokerusername>" PACT_BROKER_PASSWORD"<brokerpassword>" PACT_BROKER_URL="<brokerurl>" ./gradlew :service:verifyPacts

The required broker information is stored in vault. Ask in #dsp-workspaces for more info.

Additional information

Please read CONTRIBUTING.md for more information about the process of running tests and contributing code to the repository and DESIGN.md for a deeper understanding of the repository's structure and design patterns.