Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: c4gh deploy #271

Merged
merged 7 commits into from
Oct 16, 2024
Merged

fix: c4gh deploy #271

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
799fd1a
deploy: allow experimental flag to be passed on CDK build code
mmalenic Oct 15, 2024
0b79f79
feat: allow retrieving c4gh keys from secrets manager
mmalenic Oct 15, 2024
5ebf264
feat(config): allow retrieving secret from binary or string
mmalenic Oct 15, 2024
9e5393b
docs(config): add section on using Secrets Manager to fetch keys
mmalenic Oct 15, 2024
331f258
build(deploy): add options to deploy C4GH keys to secrets manager
mmalenic Oct 15, 2024
a3c4f77
fix(storage): ensure c4gh data is read after determining the header size
mmalenic Oct 16, 2024
98b7f84
refactor(search): remove unnecessary Arcs and unwraps in test code
mmalenic Oct 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

24 changes: 13 additions & 11 deletions deploy/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,19 @@ The CDK code in this directory constructs a CDK app from [`HtsgetLambdaStack`][h

These are general settings for the CDK deployment.

| Name | Description | Type |
|--------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|
| <span id="config">`config`</span> | The location of the htsget-rs server config. This must be specified. This config file configures the htsget-rs server. See [htsget-config] for a list of available server configuration options. | `string` |
| <span id="domain">`domain`</span> | The domain name for the Route53 Hosted Zone that the htsget-rs server will be under. This must be specified. A hosted zone with this name will either be looked up or created depending on the value of [`lookupHostedZone?`](#lookupHostedZone). | `string` |
| <span id="authorizer">`authorizer`</span> | Deployment options related to the authorizer. Note that this option allows specifying an AWS [JWT authorizer][jwt-authorizer]. The JWT authorizer automatically verifies tokens issued by a Cognito user pool. | [`HtsgetJwtAuthSettings`](#htsgetjwtauthsettings) |
| <span id="subDomain">`subDomain?`</span> | The domain name prefix to use for the htsget-rs server. Together with the [`domain`](#domain), this specifies url that the htsget-rs server will be reachable under. Defaults to `"htsget"`. | `string` |
| <span id="s3BucketResources">`s3BucketResources`</span> | The buckets to serve data from. If this is not specified, this defaults to `[]`. This affects which buckets are allowed to be accessed by the policy actions which are `["s3:List*", "s3:Get*"]`. Note that this option does not create buckets, it only gives permission to access them, see the `createS3Buckets` option. This option must be specified to allow `htsget-rs` to access data in buckets that are not created in this stack. | `string[]` |
| <span id="lookupHostedZone">`lookupHostedZone?`</span> | Whether to lookup the hosted zone with the domain name. Defaults to `true`. If `true`, attempts to lookup an existing hosted zone using the domain name. Set this to `false` if you want to create a new hosted zone with the domain name. | `boolean` |
| <span id="createS3Bucket">`createS3Bucket?`</span> | Whether to create a test bucket. Defaults to true. Buckets are created with [`RemovalPolicy.RETAIN`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.RemovalPolicy.html). The correct access permissions are automatically added. | `boolean` |
| <span id="bucketName">`bucketName?`</span> | The name of the bucket created using `createS3Bucket`. The name defaults to an automatically generated CDK name, use this option to override that. This option only has an affect is `createS3Buckets` is true. | `string` |
| <span id="copyTestData">`copyTestData?`</span> | Whether to copy test data into the bucket. Defaults to true. This copies the example data under the `data` directory to those buckets. This option only has an affect is `createS3Buckets` is true. | `boolean` |
| Name | Description | Type |
|---------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|
| <span id="config">`config`</span> | The location of the htsget-rs server config. This must be specified. This config file configures the htsget-rs server. See [htsget-config] for a list of available server configuration options. | `string` |
| <span id="domain">`domain`</span> | The domain name for the Route53 Hosted Zone that the htsget-rs server will be under. This must be specified. A hosted zone with this name will either be looked up or created depending on the value of [`lookupHostedZone?`](#lookupHostedZone). | `string` |
| <span id="authorizer">`authorizer`</span> | Deployment options related to the authorizer. Note that this option allows specifying an AWS [JWT authorizer][jwt-authorizer]. The JWT authorizer automatically verifies tokens issued by a Cognito user pool. | [`HtsgetJwtAuthSettings`](#htsgetjwtauthsettings) |
| <span id="subDomain">`subDomain?`</span> | The domain name prefix to use for the htsget-rs server. Together with the [`domain`](#domain), this specifies url that the htsget-rs server will be reachable under. Defaults to `"htsget"`. | `string` |
| <span id="s3BucketResources">`s3BucketResources`</span> | The buckets to serve data from. If this is not specified, this defaults to `[]`. This affects which buckets are allowed to be accessed by the policy actions which are `["s3:List*", "s3:Get*"]`. Note that this option does not create buckets, it only gives permission to access them, see the `createS3Buckets` option. This option must be specified to allow `htsget-rs` to access data in buckets that are not created in this stack. | `string[]` |
| <span id="lookupHostedZone">`lookupHostedZone?`</span> | Whether to lookup the hosted zone with the domain name. Defaults to `true`. If `true`, attempts to lookup an existing hosted zone using the domain name. Set this to `false` if you want to create a new hosted zone with the domain name. | `boolean` |
| <span id="createS3Bucket">`createS3Bucket?`</span> | Whether to create a test bucket. Defaults to true. Buckets are created with [`RemovalPolicy.RETAIN`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.RemovalPolicy.html). The correct access permissions are automatically added. | `boolean` |
| <span id="bucketName">`bucketName?`</span> | The name of the bucket created using `createS3Bucket`. The name defaults to an automatically generated CDK name, use this option to override that. This option only has an affect is `createS3Buckets` is true. | `string` |
| <span id="copyTestData">`copyTestData?`</span> | Whether to copy test data into the bucket. Defaults to true. This copies the example data under the `data` directory to those buckets. This option only has an affect is `createS3Buckets` is true. | `boolean` |
| <span id="features">`copyTestData?`</span> | Whether to create secrets corresponding to C4GH public and private keys that can be used with C4GH storage. This copies the private and public keys in the data directory. Note that private keys copied here are visible in the CDK template. This is not considered secure and should only be used for test data. Real secrets should be manually provisioned or created outside the CDK template. Defaults to false. Secrets are created with [`RemovalPolicy.RETAIN`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.RemovalPolicy.html). | `boolean` |
| <span id="features">`features?`</span> | Additional features to compile htsget-rs with. Defaults to `[]`. `s3-storage` is always enabled. | `string[]` |

#### HtsgetJwtAuthSettings

Expand Down
3 changes: 3 additions & 0 deletions deploy/bin/settings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,7 @@ export const SETTINGS: HtsgetSettings = {
// jwtAudience: ["audience"],
// cogUserPoolId: "user-pool-id",
},
// Enable additional features for compiling htsget-rs. `s3-storage` is always enabled.
features: ["experimental"],
copyExampleKeys: true,
};
5 changes: 5 additions & 0 deletions deploy/config/example_deploy.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,8 @@ environment = "dev"
regex = '^(?P<bucket>.*?)/(?P<key>.*)$'
substitution_string = '$key'
storage.backend = 'S3'

[resolvers.storage.keys]
location = "SecretsManager"
private_key = "htsget-rs/c4gh-private-key" # pragma: allowlist secret
brainstorm marked this conversation as resolved.
Show resolved Hide resolved
recipient_public_key = "htsget-rs/c4gh-recipient-public-key"
48 changes: 47 additions & 1 deletion deploy/lib/htsget-lambda-stack.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import {
CfnOutput,
Duration,
RemovalPolicy,
SecretValue,
Stack,
StackProps,
Tags,
Expand Down Expand Up @@ -42,6 +43,7 @@ import {
BucketEncryption,
} from "aws-cdk-lib/aws-s3";
import { BucketDeployment, Source } from "aws-cdk-lib/aws-s3-deployment";
import { Secret } from "aws-cdk-lib/aws-secretsmanager";

/**
* Settings related to the htsget lambda stack.
Expand Down Expand Up @@ -101,6 +103,20 @@ export type HtsgetSettings = {
* directory to those buckets. This option only has an affect is `createS3Buckets` is true.
*/
copyTestData?: boolean;

/**
* Whether to create secrets corresponding to C4GH public and private keys that can be used with C4GH storage.
* This copies the private and public keys in the data directory. Note that private keys copied here are
* visible in the CDK template. This is not considered secure and should only be used for test data. Real secrets
* should be manually provisioned or created outside the CDK template. Defaults to false. Secrets are created
* with [`RemovalPolicy.RETAIN`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.RemovalPolicy.html).
*/
copyExampleKeys?: boolean;

/**
* Additional features to compile htsget-rs with. Defaults to `[]`. `s3-storage` is always enabled.
*/
features?: string[];
};

/**
Expand Down Expand Up @@ -211,13 +227,43 @@ export class HtsgetLambdaStack extends Stack {
new CfnOutput(this, "HtsgetBucketName", { value: bucket.bucketName });
}

if (settings.copyExampleKeys) {
const dataDir = path.join(__dirname, "..", "..", "data", "c4gh", "keys");
const private_key = new Secret(this, "SecretPrivateKey", {
secretName: "htsget-rs/c4gh-private-key", // pragma: allowlist secret
secretStringValue: SecretValue.unsafePlainText(
readFileSync(path.join(dataDir, "bob.sec")).toString(),
),
removalPolicy: RemovalPolicy.RETAIN,
});
const public_key = new Secret(this, "SecretPublicKey", {
secretName: "htsget-rs/c4gh-recipient-public-key", // pragma: allowlist secret
secretStringValue: SecretValue.unsafePlainText(
readFileSync(path.join(dataDir, "alice.pub")).toString(),
),
removalPolicy: RemovalPolicy.RETAIN,
});

lambdaRole.addToPolicy(
new PolicyStatement({
actions: ["secretsmanager:GetSecretValue"],
resources: [private_key.secretArn, public_key.secretArn],
}),
);
}

lambdaRole.addManagedPolicy(
ManagedPolicy.fromAwsManagedPolicyName(
"service-role/AWSLambdaBasicExecutionRole",
),
);
lambdaRole.addToPolicy(s3BucketPolicy);

let features = settings.features ?? [];
features = features
.filter((f) => f !== "s3-storage")
.concat(["s3-storage"]);

let htsgetLambda = new RustFunction(this, id + "Function", {
manifestPath: path.join(__dirname, "..", ".."),
binaryName: "htsget-lambda",
Expand All @@ -227,7 +273,7 @@ export class HtsgetLambdaStack extends Stack {
CARGO_PROFILE_RELEASE_LTO: "true",
CARGO_PROFILE_RELEASE_CODEGEN_UNITS: "1",
},
cargoLambdaFlags: ["--features", "s3-storage"],
cargoLambdaFlags: ["--features", features.join(",")],
},
memorySize: 128,
timeout: Duration.seconds(28),
Expand Down
Loading
Loading