Skip to content

Commit

Permalink
Merge pull request #13 from Ontotext-AD/packer-improvements
Browse files Browse the repository at this point in the history
Improve Packer structure
  • Loading branch information
mihailradkov authored Feb 29, 2024
2 parents 008c97a + 315a1a9 commit 185077c
Show file tree
Hide file tree
Showing 5 changed files with 330 additions and 93 deletions.
31 changes: 26 additions & 5 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,28 @@

All notable changes to the Packer template for creating GraphDB AMIs will be documented in this file.

## [1.4.0]
## 1.5.0

- Restructured the sources and grouped the variables
- Increased default GraphDB version to 10.6.1
- Renamed some variables for clarity
- `gdb_version` is now named `graphdb_version`
- `source_ami_name_filter_x86-64` is now named `source_ami_name_filter_x86_64`
- `iam_instance_profile` is now named `build_iam_instance_profile`
- Added new configuration variables
- `access_key` and `secret_key` for static credentials authentication
- `shared_credentials_file` and `shared_credentials_file_profile`
- `source_ami_owners_x86_64` and `source_ami_owners_arm64` for filtering the source AMI owners
- `skip_create_ami` to avoid packaging and publishing the AMIs
- `ami_org_arns` and `ami_ou_arns` for additional access control
- `ami_tags` and `build_tags` for extra AMI tags
- `build_region` to specify the exact region where AMIs are built
- `build_assign_public_ip_address`, `build_security_group_cidrs` and `build_security_group_source_public_ip` to control the public access
to the build EC2 instances
- `user_data_file` to provide custom user data script
- `ssh_interface` and `ssh_username` to control how remote access is established

## 1.4.0

- Installed cloudwatch agent and its needed configurations to be able to push metrics
- Added editorconfig to the project
Expand All @@ -13,7 +34,7 @@ All notable changes to the Packer template for creating GraphDB AMIs will be doc
- Updated README.md
- Added `ssh_clear_authorized_keys` in favor of shredding them in the installation script.

## [1.3.0]
## 1.3.0

- Tuned GraphDB's max RAM percentage to allow bigger heap sizes
- Limited the cluster proxy to 1GB heap at most
Expand All @@ -25,14 +46,14 @@ All notable changes to the Packer template for creating GraphDB AMIs will be doc
- Added `encrypt_boot` to be false
- Added shredding of `/root/.ssh/authorized_keys` and `/home/ubuntu/.ssh/authorized_keys`

## [1.2.0]
## 1.2.0

- Added new configuration for AMI groups `ami_groups`
- Changed `ssh_interface` to `session_manager`
- Added `iam_instance_profile` variable required from the `session_manager`
- Added `Build_Timestamp` tag to the AMIs

## [1.1.0]
## 1.1.0

- Added parallel building of `arm64` and `amd64` based AMIs
- Added AWS cli to be installed based on the architecture
Expand All @@ -42,7 +63,7 @@ All notable changes to the Packer template for creating GraphDB AMIs will be doc
- Changed default instance type for x86-64 to `t3.small`
- Added tags to the AMIs

## [1.0.0]
## 1.0.0

- Initial release of the Packer template.
- Added configuration to create GraphDB AMIs on AWS.
Expand Down
19 changes: 10 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,16 @@ Follow these steps to build an AMI for GraphDB using Packer:
The Packer configuration allows you to customize various parameters, such as the GraphDB version, AWS region,
instance type, VPC ID, and subnet ID. To do so, create a variables file `variables.pkrvars.hcl`, example file:
```bash
gdb_version = "10.3.3"
build_aws_regions = ["us-east-1"]
graphdb_version = "10.3.3"
ami_regions = ["us-east-1"]
build_vpc_id = "<your-vpc-id>"
build_subnet_id = "<your-subnet-id>"
build_iam_instance_profile = "<your-instance-profile>"
ami_groups = [] # Value "all" will make the AMI public
build_instance_type_x86-64 = "t3.small" # default
build_instance_type_x86_64 = "t3.small" # default
build_instance_type_arm64 = "t4g.small" # default
source_ami_name_filter_arm64 = "ubuntu/images/hvm-ssd/ubuntu-*-22.04-arm64-server-*" # default
source_ami_name_filter_x86-64 = "ubuntu/images/hvm-ssd/ubuntu-*-22.04-amd64-server-*" # default
source_ami_name_filter_x86_64 = "ubuntu/images/hvm-ssd/ubuntu-*-22.04-amd64-server-*" # default
```

4. **Build the AMI**:
Expand All @@ -60,16 +61,16 @@ Follow these steps to build an AMI for GraphDB using Packer:
You can customize the Packer configuration and provisioning scripts to suit your specific requirements.

The following points can be customized in a packer variables file `variables.pkrvars.hcl`:
- **GraphDB Version**: You can change the GraphDB version by modifying the `gdb_version` variable file.
- **AWS Regions**: Modify the `build_aws_region` variable to specify a different AWS region.
- **Instance Type**: Adjust the `build_instance_type_arm64` and `build_instance_type_x86-64` variables to select
- **GraphDB Version**: You can change the GraphDB version by modifying the `graphdb_version` variable.
- **AWS Regions**: Modify the `build_region` variable to specify a different AWS region.
- **Instance Type**: Adjust the `build_instance_type_arm64` and `build_instance_type_x86_64` variables to select
different EC2 instance types for building the AMI images.
- **AMI Groups**: You can specify the groups the AMIs will be made available to via the `ami_groups` variable.
A list of strings is accepted.
- **iam_instance_profile**: AIM Instance profile required for the session manager access.
- **build_iam_instance_profile**: AIM Instance profile required for the session manager access.
See https://developer.hashicorp.com/packer/integrations/hashicorp/amazon/latest/components/builder/ebs#session-manager-connections
- **Network Configuration**: Update the `build_vpc_id` and `build_subnet_id` variables to match your VPC and subnet settings.
- **Source AMI**: Use the `source_ami_name_filter_arm64` and `source_ami_name_filter_x86-64` variables to specify the
- **Source AMI**: Use the `source_ami_name_filter_arm64` and `source_ami_name_filter_x86_64` variables to specify the
source ami name filter for each AMI, for example:
- `"ubuntu/images/hvm-ssd/ubuntu-*-22.04-arm64-server-*"` - Ubuntu with `arm64` architecture.
- `"ubuntu/images/hvm-ssd/ubuntu-*-22.04-amd64-server-*"` - Ubuntu with `amd64` architecture.
Expand Down
194 changes: 135 additions & 59 deletions aws-ami.pkr.hcl
Original file line number Diff line number Diff line change
@@ -1,72 +1,148 @@
# Local variable to generate a timestamp for unique AMI naming.
locals {
# Generates a timestamp for unique AMI naming.
timestamp = regex_replace(timestamp(), "[- TZ:]", "")
}

source "amazon-ebs" "ubuntu-x86-64" {
ami_name = "ami-ontotext-graphdb-${var.gdb_version}-x86-64-${local.timestamp}"
instance_type = "${var.build_instance_type_x86-64}"
vpc_id = "${var.build_vpc_id}"
subnet_id = "${var.build_subnet_id}"
ami_regions = "${var.build_aws_regions}"
ami_groups = "${var.ami_groups}"
encrypt_boot = false
ebs_optimized = true

tags = {
GDB_Version = "${var.gdb_version}"
CPU_Architecture = "x86-64"
Build_Timestamp = "${local.timestamp}"
data "amazon-ami" "ubuntu_x86_64" {
filters = {
name = var.source_ami_name_filter_x86_64
root-device-type = "ebs"
virtualization-type = "hvm"
}
most_recent = true
owners = var.source_ami_owners_x86_64
}

source_ami_filter {
filters = {
name = "${var.source_ami_name_filter_x86-64}"
root-device-type = "ebs"
virtualization-type = "hvm"
}
most_recent = true
owners = ["099720109477"]
}
source "amazon-ebs" "ubuntu_x86_64" {
skip_create_ami = var.skip_create_ami

#
# AMI configurations
#
ami_name = "ami-ontotext-graphdb-${var.graphdb_version}-x86-64-${local.timestamp}"
ami_description = "GraphDB v${var.graphdb_version} by Ontotext"

ami_virtualization_type = "hvm"
encrypt_boot = false
ebs_optimized = true
ena_support = true

ami_regions = var.ami_regions
ami_users = var.ami_users
ami_groups = var.ami_groups
ami_org_arns = var.ami_org_arns
ami_ou_arns = var.ami_ou_arns

tags = merge({
GraphDB_Version = var.graphdb_version
CPU_Architecture = "x86-64"
Build_Timestamp = local.timestamp
}, var.ami_tags)

#
# Access configurations
#
access_key = var.access_key
secret_key = var.secret_key

shared_credentials_file = var.shared_credentials_file
profile = var.shared_credentials_file_profile

#
# Build configurations
#
region = var.build_region
vpc_id = var.build_vpc_id
subnet_id = var.build_subnet_id

instance_type = var.build_instance_type_x86_64
source_ami = data.amazon-ami.ubuntu_x86_64.id
iam_instance_profile = var.build_iam_instance_profile
user_data_file = var.user_data_file

associate_public_ip_address = var.build_assign_public_ip_address
temporary_security_group_source_cidrs = var.build_security_group_cidrs
temporary_security_group_source_public_ip = var.build_security_group_source_public_ip

run_tags = var.build_tags

ssh_username = "ubuntu"
associate_public_ip_address = true
ssh_interface = "session_manager"
communicator = "ssh"
iam_instance_profile = "${var.iam_instance_profile}"
ssh_clear_authorized_keys = true
#
# SSH / Session Manager
#
communicator = "ssh"
ssh_interface = var.ssh_interface
ssh_username = "ubuntu"
ssh_clear_authorized_keys = true
}

source "amazon-ebs" "ubuntu-arm64" {
ami_name = "ami-ontotext-graphdb-${var.gdb_version}-arm64-${local.timestamp}"
instance_type = "${var.build_instance_type_arm64}"
vpc_id = "${var.build_vpc_id}"
subnet_id = "${var.build_subnet_id}"
ami_regions = "${var.build_aws_regions}"
ami_groups = "${var.ami_groups}"
encrypt_boot = false
ebs_optimized = true

tags = {
GDB_Version = "${var.gdb_version}"
CPU_Architecture = "arm64"
Build_Timestamp = "${local.timestamp}"
data "amazon-ami" "ubuntu_arm64" {
filters = {
name = var.source_ami_name_filter_arm64
root-device-type = "ebs"
virtualization-type = "hvm"
}
most_recent = true
owners = var.source_ami_owners_arm64
}

source_ami_filter {
filters = {
name = "${var.source_ami_name_filter_arm64}"
root-device-type = "ebs"
virtualization-type = "hvm"
}
most_recent = true
owners = ["099720109477"]
}
source "amazon-ebs" "ubuntu_arm64" {
skip_create_ami = var.skip_create_ami

#
# AMI configurations
#
ami_name = "ami-ontotext-graphdb-${var.graphdb_version}-arm64-${local.timestamp}"
ami_description = "GraphDB v${var.graphdb_version} by Ontotext"

ami_virtualization_type = "hvm"
encrypt_boot = false
ebs_optimized = true
ena_support = true

ami_regions = var.ami_regions
ami_users = var.ami_users
ami_groups = var.ami_groups
ami_org_arns = var.ami_org_arns
ami_ou_arns = var.ami_ou_arns

tags = merge({
GraphDB_Version = var.graphdb_version
CPU_Architecture = "arm64"
Build_Timestamp = local.timestamp
}, var.ami_tags)

#
# Access configurations
#
access_key = var.access_key
secret_key = var.secret_key

shared_credentials_file = var.shared_credentials_file
profile = var.shared_credentials_file_profile

#
# Build configurations
#
region = var.build_region
vpc_id = var.build_vpc_id
subnet_id = var.build_subnet_id

instance_type = var.build_instance_type_arm64
source_ami = data.amazon-ami.ubuntu_arm64.id
iam_instance_profile = var.build_iam_instance_profile
user_data_file = var.user_data_file

associate_public_ip_address = var.build_assign_public_ip_address
temporary_security_group_source_cidrs = var.build_security_group_cidrs
temporary_security_group_source_public_ip = var.build_security_group_source_public_ip

run_tags = var.build_tags

ssh_username = "ubuntu"
associate_public_ip_address = true
ssh_interface = "session_manager"
communicator = "ssh"
iam_instance_profile = "${var.iam_instance_profile}"
ssh_clear_authorized_keys = true
#
# SSH / Session Manager
#
communicator = "ssh"
ssh_interface = var.ssh_interface
ssh_username = "ubuntu"
ssh_clear_authorized_keys = true
}
6 changes: 3 additions & 3 deletions build.pkr.hcl
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
build {
name = "graphdb-ami"
sources = [
"source.amazon-ebs.ubuntu-x86-64",
"source.amazon-ebs.ubuntu-arm64"
"source.amazon-ebs.ubuntu_x86_64",
"source.amazon-ebs.ubuntu_arm64"
]

provisioner "file" {
Expand All @@ -18,7 +18,7 @@ build {

provisioner "shell" {
environment_vars = [
"GRAPHDB_VERSION=${var.gdb_version}",
"GRAPHDB_VERSION=${var.graphdb_version}",
]
inline = ["sudo -E bash /tmp/install_graphdb.sh"]
max_retries = 3
Expand Down
Loading

0 comments on commit 185077c

Please sign in to comment.