elasticsearch-asg

A number of little applications I needed setting up Elasticsearch on AWS Autoscaling Groups. These target Elasticsearch >= 7.0.

• cloudwatcher () - Push metrics about an Elasticsearch cluster to AWS CloudWatch, mainly to inform AWS Target Tracing Scaling Policies.

• drainer () - Remove shards from Elasticsearch nodes on EC2 instances that are about to be terminated - either by an AWS AutoScaling Group downscaling or by Spot Instance interruption - by consuming CloudWatch Events from an SQS Queue.

• throttler () - Regulate an AWS AutoScaling Group running Elasticsearch by preventing new autoscaling actions until the cluster is stable (not red, no relocating shards, etc).

• snapshooter () - Take snapshots of Elasticsearch cluster on a schedule, and clean up old ones with downsampling.

Directory layout

• /cmd - A little bit of glue to run each application.
• /pkg - Packages that should theoretically be reusable in other projects.
• /internal - Code that isn't reusable in other projects.
  • /internal/app - The implementation of each application.
  • /internal/pkg - Packages used across multiple applications.

How-To

Start developing

You need a working Go >= 1.13 environment. Then clone this repository:

mkdir -p $GOPATH/src/github.com/mintel/
cd $GOPATH/src/github.com/mintel/
git clone git@github.com:mintel/elasticsearch-asg.git
cd elasticsearch-asg

Run tests

go test ./...

Build binaries

for app in cloudwatcher drainer snapshooter throttler; do
    go build "./cmd/$app"
done

Build Docker containers

for app in cloudwatcher drainer snapshooter throttler; do
    docker build . -f "Dockerfile.$app" -t "mintel/elasticsearch-$app:latest"
done

Debug

Install Delve.

To debug one of the apps:

dlv debug ./cmd/cloudwatcher -- --elasticsearch.url=http://localhost:9200

To debug tests in the current directory:

dlv test

See the Delve Getting Started guide for more details.

Make a new release

If you tag a commit with a semantic version, Docker Hub will build it as a separate tag. For example:

git tag -a v1.2.3 -m "Release v1.2.3"
git push --follow-tags

Will build Docker images like mintel/elasticsearch-cloudwatcher:1.2.3.