Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add initial support for Elastic Stack 9.x #2102

Merged
merged 13 commits into from
Sep 19, 2024
Merged

Add initial support for Elastic Stack 9.x #2102

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
164e1ca
Update default images if no version specified
mrodm Sep 13, 2024
1135e80
Add test stack command for 9.x stack
mrodm Sep 13, 2024
2636638
Update tests
mrodm Sep 13, 2024
031aa4b
Create updatecli config for 9.x-snapshot
mrodm Sep 13, 2024
03ad1e9
Update names and ids github workflows
mrodm Sep 13, 2024
0dd9078
disable target temporarily
mrodm Sep 13, 2024
d447532
Remove prefix 'v' from docker compose version
mrodm Sep 17, 2024
10a5d76
Test without installing docker/docker-compose
mrodm Sep 17, 2024
332c2e5
Show version in error log message
mrodm Sep 17, 2024
d918cc2
Revert "Test without installing docker/docker-compose"
mrodm Sep 17, 2024
40deb26
Remove TODO message
mrodm Sep 17, 2024
afee3c6
Enable updatecli configuration
mrodm Sep 18, 2024
168ae0b
Merge remote-tracking branch 'upstream/main' into support-elastic-age…
mrodm Sep 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .buildkite/pipeline.trigger.integration.tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ STACK_COMMAND_TESTS=(
test-stack-command-7x
test-stack-command-86
test-stack-command-8x
test-stack-command-9x
test-stack-command-with-apm-server
)

Expand Down
2 changes: 1 addition & 1 deletion .buildkite/scripts/stack_tests_windows.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ fixCRLF

withGolang $env:GO_VERSION
withDocker $env:DOCKER_VERSION
withDockerCompose $env:DOCKER_COMPOSE_VERSION
withDockerCompose $env:DOCKER_COMPOSE_VERSION.Substring(1)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Required to remove v from the version string for the chocolotey tool. Currently, this environment variable has this value v2.24.1.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, how has it worked in the past?

Copy link
Contributor Author

@mrodm mrodm Sep 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably, it was failing in the past too. I realized that even if with those commands were failing, the execution of the script continued.
I see that at the beginning of the file is defined

$ErrorActionPreference = "Stop"

but it does not stopped the execution. Not sure if choco command does not return any error in that scenario, or when the error is inside a function there are some differences in how to manage the errors.


Write-Host "--- Docker Info"
docker info
Expand Down
4 changes: 2 additions & 2 deletions ...atecli.d/bump-latest-snapshot-version.yml → ...cli.d/bump-latest-8x-snapshot-version.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: Bump latest elastic-stack test version
pipelineid: 'bump-latest-elastic-stack-version'
name: Bump latest 8.x-SNAPSHOT elastic-stack test version
pipelineid: 'bump-latest-8x-elastic-stack-version'

actions:
default:
Expand Down
44 changes: 44 additions & 0 deletions .github/workflows/updatecli/updatecli.d/bump-latest-9x-snapshot-version.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
name: Bump latest 9.x-SNAPSHOT elastic-stack test version
pipelineid: 'bump-latest-9x-elastic-stack-version'

actions:
default:
title: '[updatecli] Update latest snapshot to {{ source "latestSnapshot" }}'
kind: github/pullrequest
scmid: default
spec:
labels:
- automation
- dependency

scms:
default:
kind: github
spec:
owner: '{{ .scm.owner }}'
repository: '{{ .scm.repository }}'
user: '{{ requiredEnv "GITHUB_ACTOR" }}'
username: '{{ requiredEnv "GITHUB_ACTOR" }}'
token: '{{ requiredEnv "GITHUB_TOKEN" }}'
commitusingapi: true
branch: main

sources:
latestSnapshot:
name: Get latest snapshot build
kind: json
spec:
file: https://storage.googleapis.com/artifacts-api/snapshots/main.json
key: .build_id

targets:
update-snapshot:
name: '[updatecli] Update latest snapshot to {{ source "latestSnapshot" }}'
kind: file
sourceid: latestSnapshot
scmid: default
spec:
file: Makefile
matchpattern: '(./scripts/test-stack-command.sh) 9\.[^\s]+-SNAPSHOT'
replacepattern: '$1 {{ source "latestSnapshot" }}-SNAPSHOT'
5 changes: 4 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,16 @@ test-stack-command-86:
test-stack-command-8x:
./scripts/test-stack-command.sh 8.16.0-5f5fcfdc-SNAPSHOT

test-stack-command-9x:
./scripts/test-stack-command.sh 9.0.0-SNAPSHOT

test-stack-command-with-apm-server:
APM_SERVER_ENABLED=true ./scripts/test-stack-command.sh

test-stack-command-with-self-monitor:
SELF_MONITOR_ENABLED=true ./scripts/test-stack-command.sh

test-stack-command: test-stack-command-default test-stack-command-7x test-stack-command-800 test-stack-command-8x test-stack-command-with-apm-server
test-stack-command: test-stack-command-default test-stack-command-7x test-stack-command-800 test-stack-command-8x test-stack-command-9x test-stack-command-with-apm-server

test-check-packages: test-check-packages-with-kind test-check-packages-other test-check-packages-parallel test-check-packages-with-custom-agent test-check-packages-benchmarks test-check-packages-false-positives test-check-packages-with-logstash

Expand Down
6 changes: 3 additions & 3 deletions internal/install/application_configuration.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,13 +153,13 @@ func (ac *ApplicationConfiguration) SetCurrentProfile(name string) {
// This is mandatory as "elastic-agent-complete" is available since 7.15.0-SNAPSHOT.
func selectElasticAgentImageName(version, agentBaseImage string) string {
if version == "" { // as version is optional and can be empty
return elasticAgentLegacyImageName
return elasticAgentWolfiImageName
Copy link
Contributor Author

@mrodm mrodm Sep 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about changing these defaults values to use the wolfi image docker.elastic.co/elastic-agent/elastic-agent-wolfi (e.g. no version specified) instead of the legacy image (beats namespace) docker.elastic.co/beats/elastic-agent?

Or should we keep unchanged this default?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can adapt this to the new default, yes, if we are sure that in old versions the beats namespace is used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the same behaviour is kept.
As a reference, I add here the docker images used in different versions:

  • 7.14.0 
     $ docker ps --format "{{.Names}} {{.Image}}" |grep elastic-agent
elastic-package-stack-elastic-agent-1 docker.elastic.co/beats/elastic-agent:7.14.0
elastic-package-stack-fleet-server-1 docker.elastic.co/beats/elastic-agent:7.14.0
  • 7.15.0 
     $ docker ps --format "{{.Names}} {{.Image}}" |grep elastic-agent
elastic-package-stack-elastic-agent-1 docker.elastic.co/beats/elastic-agent-complete:7.15.0
elastic-package-stack-fleet-server-1 docker.elastic.co/beats/elastic-agent-complete:7.15.0
  • 8.2.0 
     $ docker ps --format "{{.Names}} {{.Image}}" |grep elastic-agent
elastic-package-stack-elastic-agent-1 docker.elastic.co/elastic-agent/elastic-agent-complete:8.2.0
elastic-package-stack-fleet-server-1 docker.elastic.co/elastic-agent/elastic-agent-complete:8.2.0
  • 8.16.0 (SNAPSHOT) 
     $ docker ps --format "{{.Names}} {{.Image}}" |grep elastic-agent
elastic-package-stack-elastic-agent-1 docker.elastic.co/elastic-agent/elastic-agent-wolfi:8.16.0-SNAPSHOT
elastic-package-stack-fleet-server-1 docker.elastic.co/elastic-agent/elastic-agent-wolfi:8.16.0-SNAPSHOT
  • 9.0.0 (SNAPSHOT) 
     $ docker ps --format "{{.Names}} {{.Image}}" |grep elastic-agent
elastic-package-stack-elastic-agent-1 docker.elastic.co/elastic-agent/elastic-agent-wolfi:9.0.0-SNAPSHOT
elastic-package-stack-fleet-server-1 docker.elastic.co/elastic-agent/elastic-agent-wolfi:9.0.0-SNAPSHOT

}

v, err := semver.NewVersion(version)
if err != nil {
logger.Errorf("stack version not in semver format (value: %s): %v", v, err)
return elasticAgentLegacyImageName
logger.Errorf("stack version not in semver format (value: %s): %v", version, err)
return elasticAgentWolfiImageName
}

disableWolfiImages := false
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it is now, the Elastic Agent image is chosen as:

  • If stack >= 8.16.0-SNAPSHOT,
    • by default it is used wolfi image_ docker.elastic.co/elastic-agent/elastic-agent-wolfi.
    • if ELASTIC_PACKAGE_DISABLE_ELASTIC_AGENT_WOLFI=true, then it is selected the Elastic Agent image as before. So if stack >= 7.15.0-SNAPSHOT, it will be used the complete image.
  • If the configuration file of the system test has base_image: complete:
    • It will be used the complete version of the Elastic Agent image if the stack version allows it.
  • If the configuration file of the system test has base_image: systemd:
    • It will be used the systemd version of the Elastic Agent image. That means docker.elastic.co/elastic-agent/elastic-agent or docker.elastic.co/beats/elastic-agent depending on the stack version.

Would that be ok? Specially if stack version is set to 9.0.0 and the environment variable to disable wolfi is set to true.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the same set of images is going to be available in 9.0 keeping the same logic sounds good to me.

Expand Down
32 changes: 16 additions & 16 deletions internal/install/application_configuration_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,43 +14,43 @@ import (
func TestSelectElasticAgentImageName_NoVersion(t *testing.T) {
var version string
selected := selectElasticAgentImageName(version, "")
assert.Equal(t, selected, elasticAgentLegacyImageName)
assert.Equal(t, elasticAgentWolfiImageName, selected)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the test changing the value, it would return the wolfi image instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In principle it LGTM. Maybe another option is to return an error if the version is empty? Not sure about the implications of the change, but would look safer to me to know where/if we are passing an empty version.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commands like elastic-package stack down do not set any version, and if a error is returned here, it will fail the command.

}

func TestSelectElasticAgentImageName_OlderStack(t *testing.T) {
version := "7.14.99-SNAPSHOT"
selected := selectElasticAgentImageName(version, "")
assert.Equal(t, selected, elasticAgentLegacyImageName)
assert.Equal(t, elasticAgentLegacyImageName, selected)
}

func TestSelectElasticAgentImageName_FirstStackWithCompleteAgent(t *testing.T) {
version := stackVersion715
selected := selectElasticAgentImageName(version, "")
assert.Equal(t, selected, elasticAgentCompleteLegacyImageName)
assert.Equal(t, elasticAgentCompleteLegacyImageName, selected)
}

func TestSelectElasticAgentImageName_NextStackWithAgentComplete(t *testing.T) {
version := "7.16.0-SNAPSHOT"
selected := selectElasticAgentImageName(version, "")
assert.Equal(t, selected, elasticAgentCompleteLegacyImageName)
assert.Equal(t, elasticAgentCompleteLegacyImageName, selected)
}

func TestSelectElasticAgentImageName_OwnNamespace(t *testing.T) {
version := "8.2.0-SNAPSHOT"
selected := selectElasticAgentImageName(version, "")
assert.Equal(t, selected, elasticAgentCompleteImageName)
assert.Equal(t, elasticAgentCompleteImageName, selected)
}

func TestSelectElasticAgentImageName_OwnNamespace_Release(t *testing.T) {
version := "8.2.0"
selected := selectElasticAgentImageName(version, "")
assert.Equal(t, selected, elasticAgentCompleteImageName)
assert.Equal(t, elasticAgentCompleteImageName, selected)
}

func TestSelectElasticAgentImageName_NextStackInOwnNamespace(t *testing.T) {
version := "8.4.0-SNAPSHOT"
selected := selectElasticAgentImageName(version, "")
assert.Equal(t, selected, elasticAgentCompleteImageName)
assert.Equal(t, elasticAgentCompleteImageName, selected)
}

func TestSelectElasticAgentImageName_DefaultImage816_WithoutEnvVar(t *testing.T) {
Expand All @@ -60,56 +60,56 @@ func TestSelectElasticAgentImageName_DefaultImage816_WithoutEnvVar(t *testing.T)
os.Unsetenv(disableElasticAgentWolfiEnvVar)

selected := selectElasticAgentImageName(version, "")
assert.Equal(t, selected, elasticAgentWolfiImageName)
assert.Equal(t, elasticAgentWolfiImageName, selected)
}

func TestSelectElasticAgentImageName_DisableWolfiImageEnvVar(t *testing.T) {
version := stackVersion8160
t.Setenv(disableElasticAgentWolfiEnvVar, "true")
selected := selectElasticAgentImageName(version, "")
assert.Equal(t, selected, elasticAgentCompleteImageName)
assert.Equal(t, elasticAgentCompleteImageName, selected)
}
func TestSelectElasticAgentImageName_EnableWolfiImageEnvVar(t *testing.T) {
version := stackVersion8160
t.Setenv(disableElasticAgentWolfiEnvVar, "false")
selected := selectElasticAgentImageName(version, "")
assert.Equal(t, selected, elasticAgentWolfiImageName)
assert.Equal(t, elasticAgentWolfiImageName, selected)
}

func TestSelectCompleteElasticAgentImageName_ForceCompleteImage(t *testing.T) {
version := stackVersion8160
selected := selectElasticAgentImageName(version, "complete")
assert.Equal(t, selected, elasticAgentCompleteImageName)
assert.Equal(t, elasticAgentCompleteImageName, selected)
}

func TestSelectCompleteElasticAgentImageName_ForceDefaultImage_DisabledEnvVar(t *testing.T) {
version := stackVersion8160
t.Setenv(disableElasticAgentWolfiEnvVar, "true")
selected := selectElasticAgentImageName(version, "default")
assert.Equal(t, selected, elasticAgentCompleteImageName)
assert.Equal(t, elasticAgentCompleteImageName, selected)
}

func TestSelectCompleteElasticAgentImageName_ForceDefaultImage_EnabledEnvVar(t *testing.T) {
version := stackVersion8160
t.Setenv(disableElasticAgentWolfiEnvVar, "false")
selected := selectElasticAgentImageName(version, "default")
assert.Equal(t, selected, elasticAgentWolfiImageName)
assert.Equal(t, elasticAgentWolfiImageName, selected)
}

func TestSelectCompleteElasticAgentImageName_ForceDefaultImageOldStack(t *testing.T) {
version := "8.15.0-SNAPSHOT"
selected := selectElasticAgentImageName(version, "default")
assert.Equal(t, selected, elasticAgentCompleteImageName)
assert.Equal(t, elasticAgentCompleteImageName, selected)
}

func TestSelectCompleteElasticAgentImageName_ForceSystemDImage(t *testing.T) {
version := stackVersion8160
selected := selectElasticAgentImageName(version, "systemd")
assert.Equal(t, selected, elasticAgentImageName)
assert.Equal(t, elasticAgentImageName, selected)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the order to ensure that the values are the expected and the actual ones as required by the assert functions.
https://pkg.go.dev/github.com/stretchr/testify/assert#Equal

}

func TestSelectCompleteElasticAgentImageName_ForceSystemDImageOldStack(t *testing.T) {
version := stackVersion715
selected := selectElasticAgentImageName(version, "systemd")
assert.Equal(t, selected, elasticAgentLegacyImageName)
assert.Equal(t, elasticAgentLegacyImageName, selected)
}
Loading