Add serverless provider #1374

mrodm · 2023-07-28T15:31:50Z

This PR adds a new provider to create Serverless projects.

It requires to set these two environment variables :

EC_API_KEY
EC_HOST

Notes:

In pipeline tests, Geo IP field keys have been skipped
- It cannot be set a specific GeoIP database to make the test stable.
A local agent is started locally (using docker-compose). This agent is configured and connected to the new project.
- As done locally, if elastic-agent starts to fail, it is retried once

Tested with the following packages from integrations repository:

elastic_package_registry
nginx
apache
haproxy
system
mysql
squid
gcp
tomcat
zookeeper
winlog
zoom
sql_input
redis
aws (just asset and pipeline tests)

How to test

export EC_API_KEY="xx"
# if not defined EC_HOST, it will be used the default value
export EC_HOST="xx"

elastic-package profiles create serverless --from default
elastic-package profiles use serverless

# check that this new profile is selected
elastic-package profiles list

# Boot up a new project and check its status
elastic-package stack up -v -d --version 8.9.0 --provider serverless
elastic-package stack status

# from the package folder install or test
elastic-package install
elastic-package test -v

# Delete the project
elastic-package install down -v

fleet server endpoint is built based on the elasticsearch endpoint fix creation

mrodm · 2023-08-16T16:25:18Z

/test

elasticmachine · 2023-08-16T17:02:11Z

💚 Build Succeeded

Buildkite Build
Commit: 3f38c52

History

💔 Build #1245 failed 3f38c52
💔 Build #1244 failed d0f9a9b
💔 Build #1243 failed 601eec1
💔 Build #1242 failed 4debb1e

cc @mrodm

Raise an error in case elasticsearch project is created, since it does not have fleet service.

internal/stack/serverless.go

mrodm · 2023-08-23T11:02:56Z

internal/stack/dump.go

-var observedServices = []string{"elasticsearch", elasticAgentService, fleetServerService, "kibana", "package-registry"}
-


Now dump and status would be getting the services directly from the docker daemon. Not all these services are available when serverless provider is used.

These containers are obtained from here:
https://github.com/elastic/elastic-package/pull/1374/files#diff-b4efb1ac558d15d3077525cb9be59dce12ee723a4749253e9ca7cc860419f1f8R401-R462

Added descriptions for new profile configuirations added

mrodm · 2023-09-06T08:54:18Z

buildkite test this

mrodm · 2023-09-06T09:40:39Z

buildkite test this

mrodm · 2023-09-06T10:38:59Z

/test

jsoriano

Great job here 🚀

internal/profile/_static/config.yml.example

internal/serverless/client.go

jsoriano · 2023-09-06T17:29:20Z

internal/serverless/client.go

 	if host == "" {
-		return nil, fmt.Errorf("unable to obtain value from %s environment variable", hostEnvName)
+		logger.Debugf("Using default host URL: %s", defaultHostURL)


Nit. Maybe we can print in any case what endpoint is used.

Moved this logger.Debug after creating the client with all the options.

I also ensured that environment variables (EC_HOST) has preference over the configuration set in the profile (e.g. ~/.elastic-package/profiles/serverless/config.yml)

internal/stack/serverless.go

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

mrodm · 2023-09-07T08:29:10Z

internal/serverless/client.go

+	host := os.Getenv(elasticCloudEndpointEnv)
+	if host != "" {
+		c.host = host
+	}


If there is an environment variable defined it would have preference over the configuration set in the profile.

jsoriano

LGTM! 🌻

mrodm · 2023-09-07T09:06:57Z

buildkite test this

elasticmachine · 2023-09-07T09:43:33Z

💚 Build Succeeded

Buildkite Build
Commit: e90f283

History

💔 Build #1478 failed e90f283
💚 Build #1476 succeeded a24aa47
💚 Build #1475 succeeded acfce7d
💚 Build #1473 succeeded 74b90d2
💔 Build #1472 failed 74b90d2
💔 Build #1471 failed 74b90d2

cc @mrodm

Draft for serverless

1dcbe0a

mrodm self-assigned this Jul 28, 2023

mrodm force-pushed the add_provider branch 2 times, most recently from 601eec1 to d0f9a9b Compare July 28, 2023 15:52

Ensure endpoints are healthy

3f38c52

fleet server endpoint is built based on the elasticsearch endpoint fix creation

mrodm force-pushed the add_provider branch from d0f9a9b to 3f38c52 Compare July 28, 2023 15:57

mrodm added 19 commits August 17, 2023 13:23

Retrieve credentials

4b3da63

Add status output

21ce687

Request credentials when creating project

ef5e47e

Move package to internal

c04ee5d

Add elastic agent to serverless project

f463215

Remove debug outputs

a54a1aa

Create elasticsearch client in stack

7f091eb

Create kibana client in stack

98acfbd

Use kibana client from stack in serverless

3084d0a

Fix kibana client

27fff3c

Use kibana stack client to create agent and package policy

57eb28e

Add method to get default fleet URL to kibana client

81b213c

Move creation of agent policy to project

4e800e3

Fleet should be just checked if project is observability

3302ca2

Use raw http request

54395dc

Use http client for Fleet

bf42d9f

Raise an error in case elasticsearch project is created, since it does not have fleet service.

Merge remote upstream/main into add_provider

7850aab

Fix lint

4c448a8

Retrieve local services from docker daemon directly

8ff731a

mrodm commented Aug 23, 2023

View reviewed changes

internal/stack/serverless.go Outdated Show resolved Hide resolved

mrodm commented Aug 23, 2023

View reviewed changes

Update kibana.CheckHealth comment

30055e1

mrodm added 4 commits September 5, 2023 11:40

Set allowed project types as a list

ff9cd38

Add new profile config to set elastic cloud host URL

57e890f

Added descriptions for new profile configuirations added

Use default host url is env. var is empty

b54e095

Fix condition allowed project type

da1761a

mrodm requested a review from jsoriano September 5, 2023 13:12

mrodm added 5 commits September 5, 2023 16:08

Raise project not exist if any config parameters do not exist

e95d1e5

Merge remote-tracking branch 'upstream/main' into add_provider

8717537

Use same compose file as in compose provider (snapshot.yml)

6bab96d

Retry docker-compose up once if elastic-agent failed

1ad954e

Add project type into status output

74b90d2

mrodm added 2 commits September 6, 2023 17:54

Add example config value

5d582b5

Fix typo in config.yml.example

acfce7d

jsoriano reviewed Sep 6, 2023

View reviewed changes

mrodm and others added 4 commits September 7, 2023 09:24

Apply suggestions from code review

a24aa47

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

Rename variables and make it private

02648f8

Show elastic cloud endpoint always as debug

582bfa9

Ensure preference between config and environment variables for host URL

6757dd3

mrodm requested a review from jsoriano September 7, 2023 08:21

Remove variables

e90f283

mrodm commented Sep 7, 2023

View reviewed changes

jsoriano approved these changes Sep 7, 2023

View reviewed changes

mrodm merged commit 9d80c6c into elastic:main Sep 7, 2023
1 check passed

mrodm deleted the add_provider branch September 7, 2023 10:09

kcreddy mentioned this pull request Sep 18, 2023

[ti_recordedfuture] Add DLM policy and update format_version to 3.0.0 elastic/integrations#7848

Merged

4 tasks

jsoriano mentioned this pull request Apr 30, 2024

[Serverless] Support testing against development serverless projects #1808

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add serverless provider #1374

Add serverless provider #1374

mrodm commented Jul 28, 2023 •

edited

Loading

mrodm commented Aug 16, 2023

elasticmachine commented Aug 16, 2023

mrodm Aug 23, 2023 •

edited

Loading

mrodm commented Sep 6, 2023

mrodm commented Sep 6, 2023

mrodm commented Sep 6, 2023

jsoriano left a comment

jsoriano Sep 6, 2023

mrodm Sep 7, 2023 •

edited

Loading

mrodm Sep 7, 2023

jsoriano left a comment

mrodm commented Sep 7, 2023

elasticmachine commented Sep 7, 2023

		var observedServices = []string{"elasticsearch", elasticAgentService, fleetServerService, "kibana", "package-registry"}

Add serverless provider #1374

Add serverless provider #1374

Conversation

mrodm commented Jul 28, 2023 • edited Loading

How to test

mrodm commented Aug 16, 2023

elasticmachine commented Aug 16, 2023

💚 Build Succeeded

History

mrodm Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

mrodm commented Sep 6, 2023

mrodm commented Sep 6, 2023

mrodm commented Sep 6, 2023

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano Sep 6, 2023

Choose a reason for hiding this comment

mrodm Sep 7, 2023 • edited Loading

Choose a reason for hiding this comment

mrodm Sep 7, 2023

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

mrodm commented Sep 7, 2023

elasticmachine commented Sep 7, 2023

💚 Build Succeeded

History

mrodm commented Jul 28, 2023 •

edited

Loading

mrodm Aug 23, 2023 •

edited

Loading

mrodm Sep 7, 2023 •

edited

Loading