CI and parser scenario testing

Parsers, scenarios and collections published on the hub must be tested whenever possible. This page documents how to understand and write those tests.

hub-tests is a repository hosting the software tool used for testing each configurations hosted in the hub or in its repository. More precisely, each configuration item consists of a yaml configuration file used to describe crowdsec behaviour.

Hub and continuous integration

Each changes in the hub repository, in the hub-tests repository or in the crowdsec repository should fire a continuous integration test.

For now, these changes are carelessly pushed on our github pages

Directory tree

Each config.yaml in the hub repository subdirectories trigger a test. To make life easier, a test is contained is one directory, thus each config.yaml has its own directory. This directory should be named after the configuration we want to test, and lies in a .testsdirectory which itself is alonside the configuration we want to test.

Example: The first written test was for crowdsecurity/sshd-logs. This sshd configuration is ./parsers/s01-parse/crowdsecurity/sshd-logs.yaml, thus the test should takes place in the directory ./parsers/s01-parse/crowdsecurity/.tests/sshd-logs. The test configuration file named config.yamlis in this directory.

configuration file format

parser_input: parser_input.yaml
parser_results: parser_results.yaml
bucket_input: bucket_input.yaml                 #unused in our example
bucket_results: bucket_result.json              #unused in our example
postoverflow_input: po_input.yaml     #unused in our example
postoverflow_results: postoverflow_results.yaml #unused in our example
reprocess: true

#configuration
index: "./config/hub/.index.json"
configurations:      
  parsers:
  - crowdsecurity/sshd-logs
  - crowdsecurity/syslog-logs

File involved for the test

The paths of these of files are defined from the same directory as the config.yaml

• parser_input.yaml: file holding serialized events fed to the parser engine. If this configuration field is empty or missing, hub-tests will look for an acquis.yaml in the test directory, and will follow its directive.
• parser_results.yaml: file holding parser result. The test results will be compared to this file, and fail if it differs. If the file doesn't exist, it is automatically generated (this is useful to create a new test).
• bucket_input.yaml: file holding serialized bucket input. It's automatically generated from parsing. This is a serialized list of events.
• bucket_results.jdon: same as parser_result.json, this file holds the bucket output.
• postoverflow_input.yaml: same as bucket input, this file holds the serialized output of the scenarios. This is a serialized list of events.
• postoverflow_results.json: same as parser result, this file holds the postoverflow result.
• reprocess: make it true if you expect some reprocess to happen. Useful only for buckets test. If scenarios are enables, it enforce postoverflows to pass even if no postoverflow is configured, to be able to repour it in the buckets.

Other part of the configuration

• index: where to find the .index.json from the root directory of the hub repository
• configurations: This is dict of lists of configurations we want to load. Valid keys for the dict are parsers, scenarios and postoverflows.

Caveats

For now a config directory is still needed with:

• simulation.yaml this file can be empty
• patterns directory with all parsers/groks patterns
• config/hub/.index.json I suspect a small bug in pkg/cwversion that makes this requirement happen.