Merge pull request #44 from pablosnt/develop

Prepare release 1.0.1

.github/workflows/unit-testing.yml

@@ -2,6 +2,8 @@ name: Unit testing
 on:
   workflow_dispatch:
   pull_request:
+    paths:
+      - 'rekono/**'
 
 env:
   REQUIRED_COVERAGE: 95

.secrets.baseline

@@ -786,7 +786,7 @@
         "filename": "rekono/testing/executions/test_base_tool.py",
         "hashed_secret": "a94a8fe5ccb19ba61c4c0873d391e987982fbbd3",
         "is_verified": false,
-        "line_number": 295,
+        "line_number": 297,
         "is_secret": false
       }
     ],
@@ -841,5 +841,5 @@
       }
     ]
   },
-  "generated_at": "2022-09-18T13:29:57Z"
+  "generated_at": "2022-09-19T16:59:42Z"
 }

.snyk

@@ -2,16 +2,4 @@
 version: v1.25.0
 # ignores vulnerabilities until expiry date; change duration by modifying expiry date
 ignore:
-  SNYK-PYTHON-PYTHONLIBNMAP-2808864:
-    - '*':
-        reason: >-
-          python-libnmap is only used to parse nmap output. It's not used to
-          execute nmap. No fix available at the moment
-        expires: 2022-09-04T17:10:10.695Z
-        created: 2022-06-04T17:10:10.697Z
-  SNYK-JS-MOMENT-2944238:
-    - '*':
-        reason: No fix available yet
-        expires: 2022-09-07T21:30:04.511Z
-        created: 2022-07-08T21:30:04.520Z
 patch: {}

CHANGELOG.md

@@ -3,6 +3,27 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.0.1] - 2022-09-20
+
+### Fixed
+
+- Retry requests to Defect-Dojo API after unexpected errors (https://github.com/pablosnt/rekono/pull/39)
+- Retry requests to NVD NIST API to avoid blocks by the API rate limit and after unexpected errors (https://github.com/pablosnt/rekono/pull/39)
+- Save unique exploits based on its `reference` instead of `edb_id` (https://github.com/pablosnt/rekono/pull/30)
+- Prevent unexpected errors parsing malformed Sslscan reports (https://github.com/pablosnt/rekono/pull/27)
+
+### Changed
+
+- Optimize calculation of executions from previous findings to make process executions faster (https://github.com/pablosnt/rekono/pull/27)
+- Allow parentheses in text values like names and descriptions (https://github.com/pablosnt/rekono/pull/29)
+
+### Security
+
+- Bump `nginx` Docker image version to `1.22-alpine` (https://github.com/pablosnt/rekono/pull/25/files)
+- Bump `node` Docker image version to `18.6.0-alpine` (https://github.com/pablosnt/rekono/pull/25/files)
+- Bump `python-libnmap` version to `0.7.3` (https://github.com/pablosnt/rekono/pull/31)
+
+
 ## [1.0.0] - 2022-08-19
 
 ### Added

CONTRIBUTING.md

@@ -9,7 +9,7 @@ Please, don't report security vulnerabilities in GitHub Issues. See our [Securit
 
 ## Contributing to Rekono
 
-You can create Pull Requests to the `develop` branch of this project. All the Pull Requests should be reviewed and approved before been merged. After that, your code will be included on the next Rekono release.
+**You can create Pull Requests to the `develop` branch of this project**. All the Pull Requests should be reviewed and approved before been merged. After that, your code will be included on the next Rekono release.
 
 In this section you can see how to achieve that and the things that you should to take into account.
 
@@ -30,7 +30,7 @@ coverage run manage.py test
 
 ### Add support for a new hacking tool
 
-The support of external hacking tools in Rekono is based on two steps:
+The support of external hacking tools in Rekono is based on the following steps:
 
 1. Define the hacking tools in the [tools/fixture](https://github.com/pablosnt/rekono/tree/main/rekono/tools/fixtures) files. There are one file for each required entity:
 
@@ -54,6 +54,14 @@ The support of external hacking tools in Rekono is based on two steps:
 
     - Override the method `parse_output_file` or `parse_plain_output` depending on the tool output type.
 
+3. Implement [unit tests](https://github.com/pablosnt/rekono/tree/main/rekono/testing/tools) to check the parser correct working. You can add your [tool reports](https://github.com/pablosnt/rekono/tree/main/rekono/testing/data/reports) as example for that.
+
+4. Add the tool reference in the [README.md](https://github.com/pablosnt/rekono#supported-tools).
+
+5. Add tool installation to the [Kali Linux Dockerfile](https://github.com/pablosnt/rekono/blob/main/docker/kali/Dockerfile).
+
+6. Add tool installation to the [Rekono CLI](https://github.com/pablosnt/rekono-cli/blob/main/rekono/installation/tools.py).
+
 ### CI/CD
 
 This project has the following checks in _Continuous Integration_:
@@ -79,5 +87,5 @@ pre-commit install
 There are some guidelines to keep the code clean and ensure the correct working of the application:
 
 - Comment your code, specially to document the classes and methods.
-- Make unit tests of all your code to ensure its correct working. It's important to keep the testing coverage over a 95% coverage.
+- Make unit tests for all your code to ensure its correct working. It's important to keep the testing coverage over a 95% coverage.
 - Don't include code vulnerabilities or vulnerable libraries.

docker-compose.yml

@@ -78,7 +78,7 @@ services:
     networks:
       - internal
     depends_on:
-      - postgres
+      - initialize
       - redis
     environment:
       - RKN_DB_HOST=postgres
@@ -105,7 +105,7 @@ services:
     cap_add:
       - NET_ADMIN
     depends_on:
-      - postgres
+      - initialize
       - redis
     environment:
       - RKN_DB_HOST=postgres
@@ -127,7 +127,7 @@ services:
       - internal
       - external
     depends_on:
-      - postgres
+      - initialize
       - redis
     environment:
       - RKN_DB_HOST=postgres
@@ -152,6 +152,7 @@ services:
       - internal
       - external
     depends_on:
+      - initialize
       - redis
       - postfix
     environment:
@@ -180,7 +181,7 @@ services:
       - internal
       - external
     depends_on:
-      - postgres
+      - initialize
       - redis
     environment:
       - RKN_DB_HOST=postgres
@@ -214,7 +215,7 @@ services:
       - internal
       - external
     depends_on:
-      - postgres
+      - initialize
       - redis
     environment:
       - RKN_DB_HOST=postgres

docker/nginx/Dockerfile

@@ -1,4 +1,4 @@
-FROM nginx:1.21.6-alpine
+FROM nginx:1.22-alpine
 
 COPY docker/nginx/nginx.conf /etc/nginx/nginx.conf
 

docker/rekono/Dockerfile

@@ -25,7 +25,7 @@ WORKDIR /code
 ENV REKONO_HOME /rekono
 
 
-FROM node:17.6.0-alpine as frontend
+FROM node:18.6.0-alpine as frontend
 
 RUN mkdir /frontend
 COPY --from=backend /code/frontend /frontend

rekono/defectdojo/api.py

@@ -1,13 +1,15 @@
 import logging
 from datetime import datetime, timedelta
 from typing import Any, Tuple
+from urllib.parse import urlparse
 
 import requests
-from defectdojo.constants import DD_DATE_FORMAT, DD_DATETIME_FORMAT
 from findings.enums import Severity
 from projects.models import Project
-
 from rekono.settings import DEFECT_DOJO as config
+from requests.adapters import HTTPAdapter, Retry
+
+from defectdojo.constants import DD_DATE_FORMAT, DD_DATETIME_FORMAT
 
 # Mapping between Rekono and Defect-Dojo severities
 SEVERITY_MAPPING = {
@@ -33,6 +35,11 @@ def __init__(self):
         self.product_type = config.get('PRODUCT_TYPE')                          # Product type name for Rekono
         self.test_type = config.get('TEST_TYPE')                                # Test type name for Rekono
         self.test = config.get('TEST')                                          # Test name for Rekono
+        schema = urlparse(self.url).scheme                                      # Get API schema
+        self.http_session = requests.Session()                                  # Create HTTP session
+        # Configure retry protocol to prevent unexpected errors
+        retries = Retry(total=10, backoff_factor=0.1, status_forcelist=[500, 502, 503, 504, 599])
+        self.http_session.mount(f'{schema}://', HTTPAdapter(max_retries=retries))
 
     def request(
         self,
@@ -60,15 +67,26 @@ def request(
             'User-Agent': 'Rekono',                                             # Rekono User-Agent
             'Authorization': f'Token {self.api_key}'                            # Authentication via API key
         }
-        response = requests.request(                                            # Defect-Dojo API request
-            method=method,
-            url=f'{self.url}{endpoint}',
-            headers=headers,
-            params=params,
-            data=data,
-            files=files,
-            verify=self.verify_tls
-        )
+        try:
+            response = self.http_session.request(                               # Defect-Dojo API request
+                method=method,
+                url=f'{self.url}{endpoint}',
+                headers=headers,
+                params=params,
+                data=data,
+                files=files,
+                verify=self.verify_tls
+            )
+        except requests.exceptions.ConnectionError:
+            response = self.http_session.request(                               # Defect-Dojo API request
+                method=method,
+                url=f'{self.url}{endpoint}',
+                headers=headers,
+                params=params,
+                data=data,
+                files=files,
+                verify=self.verify_tls
+            )
         logger.info(f'[Defect-Dojo] {method.upper()} /api/v2{endpoint} > HTTP {response.status_code}')
         if response.status_code == expected_status:
             return True, response.json()                                        # Successful request