Merge pull request #166 from LUMC/release_2.0.0

Release 2.0.0

.github/workflows/ci.yml

@@ -14,7 +14,7 @@ jobs:
     strategy:
       matrix:
         python-version:
-          - "3.6"
+          - "3.7"
     steps:
       - uses: actions/checkout@v2.3.4
       - name: Set up Python ${{ matrix.python-version }}
@@ -26,7 +26,6 @@ jobs:
       - name: Lint
         run: tox -e lint
   docs:
-    needs: lint
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2.3.4
@@ -43,12 +42,11 @@ jobs:
     strategy:
       matrix:
         python-version:
-          - "3.6"
           - "3.7"
           - "3.8"
           - "3.9"
           - "3.10"
-    needs: lint
+          - "3.11"
     steps:
       - uses: actions/checkout@v2.3.4
       - name: Set up Python ${{ matrix.python-version }}
@@ -60,7 +58,6 @@ jobs:
       - name: Run tests
         run: tox -e py3
       - name: Upload coverage report
-        if: ${{ matrix.python-version == 3.6 }} # Only upload coverage once
         uses: codecov/codecov-action@v1
 
   test-functional:

HISTORY.rst

@@ -7,6 +7,45 @@ Changelog
 .. This document is user facing. Please word the changes in such a way
 .. that users understand how the changes affect the new version.
 
+version 2.0.0
+---------------------------
+This major release greatly cleans up the output of pytest-workflow in case of
+an error as well as providing the stderr and stdout last bytes for debugging
+purposes. When the exit code test fails all other tests from the workflow
+are skipped.
+
++ Python 3.6 is no longer supported. It has been removed from github actions,
+  as such we can no longer guarantee that pytest-workflow works properly
+  with python 3.6.
++ Fix an issue where symlinks in git repositories could not be properly copied.
++ Added an optional encoding key for files, stdout and stderr so the file can
+  be opened with the proper encoding.
++ Make content tests more efficient by reading each file only once instead of
+  twice when there are both strings and regexes to check for.
++ When the ``--git-aware`` flag is used a submodule check is performed in order
+  to assert that all submodules are properly checked out. This prevents
+  unclear copying errors.
++ Tests are now skipped if the workflow does not exit with the desired exit
+  code, except for the exit code tests, to reduce visual clutter when reporting
+  failing tests.
++ Tests for checking file content are now skipped when the file does not exist
+  in order to reduce visual clutter when reporting failing tests.
++ Test and support for Python 3.11.
++ Add ``--stderr-bytes`` or ``--sb`` option to change the maximum 
+  number of bytes to display for the stderr and stdout on 
+  command failure. 
++ Add stderr and stdout to be displayed on command failure
++ Document using ``pytest.ini`` as a way of setting specific per repository
+  settings for pytest-workflow.
++ Add tests for nextflow.
++ Document the use of environment variables with pytest-workflow.
++ A minimum of pytest 7.0.0 is now a requirement for pytest-workflow.
+  This fixes the deprecation warnings that started on the release of pytest
+  7.0.0.
++ Throw a more descriptive error when a file copied with the --git-aware flag
+  is not present on the filesystem anymore.
++ Document pytest flags ``-k`` and ``-m`` are not supported.
+
 version 1.6.0
 ---------------------------
 + Add a ``--git-aware`` or ``--ga`` option to only copy copy files listed by

README.rst

@@ -34,18 +34,18 @@ pytest-workflow
 
 pytest-workflow is a workflow-system agnostic testing framework that aims 
 to make pipeline/workflow testing easy by using YAML files for the test 
-configuration. Whether you write your pipelines in WDL, snakemake, bash or
-any other workflow framework, pytest-workflow makes testing easy.
+configuration. Whether you write your pipelines in WDL, snakemake, nextflow, 
+bash or any other workflow framework, pytest-workflow makes testing easy.
 pytest-workflow is build on top of the pytest test framework.
 
-For our complete documentation checkout our
+For our complete documentation and examples checkout our
 `readthedocs page <https://pytest-workflow.readthedocs.io/>`_.
 
 
 Installation
 ============
-Pytest-workflow requires Python 3.6 or higher. It is tested on Python 3.6, 3.7,
-3.8, 3.9 and 3.10. Python 2 is not supported.
+Pytest-workflow requires Python 3.7 or higher. It is tested on Python 3.7,
+3.8, 3.9, 3.10 and 3.11.
 
 - Make sure your virtual environment is activated.
 - Install using pip ``pip install pytest-workflow``
@@ -103,6 +103,7 @@ predefined tests as well as custom tests are possible.
         must_not_contain:              # A list of strings that should NOT be in the file (optional)
           - "Cock a doodle doo"
         md5sum: e583af1f8b00b53cda87ae9ead880224   # Md5sum of the file (optional)
+        encoding: UTF-8                # Encoding for the text file (optional). Defaults to system locale.
 
   - name: simple echo                  # A second workflow. Notice the starting `-` which means
     command: "echo moo"                # that workflow items are in a list. You can add as much workflows as you want
@@ -114,6 +115,7 @@ predefined tests as well as custom tests are possible.
         - "moo"
       must_not_contain:                # List of strings that should NOT be in stout (optional)
         - "Cock a doodle doo"
+      encoding: ASCII                  # Encoding for stdout (optional). Defaults to system locale.
 
   - name: mission impossible           # Also failing workflows can be tested
     tags:                              # A list of tags that can be used to select which test
@@ -130,6 +132,7 @@ predefined tests as well as custom tests are possible.
         - "BSOD error, please contact the IT crowd"
       must_not_contain:                # A list of strings which should NOT be in stderr (optional)
         - "Mission accomplished!"
+      encoding: UTF-16                 # Encoding for stderr (optional). Defaults to system locale.
 
   - name: regex tests
     command: echo Hello, world
@@ -143,7 +146,7 @@ predefined tests as well as custom tests are possible.
         - '^Hello .*'                  # Complex regexes will break yaml if double quotes are used
 
 For more information on how Python parses regular expressions, see the `Python
-documentation <https://docs.python.org/3.6/library/re.html>`_.
+documentation <https://docs.python.org/3/library/re.html>`_.
 
 Documentation for more advanced use cases including the custom tests can be
 found on our `readthedocs page <https://pytest-workflow.readthedocs.io/>`_.

docs/conf.py

@@ -64,7 +64,7 @@
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = "en"
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.

docs/examples.rst

@@ -79,14 +79,12 @@ information on the localization of output files as well as options to modify
 the running of miniwdl from the environment.
 
 Miniwdl will localize all the output files to an ``output_links`` directory
-inside the test output directory. If you have a workflow with the output:
+inside the test output directory. If you have a workflow with the output::
 
-.. code-block::
-
-        output {
-            File moo_file = moo_task.out
-            Array[File] stats = moo_task.stats_files
-        }
+    output {
+        File moo_file = moo_task.out
+        Array[File] stats = moo_task.stats_files
+    }
 
 Inside the ``out`` directory the directories ``moo_file`` and
 ``stats`` will be created. Inside these directories will be the produced files.
@@ -109,3 +107,96 @@ The following yaml file tests a WDL pipeline run with miniwdl.
 
 Please note that the trailing slash in ``-d test-output/`` is important. It
 will ensure the files end up in the ``test-output`` directory.
+
+Nextflow example
+-----------------
+
+With nextflow each process is run in a unique directory where the output files will
+also be stored. Nextflow can output a copy of the output files to a separate workflow-outputs 
+directory. This can be achieved by defining a ``publishDir`` in the process. Through ``params.outdir``
+it is possible to define the output directory when running the code.
+
+An example code defining a ``publishDir`` is listed below. ::
+
+    process Hello {
+        publishDir = [
+            path: { "${params.outdir}/hello"}
+        ]
+
+        output:
+        path "HelloWorld.txt"
+        script:
+        """
+        echo "Hello World!" > HelloWorld.txt
+        """
+    }
+
+    workflow {
+        Hello
+    }
+
+To run the code listed above the following command can be used in which ``examplecode.nf`` is the code listed above:
+
+.. code-block:: bash
+
+    nextflow run examplecode.nf --outdir test-output
+
+``publishDir`` will make it so that all the output files of the process are copied to the given directory. 
+``--outdir`` is used to define the path the output files will go to. In this case ``HelloWorld.txt`` will 
+be copied to the  directory called ``test-output/hello``.
+
+An example yaml file that could be used to test the nextflow pipeline from ``examplecode.nf`` is listed
+below.
+
+.. code-block:: yaml
+
+    - name: My pipeline
+      command: nextflow run examplecode.nf --outdir test-output
+      files:
+        - path: "test-output/hello/HelloWorld.txt"
+
+Bash example
+------------
+
+The following is an example of a Bash file that can run directly as a script, or sourced to test each function separately:
+
+.. code-block:: bash
+
+    #!/usr/bin/env bash
+
+    function say_hello() {
+        local name="$1"
+        echo "Hello, ${name}!"
+    }
+
+    function main() {
+        say_hello world
+    }
+
+    # Only execute main when this file is run as a script
+    if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
+        main
+    fi
+
+Save the bash file as ``script.sh``, and test it with the following pytest-workflow configuration:
+
+
+.. code-block:: yaml
+
+    - name: test bash script
+      command: bash script.sh
+      stdout:
+        contains:
+          - "Hello, world!"
+
+    - name: test bash function
+      command: >
+        bash -c "
+        source script.sh;
+        say_hello pytest-workflow
+        "
+      stdout:
+        contains:
+          - "Hello, pytest-workflow!"
+        must_not_contain:
+          - "Hello, world!"

docs/installation.rst

@@ -2,8 +2,7 @@
 Installation
 ============
 
-Pytest-workflow is tested on python 3.6, 3.7, 3.8, 3.9 and 3.10. Python 2 is not
-supported.
+Pytest-workflow is tested on python 3.7, 3.8, 3.9, 3.10 and 3.11.
 
 In a virtual environment
 ------------------------

docs/known_issues.rst

@@ -21,4 +21,4 @@ Known issues
   ``contains_regex`` and ``must_not_contain_regex``, since this collides with
   Python's usage of the same character to escape special characters in strings.
   Please see the `Python documentation on regular expressions
-  <https://docs.python.org/3.6/library/re.html>`_ for details.
+  <https://docs.python.org/3/library/re.html>`_ for details.

docs/running_pytest_workflow.rst

@@ -26,6 +26,20 @@ Specific pytest options for pytest workflow
     :func: __pytest_workflow_cli
     :prog: pytest
 
+Setting specific per-project settings using pytest.ini
+-----------------------------------------------------------------
+pytest can be configured `using a pytest.ini file
+<https://docs.pytest.org/en/7.1.x/reference/customize.html#pytest-ini>`_.
+This mechanic can be used to set specific settings in each repository
+where pytest workflow is used.
+
+For example a pytest.ini with the following contents::
+
+    [pytest]
+    addopts = --git-aware
+
+Can be used in a git repository with a workflow.
+
 Temporary directory cleanup and creation
 ----------------------------------------
 
@@ -115,3 +129,9 @@ Internally names and tags are handled the same so if the following tests:
       - hello
 
 are run with ``pytest --tag hello`` then both ``hello`` and ``hello2`` are run.
+
+.. note::
+
+    The pytest flags ``-k`` and ``-m`` are not supported by pytest-workflow.
+    Rational for this design decision can be `found within GitHub issue #155
+    <https://github.com/LUMC/pytest-workflow/issues/155#issuecomment-1334911457>`_.

docs/writing_tests.rst

@@ -40,6 +40,7 @@ Test options
         must_not_contain:              # A list of strings that should NOT be in the file (optional)
           - "Cock a doodle doo"
         md5sum: e583af1f8b00b53cda87ae9ead880224   # Md5sum of the file (optional)
+        encoding: UTF-8                # Encoding for the text file (optional). Defaults to system locale.
 
   - name: simple echo                  # A second workflow. Notice the starting `-` which means
     command: "echo moo"                # that workflow items are in a list. You can add as much workflows as you want
@@ -51,6 +52,7 @@ Test options
         - "moo"
       must_not_contain:                # List of strings that should NOT be in stout (optional)
         - "Cock a doodle doo"
+      encoding: ASCII                  # Encoding for stdout (optional). Defaults to system locale.
 
   - name: mission impossible           # Also failing workflows can be tested
     tags:                              # A list of tags that can be used to select which test
@@ -60,13 +62,14 @@ Test options
     files:
       - path: "fail.log"               # Multiple files can be tested for each workflow
       - path: "TomCruise.txt.gz"       # Gzipped files can also be searched, provided their extension is '.gz'
-        contains: 
+        contains:
           - "starring"
     stderr:                            # Options for testing stderr (optional)
       contains:                        # A list of strings which should be in stderr (optional)
         - "BSOD error, please contact the IT crowd"
       must_not_contain:                # A list of strings which should NOT be in stderr (optional)
         - "Mission accomplished!"
+      encoding: UTF-16                 # Encoding for stderr (optional). Defaults to system locale.
 
   - name: regex tests
     command: echo Hello, world
@@ -83,13 +86,40 @@ Test options
 The above YAML file contains all the possible options for a workflow test.
 
 Please see the `Python documentation on regular expressions
-<https://docs.python.org/3.6/library/re.html>`_ to see how Python handles escape
+<https://docs.python.org/3/library/re.html>`_ to see how Python handles escape
 sequences.
 
 .. note::
     Workflow names must be unique. Pytest workflow will crash when multiple
     workflows have the same name, even if they are in different files.
 
+Environment variables
+----------------------
+Pytest-workflow runs tests in the same environment as in which the pytest
+executable was started. This means programs started in tests can use
+environnment variables. However, environment variables inside the command
+section itself are quoted by pytest-workflow using
+`shlex.quote <https://docs.python.org/3/library/shlex.html#shlex.quote>`_.
+See the examples below:
+
+.. code-block:: YAML
+
+    - name: Try to use an environment variable
+      command: echo $MY_VAR
+      # Output will be literally "$MY_VAR"
+
+    - name: Circumenvent shlex quoting by explicitly starting the command in a shell.
+      command: bash -c 'echo $MY_VAR'
+      # Output will be the content of $MY_VAR
+
+    - name: Use a program that checks an environment variable
+      command: singularity run my_container.sif
+      # Correctly uses "SINGULARITY_" prefixed variables
+
+If you want to use shell scripting features such as environment
+variables inside ``command``, you need to explicitly set the shell as shown
+above.
+
 Writing custom tests
 --------------------
 

requirements.txt

@@ -1,3 +1,3 @@
 pyyaml
-pytest>=5.4.0
+pytest>=7.0.0
 jsonschema