Codacy code coverage reporting #443

kvrigor · 2024-10-02T16:21:37Z

Continuation of adding test coverage support which was initially started in #405.

kvrigor · 2024-10-03T07:48:16Z

Codacy integration seems working. Link to report: https://app.codacy.com/gh/pyccel/psydac/coverage?branch=codacy-codecov

codacy-production · 2024-10-04T10:21:07Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
Report missing for `5ff3884`¹	✅ ∅

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`5ff3884`)	Report Missing	Report Missing	Report Missing
Head commit (`a443720`)	61320	36792	60.00%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#443)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct. ↩

kvrigor · 2024-10-04T13:55:38Z

Detailed coverage results are also shown in CI: https://github.com/pyccel/psydac/actions/runs/11180714225/job/31083050038?pr=443#step:22:20

yguclu · 2024-10-07T14:54:52Z

The single-process tests on Ubuntu with Python 3.12 are taking almost one hour and a half to run. This does not happen with the other jobs. Do you see a reason for this?

See https://coverage.readthedocs.io/en/7.6.3/subprocess.html#measuring-sub-processes

kvrigor · 2024-10-14T15:20:32Z

The single-process tests on Ubuntu with Python 3.12 are taking almost one hour and a half to run. This does not happen with the other jobs. Do you see a reason for this?

~~This could be an issue in coverage. See nedbat/coveragepy#1862~~

EDIT 12-Nov: Root cause of slowdown due to changes in Python 3.11. See python/cpython#93516

Overhead due to coverage data collection is the same for coverage.py and pytest-cov

pytest_cache should be saved whenever its contents changes. Cache saving only triggers if the associated cache key changes; thus the cache key should automatically update based on the contents of pytest_cache, which is uniquely represented by its hash.

All single process tests will still be run regardless of test order; thus we don't really reduce test time.

kvrigor · 2024-10-24T16:36:24Z

@campospinto's suggestion is neat! coverage now runs only on macOS and the Ubuntu test time is back to normal. This PR is ready for review.

yguclu

Good job! I have some questions below.

By the way, I thought you wanted to use Pytest's cache to run the failed tests first. Have you abandoned the idea?

pyproject.toml

yguclu · 2024-10-24T16:52:50Z

pyproject.toml

+
+    # Don't complain if non-runnable code isn't run:
+    "if 0:",
+    "if __name__ == .__main__.:",


I find the use of the dots . a bit confusing in this regex. You actually want to match just the single and double quote marks, right?

You actually want to match just the single and double quote marks, right?

Exactly; this snippet was based from here: https://coverage.readthedocs.io/en/latest/excluding.html#advanced-exclusion

…oml, changed if 0 -: if False

kvrigor added 9 commits April 24, 2024 08:41

Added Codecov to CI

3c1ad9b

Merge branch 'pyccel:devel' into codecov-test

f4fff9e

Merge branch 'pyccel:devel' into codecov-test

eecc6f4

Merge branch 'pyccel:devel' into codecov-test

0d2675b

coverage: exlude unit tests and pyccelised kernels in the report

cd78ac3

CI.yml: added Codacy coverage reporter action

1bafc85

pyproject.toml: deleted redundant pytest-cov

1715bfd

CI.yml: Codacy reporter should run under pytest folder

ae33ed0

CI.yml: Fixed path to coverage.xml

ab102b9

CI.yml: Set coverage config file to pyproject.toml

03f605d

kvrigor changed the title ~~Codacy codecov~~ Codacy code coverage reporting Oct 4, 2024

kvrigor added 2 commits October 4, 2024 15:16

CI.yml: Printed detailed coverage results

07aa1e1

Excluded examples dir from coverage

65c27de

kvrigor marked this pull request as ready for review October 4, 2024 14:19

kvrigor requested a review from yguclu October 4, 2024 14:19

'--cov-report term-missing' is redundant

4949d39

kvrigor added 10 commits October 8, 2024 07:44

Dummy change for checking test runtime

36620af

CI.yml: Generated coverage report on another step

3703d5b

CI.yml: Run tests via coverage instead of pytest-cov

f175d0b

CI.yml: Specified full path to pyproject.toml

10d96d4

CI.yml: Be explicit about the location of the .coverage file

fc6e380

CI.yml: Ran 'coverage xml' under ./pytest

d2a73f1

CI.yml: Run coverage at repo root

e23716e

Fixed missing coverage data by setting coverage subprocess hooks

6d4aedd

See https://coverage.readthedocs.io/en/7.6.3/subprocess.html#measuring-sub-processes

Fixed typo when unsetting coverage env vars

d995440

Made coverage env settings permanent for now

338aa9a

kvrigor marked this pull request as draft October 14, 2024 14:32

kvrigor added 14 commits October 14, 2024 17:25

Try running coverage under ./pytest

ee41b76

Revert to pytest-cov

934a705

Overhead due to coverage data collection is the same for coverage.py and pytest-cov

Fixed path to coverage.xml

b7d61a5

Cached ./pytest_cache

43b35e3

Pytest cache must always be saved regardless of test result

051f1ec

Only run coverage tests on macOS

02647ba

Break test_api_3d_scalar

cf024e8

Pytest cache must be updated every run

70eeff4

Fixed test_api_3d_scalar.py

65742fd

Break test_2d_biharmonic.py

6d28291

Break test_2d_laplace.py

06cb2f8

Discarded --failed-first idea

85f869d

All single process tests will still be run regardless of test order; thus we don't really reduce test time.

Fixed deliberately broken tests

5567cf7

kvrigor marked this pull request as ready for review October 24, 2024 16:29

yguclu requested changes Oct 24, 2024

View reviewed changes

Declared all pytest packages as optional dependencies; in pyproject.t…

a443720

…oml, changed if 0 -: if False

kvrigor requested a review from yguclu November 6, 2024 11:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Codacy code coverage reporting #443

Codacy code coverage reporting #443

kvrigor commented Oct 2, 2024

kvrigor commented Oct 3, 2024

codacy-production bot commented Oct 4, 2024 •

edited

Loading

kvrigor commented Oct 4, 2024

yguclu commented Oct 7, 2024 •

edited

Loading

kvrigor commented Oct 14, 2024 •

edited

Loading

kvrigor commented Oct 24, 2024

yguclu left a comment

yguclu Oct 24, 2024

kvrigor Oct 28, 2024

Codacy code coverage reporting #443

Are you sure you want to change the base?

Codacy code coverage reporting #443

Conversation

kvrigor commented Oct 2, 2024

kvrigor commented Oct 3, 2024

codacy-production bot commented Oct 4, 2024 • edited Loading

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

Footnotes

kvrigor commented Oct 4, 2024

yguclu commented Oct 7, 2024 • edited Loading

kvrigor commented Oct 14, 2024 • edited Loading

kvrigor commented Oct 24, 2024

yguclu left a comment

Choose a reason for hiding this comment

yguclu Oct 24, 2024

Choose a reason for hiding this comment

kvrigor Oct 28, 2024

Choose a reason for hiding this comment

codacy-production bot commented Oct 4, 2024 •

edited

Loading

yguclu commented Oct 7, 2024 •

edited

Loading

kvrigor commented Oct 14, 2024 •

edited

Loading