-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding regression tests #195
base: dev
Are you sure you want to change the base?
Conversation
Looks like the files are not perfectly reproducible and I'm seeing lines in the regression test not matching the lines in the CI produced files Errors like this are appearing, where the coefficients for the surfaces don't fully match but are very close as mentioned in this issue
this lack of reproducibility makes regression tests difficult so perhaps we should see if it is possible to make the files fully reproducible prior to the regression testing. Or alternatively we can introduce some tolerance on numbers so they don't have to match perfectly Any comments on the preferred direction would be appreciated |
I've added some extra code to the tests that will split each line up and check each token matches. if the token is a float then there is some tolerance on the checking (1e-6) |
Description
I've produced CSG files for all the step files in the repo and added these to a folder in the tests folder. Now whenever we make a PR a regression test will run that compares all the CSG files converted during the existing CI tests with those in the regression test folder. This takes less than a second so we won't notice any CI time increase.
One of the common pains of regression tests is that from time to time we will find a bug and fix it which changes the CSG files in a good way. However this will fail the regression tests as the test just checks if the newly produced CSG files match the older files.
To help smooth out this annoyance I've include a script
tests/update_regression_test_files.py
that will remake all the regression test files.Most of the files added in this PR are the regression test CSG files (208 files) however to help with the review the important files edited in this PR are
Note that while I was making this I had to do some messy Path manipulation to ensure each stp file in the testing/inputSTEP folder has its own folder with no naming overlaps. One thing I would like to do in the future is to rename the files in the testing/inputSTEP folder in a better way so that we don't have files with the same name even when they are in different subfolders. Also if I had the ability to specify different file names stems for the CSG files then that would also make the Paths more elegant and concise (which makes me keen on sperate export_csg methods for each MC code). So please excuse the long Path operations as this is something I can clean up in a following PR and I want to keep this one on topic.
Fixes issue
helps with issue #46
Checklist