Using CPython Unit Tests for testing

[AlecHaring]

I've been thinking -- what if we used the unit tests from the cpython repo to test Pyarmor. We could make a script that downloads the unit tests, obfuscates them, and then runs them. The cpython tests try to test all Python features, so using them would ensure Pyarmor handles everything correctly.

We could take it a step further and add a GitHub workflow that runs the tests on multiple OSs, versions of Python, and obfuscation modes.

[jondy]

Good idea. That's what old Pyarmor 7.x has been done. But for Pyarmor 8.0, it's still not.

When I did this kind of test in Pyarmor 7.x, the problem is that some tests may crash. Some are bugs of Pyarmor, some are not. Because Pyarmor changes the structure of code object and may broken some tests. Considering the number of test cases, it's time consuming work to ignore these unexpected tests.

Now I'm writing some unit tests for most frequent Python features, and I plan to release 8.1 on April 10, this version will pass my unit tests and It should work for most of Python features.

After that, it's time to do this test, it's really helpful to ensure Pyarmor handles everything correctly.

Thank you pushing Pyarmor better, appreciated your help

[AlecHaring]

Because Pyarmor changes the structure of code object and may broken some tests.

Can you give an example of a test that would break after being obfuscated, but is not a bug of Pyarmor?

When you release the unit tests with 8.1, I'll contribute to them and port tests over from cpython.

My main point with this discussion was to get an understanding of what your goals/visions are regarding unit testing. I was thinking about making unit tests and then creating a PR, but wanted to make sure it was something you'd want. Now that I know you're working on some unit tests, I'll wait for those to be released instead of starting from scratch :)

[jondy]

Here it's super mode test script of Pyarmor 7 to run unit tests in the lib/test

pyarmor/tests/super-test.sh

Line 73 in 2ddf3cc

NOTESTS="test_profilehooks test_sys_setprofile test_sys_settrace test_cprofile

test_inspect.py broken unit tests, because some test cases try to read the source. And in obfuscated scripts, sys._getframe(0) is different from the plain scripts, it will broken test cases which use frame to do some special thing.

The unit tests what I write is to test unopened source, and it can't be run out of my machine.

But there are 2 kinds of tests are in my plan

One is your idea, using Cython unit tests to verify all the python features could work with Pyarmor.

The other is write some test scripts for each Pyarmor feature.

For example, test rft feature, add some test cases in this folder structure

pyarmor/
    v8-tests/
        rft/
            README.md
            testcase1/
                run.sh
                pyarmor_test/
                        main.py
                        class_test.py
                        inner_test/
                              __init__.py
             testcase2/...

These test cases could be used for regression tests, and it also could help other users understand how to use this feature by checking rich comment script v8-tests/rft/testcase1/run.sh

Appreciated any help on both of these tests.