Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for benchmark test from forks #1345

Merged
merged 1 commit into from
Nov 11, 2024
Merged

Add support for benchmark test from forks #1345

merged 1 commit into from
Nov 11, 2024

Conversation

YigitElma
Copy link
Collaborator

@YigitElma YigitElma commented Nov 4, 2024

Benchmark action requires write access for commenting which forks don't have. This change makes the benchmark workflow coming from a fork run on our CI and thus have proper access.

This shouldn't change the general permissions for fork repos, so someone in the DESC team must still approve to test to run for such PRs.

https://github.com/thollander/actions-comment-pull-request?tab=readme-ov-file#permissions

  • Make sure this is secure! Check from a fork repo!

Copy link
Contributor

github-actions bot commented Nov 4, 2024

|             benchmark_name             |         dt(%)          |         dt(s)          |        t_new(s)        |        t_old(s)        | 
| -------------------------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- |
 test_build_transform_fft_midres         |     -4.19 +/- 5.76     | -2.66e-02 +/- 3.66e-02 |  6.08e-01 +/- 2.4e-02  |  6.35e-01 +/- 2.8e-02  |
 test_build_transform_fft_highres        |     -0.19 +/- 2.33     | -1.86e-03 +/- 2.33e-02 |  1.00e+00 +/- 1.2e-02  |  1.00e+00 +/- 2.0e-02  |
 test_equilibrium_init_lowres            |     +2.14 +/- 5.81     | +8.23e-02 +/- 2.23e-01 |  3.93e+00 +/- 1.6e-01  |  3.85e+00 +/- 1.5e-01  |
 test_objective_compile_atf              |     +2.59 +/- 5.59     | +2.05e-01 +/- 4.44e-01 |  8.14e+00 +/- 3.1e-01  |  7.94e+00 +/- 3.1e-01  |
 test_objective_compute_atf              |     +3.27 +/- 4.18     | +3.52e-04 +/- 4.50e-04 |  1.11e-02 +/- 3.3e-04  |  1.08e-02 +/- 3.0e-04  |
 test_objective_jac_atf                  |     +2.46 +/- 3.23     | +4.76e-02 +/- 6.26e-02 |  1.99e+00 +/- 4.7e-02  |  1.94e+00 +/- 4.2e-02  |
 test_perturb_1                          |     +5.91 +/- 2.82     | +7.59e-01 +/- 3.62e-01 |  1.36e+01 +/- 1.4e-01  |  1.28e+01 +/- 3.3e-01  |
 test_proximal_jac_atf                   |     +1.30 +/- 1.40     | +1.07e-01 +/- 1.16e-01 |  8.33e+00 +/- 6.7e-02  |  8.23e+00 +/- 9.4e-02  |
 test_proximal_freeb_compute             |     -0.98 +/- 0.94     | -1.80e-03 +/- 1.72e-03 |  1.82e-01 +/- 1.5e-03  |  1.84e-01 +/- 8.6e-04  |
 test_build_transform_fft_lowres         |     +0.47 +/- 2.15     | +2.42e-03 +/- 1.10e-02 |  5.14e-01 +/- 6.9e-03  |  5.12e-01 +/- 8.5e-03  |
 test_equilibrium_init_medres            |     +0.52 +/- 0.74     | +2.10e-02 +/- 2.97e-02 |  4.05e+00 +/- 2.2e-02  |  4.03e+00 +/- 1.9e-02  |
 test_equilibrium_init_highres           |     +0.36 +/- 0.63     | +1.92e-02 +/- 3.40e-02 |  5.38e+00 +/- 2.9e-02  |  5.36e+00 +/- 1.8e-02  |
 test_objective_compile_dshape_current   |     +0.13 +/- 0.97     | +4.98e-03 +/- 3.68e-02 |  3.81e+00 +/- 2.8e-02  |  3.80e+00 +/- 2.4e-02  |
 test_objective_compute_dshape_current   |     +1.56 +/- 1.70     | +5.57e-05 +/- 6.06e-05 |  3.63e-03 +/- 5.2e-05  |  3.58e-03 +/- 3.2e-05  |
 test_objective_jac_dshape_current       |     +2.76 +/- 6.35     | +1.09e-03 +/- 2.51e-03 |  4.07e-02 +/- 2.0e-03  |  3.96e-02 +/- 1.5e-03  |
 test_perturb_2                          |     +0.13 +/- 0.66     | +2.24e-02 +/- 1.13e-01 |  1.73e+01 +/- 5.2e-02  |  1.73e+01 +/- 1.0e-01  |
 test_proximal_freeb_jac                 |     +0.68 +/- 1.67     | +4.96e-02 +/- 1.22e-01 |  7.38e+00 +/- 8.2e-02  |  7.33e+00 +/- 9.1e-02  |
 test_solve_fixed_iter                   |     -0.25 +/- 54.25    | -1.24e-02 +/- 2.70e+00 |  4.96e+00 +/- 1.9e+00  |  4.98e+00 +/- 1.9e+00  |

@YigitElma YigitElma self-assigned this Nov 5, 2024
@sinaatalay sinaatalay removed their request for review November 6, 2024 18:31
@YigitElma
Copy link
Collaborator Author

Can someone merge this @f0uriest @dpanici ?

@dpanici dpanici merged commit 9d1d730 into master Nov 11, 2024
24 checks passed
f0uriest added a commit that referenced this pull request Nov 12, 2024
For some reason, changes from #1345 try to initialize 2 separate
benchmarks for a single PR and cancel the second one with `Canceling
since a higher priority waiting request for 'Benchmarks-1344' exists`.
The second one seems to run properly but this is still annoying.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
override codecov Override codecov
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants