Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Lumi CCE17 (17.0.1 with rocm 6.0.3) toolchain and env file #97

Merged
merged 1 commit into from
Nov 4, 2024
Merged

Add Lumi CCE17 (17.0.1 with rocm 6.0.3) toolchain and env file #97

merged 1 commit into from
Nov 4, 2024

Conversation

MichaelSt98
Copy link
Contributor

Attention: no support for serialbox.
Serialbox currently relies on optionally implemented signed integer types (e.g. int32_t). However, Cray doesn't implement those (or at least not all required ones).

Build on Lumi with:

./cloudsc-bundle build --clean --build-dir=build-dev-cce-17-pr --arch=arch/eurohpc/lumi/cray-gpu/17.0.1 --with-hip --with-gpu

Execute:

./bin/dwarf-cloudsc-hip-hoist 1 262144 64
     NUMOMP=1, NGPTOT=262144, NPROMA=64, NGPBLKS=4096
     NUMOMP    NGPTOT  #GP-cols     #BLKS    NPROMA tid# : Time(msec)  MFlops/s     col/s
          1    262144    262144         0        64    0 :        103    316962   2539290 @ core#
          1    262144    262144      4096        64   -1:       1410     23198    185852 TOTAL
             Variable Dim             MinValue             MaxValue            AbsMaxErr         AvgAbsErr/GP          MaxRelErr-%
                PLUDE 2D1  0.0000000000000e+00  1.0267201089823e-04  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00      
             PCOVPTOT 2D3  0.0000000000000e+00  1.0000000000000e+00  2.2204460492503e-15  7.7723351487316e-16  2.9166294321752e-15      
     PRAINFRAC_TOPRFZ 2D1  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00      
               PFSQLF 2D3 -2.5817563600963e-06  4.0808628024869e-06  1.1473325968356e-19  1.9491894861130e-19  1.9141633121661e-12  !!!!
               PFSQIF 2D3 -2.8108738214051e-06  3.8878365981669e-06  1.6495966647777e-19  1.9159555837696e-19  8.5500946370572e-13  !!!!
              PFCQLNG 2D1 -3.5555642908082e-08  1.4133894678714e-09  3.3087224502121e-24  4.2309994566325e-24  0.0000000000000e+00      
              PFCQNNG 2D1 -5.2850140216625e-07  8.7885623295760e-09  5.2939559203394e-23  2.3127969379217e-23  0.0000000000000e+00      
               PFSQRF 2D3 -2.5817563600963e-06  4.0808628024869e-06  1.1473325968356e-19  1.8729533284351e-19  1.9368237270838e-12  !!!!
               PFSQSF 2D3 -2.7688769863077e-06  3.8603498954962e-06  1.7173593005581e-19  1.9022506087981e-19  8.5380046435793e-13  !!!!
              PFCQRNG 2D1 -3.5555642908082e-08  1.4133894678714e-09  3.3087224502121e-24  4.0294350860824e-24  0.0000000000000e+00      
              PFCQSNG 2D1 -5.2850140216625e-07  8.7885623295760e-09  5.2939559203394e-23  2.1069493041057e-23  0.0000000000000e+00      
             PFSQLTUR 2D1 -8.1159493430619e-07  2.6534563852270e-07  1.0587911840679e-22  3.4886441373701e-23  0.0000000000000e+00      
             PFSQITUR 2D1 -2.6516903010901e-06  3.8645677692316e-07  2.1175823681358e-22  1.2054745469441e-22  0.0000000000000e+00      
               PFPLSL 2D1  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00      
               PFPLSN 2D3  0.0000000000000e+00  2.9699580041076e-05  2.8290900438294e-19  3.2940259605044e-19  1.9022491754424e-13      
               PFHPSL 2D1 -0.0000000000000e+00 -0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00      
               PFHPSN 2D3 -8.4183459626431e+01 -0.0000000000000e+00  7.9936057773011e-13  9.3407143263467e-13  1.9030229052102e-13      
       TENDENCY_LOC%A 2D3 -2.7777777777778e-04  2.6313098096235e-04  8.4025668367627e-18  1.2733539001338e-18  2.7894966900677e-13  !!!!
       TENDENCY_LOC%Q 2D1 -1.9973824048511e-08  1.4955800737964e-08  1.7470054537120e-21  7.2556416205091e-22  0.0000000000000e+00      
       TENDENCY_LOC%T 2D3 -4.1322003941667e-05  5.4352784463613e-05  5.0483163656356e-18  2.3720080878698e-18  1.0673888878781e-12  !!!!
     TENDENCY_LOC%CLD 2D1 -1.2224810361478e-08  9.8165463509733e-09  7.2936650511863e-22  3.8539800942977e-22  0.0000000000000e+00

Attention: unless optimisation is lowered to -O0 HIP SCC-K-CACHING runs, but gives wrong results!

 ./bin/dwarf-cloudsc-hip-k-caching 1 262144 64
     NUMOMP=1, NGPTOT=262144, NPROMA=64, NGPBLKS=4096
     NUMOMP    NGPTOT  #GP-cols     #BLKS    NPROMA tid# : Time(msec)  MFlops/s     col/s
          1    262144    262144         0        64    0 :         81    403880   3235615 @ core#
          1    262144    262144      4096        64   -1:       1388     23559    188740 TOTAL
             Variable Dim             MinValue             MaxValue            AbsMaxErr         AvgAbsErr/GP          MaxRelErr-%
                PLUDE 2D1  0.0000000000000e+00  1.0267201089823e-04  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00      
             PCOVPTOT 2D3  0.0000000000000e+00  1.0000000000000e+00  4.5901993317443e-09  1.8352935183119e-09  6.8870821828400e-09  !!!!
     PRAINFRAC_TOPRFZ 2D1  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00      
               PFSQLF 2D3 -2.5817563593169e-06  4.0808628061397e-06  1.3548794010555e-13  1.6196846460487e-13  1.5905795454129e-06  !!!!
               PFSQIF 2D3 -2.8108738308272e-06  3.8878365967156e-06  4.3276962459680e-14  1.2224331904750e-13  5.4551992512670e-07  !!!!
              PFCQLNG 2D1 -3.5555642908082e-08  1.4133894678714e-09  3.3087224502121e-24  4.2309994566325e-24  0.0000000000000e+00      
              PFCQNNG 2D1 -5.2850140216625e-07  8.7885623295760e-09  5.2939559203394e-23  2.3127969379217e-23  0.0000000000000e+00      
               PFSQRF 2D3 -2.5817563593169e-06  4.0808628061397e-06  1.3548794010555e-13  1.4888439552623e-13  1.5396156725841e-06  !!!!
               PFSQSF 2D3 -2.7688769956890e-06  3.8603498940675e-06  4.1936549920128e-14  1.1492013125229e-13  5.1580407425537e-07  !!!!
              PFCQRNG 2D1 -3.5555642908082e-08  1.4133894678714e-09  3.3087224502121e-24  4.0294350860824e-24  0.0000000000000e+00      
              PFCQSNG 2D1 -5.2850140216625e-07  8.7885623295760e-09  5.2939559203394e-23  2.1069493041057e-23  0.0000000000000e+00      
             PFSQLTUR 2D1 -8.1159493430619e-07  2.6534563852270e-07  1.0587911840679e-22  3.4886441373701e-23  0.0000000000000e+00      
             PFSQITUR 2D1 -2.6516903010901e-06  3.8645677692316e-07  2.1175823681358e-22  1.2054745469441e-22  0.0000000000000e+00      
               PFPLSL 2D1  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00      
               PFPLSN 2D3  0.0000000000000e+00  2.9699579984120e-05  3.0081509538768e-13  5.0795087403549e-13  2.9333379362659e-07  !!!!
               PFHPSL 2D1 -0.0000000000000e+00 -0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00  0.0000000000000e+00      
               PFHPSN 2D3 -8.4183459464987e+01 -0.0000000000000e+00  8.5266038851728e-07  1.4397867542798e-06  2.9333379399949e-07  !!!!
       TENDENCY_LOC%A 2D3 -2.7777777777778e-04  2.6313098096235e-04  2.1690593440295e-11  2.4385455710285e-12  5.3420457566810e-07  !!!!
       TENDENCY_LOC%Q 2D3 -1.9973824048511e-08  1.4955800737715e-08  1.4805107136934e-15  1.0729974769284e-15  1.3698322142486e-06  !!!!
       TENDENCY_LOC%T 2D3 -4.1322003940963e-05  5.4352784463611e-05  3.9282906456526e-12  2.9547991993955e-12  1.3296412636511e-06  !!!!
     TENDENCY_LOC%CLD 2D3 -1.2224810346962e-08  9.8165463499650e-09  9.9302486600872e-09  1.9683485618857e-09  4.9326101410846e+00  !!!!

@reuterbal reuterbal mentioned this pull request Nov 4, 2024
Merged
reuterbal
reuterbal approved these changes Nov 4, 2024
View reviewed changes
Copy link
Collaborator

@reuterbal reuterbal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Tested with #93, confirmed working

@reuterbal reuterbal merged commit e467790 into develop Nov 4, 2024
18 checks passed
@reuterbal reuterbal deleted the nams_lumi_cce17 branch November 4, 2024 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@reuterbal reuterbal reuterbal approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@MichaelSt98 @reuterbal