[BUG] Latest master takes up too much memory and fails with Fatal error: out of memory

[phated]

Describe the bug
While trying to test out #1442, I noticed that JSOO took up 10gb of RAM on my mac while building libbinaryen. It completed; however, when running in CI, we're getting fatal OoM errors. See grain-lang/libbinaryen#78 for the CI runs.

Expected behavior
Expected to take up less memory and/or not crash with OoM.

Versions
OCaml: 4.13.1 and 4.14.0
JSOO: master
OS: Linux and Windows (the Mac runners might be larger??)

[hhugo]

Can you give a bit more details.
What command should I run on the libbinaryen repo to reproduce the behavior ?
Is the issue with separate compilation or whole program ?

[hhugo]

Parsing https://github.com/grain-lang/libbinaryen/blob/main/binaryen.es5.js, memory consumption was multiplied by 5 compared to jsoo 5.0.

[phated]

@hhugo sorry for the delayed response. libbinaryen should be a standard opam & dune project. Did you get it building or do you need step-by-step instructions?

[hhugo]

Just providing the target that should be build would have been useful and saved me some time.

[phated]

@hhugo In the Grain CI, we're seeing OOM failures with 5.3 on Windows. Should we re-open this or a new issue?

[hhugo]

You see oom with 5.3 and not with 5.2 ? Have tested 5.4 ?

[hhugo]

Can you point me to CI failing jobs

[phated]

Sorry for the delay. This is only happening on Windows. Here's a failing CI run: https://github.com/grain-lang/grain/actions/runs/5502406329/jobs/10045052441

It was happening in 5.2 and 5.3 and I didn't try 5.4 because it didn't seem like there were any improvements by skimming the diff.

I reverted to 5.0.1 and we no longer get OoM on any CI.

[hhugo]

Would you be able to pass the --debug times flag so that we can see if it's still the parsing of js that causes problem? How much memory do you have in this window runner ?

[phated]

@hhugo sorry for the delay here. I'm working on this in grain-lang/grain#1909

How much memory do you have in this window runner ?

According to https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources the Windows and Linux runners have 2-core CPU (x86_64), 7 GB of RAM, 14 GB of SSD space.

[phated]

We have 4 binaries, so I have 4 sets of logs. I'll hide put them each in a collapsible element but I'm not sure they will be much different from each other.

parsing js: 23.17
Start parsing...
    read debug events: 0.00
  parsing: 2.09
Start Optimizing...
  tail calls: 0.[27](https://github.com/grain-lang/grain/actions/runs/6075457328/job/16481678247?pr=1909#step:7:28)
    flow analysis 1: 0.92
    flow analysis 2: 1.33
    flow analysis 3: 0.19
    flow analysis 4: 1.16
    flow analysis 5: 1.03
  flow analysis: 4.63
  dead code elim.: 1.78
  inlining: 0.86
  dead code elim.: 1.72
  tail calls: 0.19
    phi-simpl. 1: 0.38
    phi-simpl. 2: 0.56
  phi-simpl.: 1.38
    flow analysis 1: 0.45
    flow analysis 2: 0.59
    flow analysis 3: 0.13
    flow analysis 4: 0.48
    flow analysis 5: 0.58
  flow analysis: 2.23
  dead code elim.: 1.73
  inlining: 1.03
  dead code elim.: 1.77
    flow analysis 1: 0.34
    flow analysis 2: 0.30
    flow analysis 3: 0.14
    flow analysis 4: 0.25
    flow analysis 5: 0.48
  flow analysis: 1.52
  dead code elim.: 1.78
  inlining: 0.80
  dead code elim.: 1.53
    phi-simpl. 1: 0.20
    phi-simpl. 2: 0.23
  phi-simpl.: 0.92
    flow analysis 1: 0.27
    flow analysis 2: 0.[28](https://github.com/grain-lang/grain/actions/runs/6075457328/job/16481678247?pr=1909#step:7:29)
    flow analysis 3: 0.09
    flow analysis 4: 0.23
    flow analysis 5: 0.45
  flow analysis: 1.33
      global flow analysis (initialize): 0.70
      global flow analysis (solve): 0.88
    global flow analysis: 1.58
  free vars: 0.91
  generate closures: 1.64
  dead code elim.: 1.69
 optimizations : 33.52
Start Generation...
  code gen.: 3.13
Start Linking...
  linking: 0.94
Start Optimizing js...
    share constant: 6.86
    simpl: 1.50
    clean: 1.25
    shortten vars: 2.94
  optimizing: 16.[30](https://github.com/grain-lang/grain/actions/runs/6075457328/job/16481678247?pr=1909#step:7:31)
Start Coloring...
  coloring: 19.41
Start Checks...
  checks: 6.38
Start Writing file...
  write: 2.[36](https://github.com/grain-lang/grain/actions/runs/6075457328/job/16481678247?pr=1909#step:7:37)
compilation: 107.30

parsing js: 22.97
Start parsing...
    read debug events: 0.00
  parsing: 2.16
Start Optimizing...
  tail calls: 0.28
    flow analysis 1: 1.00
    flow analysis 2: 1.38
    flow analysis 3: 0.20
    flow analysis 4: 1.13
    flow analysis 5: 1.02
  flow analysis: 4.72
  dead code elim.: 1.94
  inlining: 0.92
  dead code elim.: 1.75
  tail calls: 0.22
    phi-simpl. 1: 0.39
    phi-simpl. 2: 0.58
  phi-simpl.: 1.45
    flow analysis 1: 0.52
    flow analysis 2: 0.70
    flow analysis 3: 0.14
    flow analysis 4: 0.56
    flow analysis 5: 0.63
  flow analysis: 2.55
  dead code elim.: 1.97
  inlining: 1.09
  dead code elim.: 1.91
    flow analysis 1: 0.34
    flow analysis 2: 0.31
    flow analysis 3: 0.17
    flow analysis 4: 0.23
    flow analysis 5: 0.61
  flow analysis: 1.67
  dead code elim.: 1.91
  inlining: 0.75
  dead code elim.: 1.67
    phi-simpl. 1: 0.36
    phi-simpl. 2: 0.31
  phi-simpl.: 1.09
    flow analysis 1: 0.27
    flow analysis 2: 0.27
    flow analysis 3: 0.11
    flow analysis 4: 0.20
    flow analysis 5: 0.45
  flow analysis: 1.30
      global flow analysis (initialize): 0.88
      global flow analysis (solve): 0.91
    global flow analysis: 1.78
  free vars: 1.00
  generate closures: 1.67
  dead code elim.: 1.92
 optimizations : 36.09
Start Generation...
  code gen.: 4.14
Start Linking...
  linking: 0.83
Start Optimizing js...
    share constant: 6.97
    simpl: 1.59
    clean: 1.53
    shortten vars: 2.69
  optimizing: 16.45
Start Coloring...
  coloring: 19.45
Start Checks...
  checks: 6.17
Start Writing file...
  write: 2.53
compilation: [110](https://github.com/grain-lang/grain/actions/runs/6075457328/job/16481678247?pr=1909#step:7:111).80

parsing js: 22.97
Start parsing...
    read debug events: 0.00
  parsing: 2.25
Start Optimizing...
  tail calls: 0.30
    flow analysis 1: 0.94
    flow analysis 2: 1.41
    flow analysis 3: 0.20
    flow analysis 4: 1.14
    flow analysis 5: 1.03
  flow analysis: 4.72
  dead code elim.: 1.83
  inlining: 0.84
  dead code elim.: 1.64
  tail calls: 0.20
    phi-simpl. 1: 0.34
    phi-simpl. 2: 0.58
  phi-simpl.: 1.38
    flow analysis 1: 0.53
    flow analysis 2: 0.61
    flow analysis 3: 0.13
    flow analysis 4: 0.48
    flow analysis 5: 0.55
  flow analysis: 2.30
  dead code elim.: 1.73
  inlining: 0.95
  dead code elim.: 1.73
    flow analysis 1: 0.33
    flow analysis 2: 0.31
    flow analysis 3: 0.14
    flow analysis 4: 0.25
    flow analysis 5: 0.53
  flow analysis: 1.56
  dead code elim.: 1.72
  inlining: 0.69
  dead code elim.: 1.50
    phi-simpl. 1: 0.31
    phi-simpl. 2: 0.30
  phi-simpl.: 1.03
    flow analysis 1: 0.25
    flow analysis 2: 0.28
    flow analysis 3: 0.08
    flow analysis 4: 0.20
    flow analysis 5: 0.44
  flow analysis: 1.25
      global flow analysis (initialize): 0.83
      global flow analysis (solve): 0.78
    global flow analysis: 1.61
  free vars: 0.92
  generate closures: 1.56
  dead code elim.: 1.70
 optimizations : 33.47
Start Generation...
  code gen.: 4.00
Start Linking...
  linking: 0.83
Start Optimizing js...
    share constant: 6.27
    simpl: 1.50
    clean: 1.48
    shortten vars: 2.63
  optimizing: 15.50
Start Coloring...
  coloring: 19.38
Start Checks...
  checks: 6.05
Start Writing file...
  write: 2.41
compilation: 106.84

parsing js: 22.92
Start parsing...
    read debug events: 0.00
  parsing: 2.25
Start Optimizing...
  tail calls: 0.28
    flow analysis 1: 1.03
    flow analysis 2: 1.44
    flow analysis 3: 0.20
    flow analysis 4: 1.13
    flow analysis 5: 1.03
  flow analysis: 4.83
  dead code elim.: 1.89
  inlining: 0.91
  dead code elim.: 1.73
  tail calls: 0.20
    phi-simpl. 1: 0.38
    phi-simpl. 2: 0.59
  phi-simpl.: 1.42
    flow analysis 1: 0.63
    flow analysis 2: 0.70
    flow analysis 3: 0.14
    flow analysis 4: 0.53
    flow analysis 5: 0.53
  flow analysis: 2.53
  dead code elim.: 1.94
  inlining: 1.03
  dead code elim.: 1.81
    flow analysis 1: 0.30
    flow analysis 2: 0.31
    flow analysis 3: 0.14
    flow analysis 4: 0.22
    flow analysis 5: 0.59
  flow analysis: 1.56
  dead code elim.: 1.70
  inlining: 0.75
  dead code elim.: 1.78
    phi-simpl. 1: 0.31
    phi-simpl. 2: 0.27
  phi-simpl.: 1.00
    flow analysis 1: 0.27
    flow analysis 2: 0.23
    flow analysis 3: 0.11
    flow analysis 4: 0.19
    flow analysis 5: 0.47
  flow analysis: 1.27
      global flow analysis (initialize): 0.88
      global flow analysis (solve): 0.78
    global flow analysis: 1.66
  free vars: 0.94
  generate closures: 1.58
  dead code elim.: 2.05
 optimizations : 35.31
Start Generation...
  code gen.: 4.25
Start Linking...
  linking: 0.89
Start Optimizing js...
    share constant: 6.58
    simpl: 1.31
    clean: 1.70
    shortten vars: 2.73
  optimizing: 16.00
Start Coloring...
  coloring: 18.88
Start Checks...
  checks: 5.66
Start Writing file...
  write: 2.38
compilation: 108.55

Also worth noting that I set -j 1 on our dune command so I wouldn't get interspersed logs and the Windows machine didn't OOM for the run, so it seems that trying to run 4 jsoo processes that each take up ~4gb of RAM is probably the issue. I'm not sure why this wouldn't result in Linux crashing with OOM (maybe swap space? does Windows swap?).

[hhugo]

Can you try the branch https://github.com/ocsigen/js_of_ocaml/tree/size2, it should improve a bit the situation.