memory leaks in 2.6.0-dev.2.0

[jakemac53]

The latest dev SDK started causing timeouts for the package:build travis build bots.

When diagnosing locally I noticed that the memory usage of the test runner has gone up by approximately 6x (for the same version of the package) so I believe something has happened in the SDK.

Repro instructions:

git clone git@github.com:dart-lang/build.git
cd build/build_runner
pub upgrade
pub run test -t integration --total-shards 4 --shard-index 1

In the latest stable sdk (2.5.0) I see the main test process using <1GB of ram, in the latest SDK (2.6.0-dev.2.0) I see it using >6GB of ram.

[jakemac53]

Note that I haven't yet been able to create an isolated repro for this - it doesn't appear to be a general issue with isolates as far as I can tell which was my first intuition.

We do also create a lot of sub-processes so that could be related.

[vsmenon]

@a-siva @mraleph - it appears something between dev.1.0 and dev.2.0 is breaking testing on travis, at least for big tests. Adding to D26.

[jakemac53]

I did figure out how to hook observatory in fwiw - if you check out the test repo you can run it directly, but you need to pass the .packages file explicitly as well.

Updated instructions:

git clone git@github.com:dart-lang/build.git
git clone git@github.com:dart-lang/test.git
cd build/build_runner
pub upgrade
dart --packages=.packages --observe ../test/pkgs/test/bin/test.dart -t integration --total-shards 4 --shard-index 1

[a-siva]

I was trying to reproduce the problem, i tried using the instructions above on
Dart 2.5.0-edge.4b207266a7001f4f1e98ff1010f0df55a890d061

While the tests are running I see 'top' reporting the following numbers for the process

PID COMMAND %CPU TIME #TH #WQ #PORT MEM PURG CMPRS PGRP
46219 dart 0.0 02:12.28 23 0 50 6281M 0B 399M 46219

The memory usage reported is about 6GB. When looking through observatory also I find similar
numbers
pid 46219
peak memory 6.0GB
current memory 5.8GB

Not sure if memory usage has increased between 2.5.0 and 2.6.0.

[a-siva]

Can you run an individual test and compare the times it takes to finish between
2.6.0-dev.1.0 and 2.6.0-dev.2.0
I am wondering if the timeouts might be a result of
31996aa enabling bytecodes be default.

[a-siva]

/cc @alexmarkov @rmacnak-google

[jakemac53]

Is there a way I can try enabling/disabling the bytecodes by default to see if that is the problem?

[alexmarkov]

@jakemac53 you need to rebuild Dart SDK in order to disable bytecode by default. Either revert 31996aa or use the following commands to build Dart SDK in out/ReleaseX64:

rm -rf out
tools/gn.py -m release -a x64 --no-bytecode
tools/build.py -m release -a x64 create_sdk

[rmacnak-google]

According to --print-metrics, the kernel service has a 6.7GB heap with bytecode on, and a 923MB heap with bytecode off.

[alexmarkov]

https://dart-review.googlesource.com/c/sdk/+/118902 brings heap.old.capacity.max of kernel-service down to 3.478 GB.