Skip to content

Conversation

jakebailey
Copy link
Member

@jakebailey jakebailey commented Jun 19, 2024

Pulled out of #58928.

Locally, this looks like a big win for typescript.js startup:

Benchmark 1 (695 runs): node ./built/local-old/typescript.js
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           144ms ± 1.76ms     140ms …  152ms         15 ( 2%)        0%
  peak_rss           89.6MB ±  541KB    88.3MB … 94.3MB          7 ( 1%)        0%
  cpu_cycles          638M  ± 5.60M      629M  …  675M          18 ( 3%)        0%
  instructions       1.07G  ±  453K     1.07G  … 1.07G           2 ( 0%)        0%
  cache_references   16.8M  ±  362K     15.7M  … 18.0M          13 ( 2%)        0%
  cache_misses       2.31M  ±  201K     2.06M  … 3.33M          34 ( 5%)        0%
  branch_misses      4.74M  ± 14.8K     4.70M  … 4.79M           6 ( 1%)        0%
Benchmark 2 (726 runs): node ./built/local/typescript.js
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           138ms ± 1.86ms     134ms …  154ms         21 ( 3%)        ⚡-  4.3% ±  0.1%
  peak_rss           89.4MB ±  661KB    88.0MB … 95.1MB         38 ( 5%)          -  0.2% ±  0.1%
  cpu_cycles          607M  ± 6.24M      598M  …  669M          24 ( 3%)        ⚡-  4.8% ±  0.1%
  instructions       1.03G  ±  457K     1.03G  … 1.04G           3 ( 0%)        ⚡-  3.4% ±  0.0%
  cache_references   15.1M  ±  334K     14.2M  … 16.0M          16 ( 2%)        ⚡- 10.1% ±  0.2%
  cache_misses       2.29M  ±  206K     1.99M  … 3.85M          28 ( 4%)          -  1.2% ±  0.9%
  branch_misses      4.48M  ± 14.4K     4.44M  … 4.54M          18 ( 2%)        ⚡-  5.5% ±  0.0%

Also, this apparently runs before services swaps out the object allocators, so it causes a decrease in monomorphism.

Co-authored-by: Titian Cernicova-Dragomir <[email protected]>
@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Jun 19, 2024
@jakebailey
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jun 19, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,153 62,153 ~ ~ ~ p=1.000 n=6
Types 50,242 50,242 ~ ~ ~ p=1.000 n=6
Memory used 192,790k (± 0.73%) 194,027k (± 0.99%) ~ 192,167k 195,837k p=0.066 n=6
Parse Time 1.57s (± 0.84%) 1.58s (± 0.65%) ~ 1.56s 1.59s p=0.506 n=6
Bind Time 0.85s (± 1.21%) 0.86s (± 1.14%) ~ 0.85s 0.87s p=0.203 n=6
Check Time 11.19s (± 0.40%) 11.19s (± 0.39%) ~ 11.13s 11.25s p=0.747 n=6
Emit Time 3.28s (± 0.83%) 3.30s (± 0.45%) ~ 3.27s 3.31s p=0.166 n=6
Total Time 16.90s (± 0.41%) 16.92s (± 0.26%) ~ 16.88s 16.98s p=0.571 n=6
angular-1 - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 944,114 944,114 ~ ~ ~ p=1.000 n=6
Types 407,051 407,051 ~ ~ ~ p=1.000 n=6
Memory used 1,218,361k (± 0.00%) 1,218,295k (± 0.00%) ~ 1,218,238k 1,218,351k p=0.066 n=6
Parse Time 7.97s (± 0.53%) 7.94s (± 0.69%) ~ 7.86s 8.00s p=0.572 n=6
Bind Time 2.24s (± 0.62%) 2.22s (± 0.44%) -0.02s (- 0.75%) 2.21s 2.23s p=0.047 n=6
Check Time 35.67s (± 0.23%) 35.70s (± 0.24%) ~ 35.62s 35.82s p=0.809 n=6
Emit Time 16.14s (± 0.50%) 16.15s (± 0.30%) ~ 16.07s 16.19s p=0.630 n=6
Total Time 62.02s (± 0.27%) 62.01s (± 0.08%) ~ 61.94s 62.07s p=0.630 n=6
mui-docs - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,133,539 2,133,539 ~ ~ ~ p=1.000 n=6
Types 926,551 926,551 ~ ~ ~ p=1.000 n=6
Memory used 2,115,712k (± 0.00%) 2,115,723k (± 0.01%) ~ 2,115,595k 2,115,949k p=0.936 n=6
Parse Time 7.87s (± 0.22%) 7.86s (± 0.19%) ~ 7.85s 7.88s p=0.567 n=6
Bind Time 2.76s (± 0.55%) 2.77s (± 0.55%) ~ 2.74s 2.78s p=0.282 n=6
Check Time 83.86s (± 0.38%) 83.81s (± 0.36%) ~ 83.46s 84.23s p=0.936 n=6
Emit Time 0.16s (± 2.52%) 0.16s (± 2.58%) ~ 0.15s 0.16s p=0.218 n=6
Total Time 94.64s (± 0.33%) 94.60s (± 0.32%) ~ 94.25s 95.02s p=0.873 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,230,761 1,230,762 +1 (+ 0.00%) ~ ~ p=0.001 n=6
Types 261,189 261,187 -2 (- 0.00%) ~ ~ p=0.001 n=6
Memory used 2,346,502k (± 0.03%) 2,345,895k (± 0.03%) ~ 2,344,838k 2,346,901k p=0.230 n=6
Parse Time 5.95s (± 0.96%) 5.99s (± 0.85%) ~ 5.92s 6.07s p=0.261 n=6
Bind Time 2.26s (± 0.33%) 2.24s (± 0.73%) ~ 2.22s 2.26s p=0.165 n=6
Check Time 39.71s (± 0.31%) 39.84s (± 0.42%) ~ 39.63s 40.02s p=0.230 n=6
Emit Time 3.10s (± 1.31%) 3.20s (± 1.19%) +0.10s (+ 3.23%) 3.14s 3.24s p=0.008 n=6
Total Time 51.04s (± 0.35%) 51.29s (± 0.29%) +0.25s (+ 0.48%) 51.04s 51.43s p=0.031 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,230,761 1,230,762 +1 (+ 0.00%) ~ ~ p=0.001 n=6
Types 261,189 261,187 -2 (- 0.00%) ~ ~ p=0.001 n=6
Memory used 2,423,509k (± 0.01%) 2,423,424k (± 0.02%) ~ 2,422,810k 2,424,157k p=0.810 n=6
Parse Time 7.75s (± 0.65%) 7.73s (± 0.56%) ~ 7.68s 7.79s p=0.810 n=6
Bind Time 2.53s (± 0.85%) 2.49s (± 1.06%) -0.05s (- 1.78%) 2.45s 2.52s p=0.016 n=6
Check Time 49.43s (± 0.19%) 49.48s (± 0.18%) ~ 49.32s 49.57s p=0.298 n=6
Emit Time 3.92s (± 2.02%) 3.86s (± 0.93%) ~ 3.81s 3.90s p=0.126 n=6
Total Time 63.65s (± 0.09%) 63.57s (± 0.18%) ~ 63.38s 63.70s p=0.261 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 258,573 258,573 ~ ~ ~ p=1.000 n=6
Types 104,819 104,819 ~ ~ ~ p=1.000 n=6
Memory used 428,198k (± 0.01%) 428,173k (± 0.01%) ~ 428,123k 428,250k p=0.230 n=6
Parse Time 3.33s (± 0.96%) 3.32s (± 1.35%) ~ 3.29s 3.39s p=0.568 n=6
Bind Time 1.31s (± 1.50%) 1.32s (± 1.41%) ~ 1.30s 1.34s p=0.324 n=6
Check Time 17.75s (± 0.25%) 17.73s (± 0.32%) ~ 17.64s 17.80s p=0.470 n=6
Emit Time 1.36s (± 0.93%) 1.37s (± 1.07%) ~ 1.35s 1.39s p=0.392 n=6
Total Time 23.75s (± 0.19%) 23.73s (± 0.38%) ~ 23.60s 23.82s p=1.000 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,565 224,565 ~ ~ ~ p=1.000 n=6
Types 93,734 93,734 ~ ~ ~ p=1.000 n=6
Memory used 369,477k (± 0.02%) 369,415k (± 0.02%) ~ 369,346k 369,541k p=0.066 n=6
Parse Time 2.77s (± 0.97%) 2.76s (± 0.87%) ~ 2.73s 2.80s p=0.747 n=6
Bind Time 1.58s (± 0.95%) 1.58s (± 0.74%) ~ 1.57s 1.60s p=1.000 n=6
Check Time 15.45s (± 0.57%) 15.47s (± 0.37%) ~ 15.38s 15.54s p=0.748 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.80s (± 0.45%) 19.81s (± 0.34%) ~ 19.73s 19.88s p=1.000 n=6
vscode - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=NaN n=0
Symbols 0 0 ~ ~ ~ p=NaN n=0
Types 0 0 ~ ~ ~ p=NaN n=0
Memory used 0k 0k ~ ~ ~ p=NaN n=0
Parse Time 0s 0s ~ ~ ~ p=NaN n=0
Bind Time 0s 0s ~ ~ ~ p=NaN n=0
Check Time 0s 0s ~ ~ ~ p=NaN n=0
Emit Time 0s 0s ~ ~ ~ p=NaN n=0
Total Time 0s 0s ~ ~ ~ p=NaN n=0
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 266,926 266,926 ~ ~ ~ p=1.000 n=6
Types 108,691 108,691 ~ ~ ~ p=1.000 n=6
Memory used 411,270k (± 0.02%) 411,323k (± 0.01%) ~ 411,269k 411,369k p=0.230 n=6
Parse Time 3.16s (± 0.33%) 3.15s (± 0.28%) ~ 3.14s 3.16s p=0.273 n=6
Bind Time 1.41s 1.41s (± 0.29%) ~ 1.41s 1.42s p=0.405 n=6
Check Time 14.25s (± 0.23%) 14.24s (± 0.25%) ~ 14.19s 14.30s p=0.935 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 18.82s (± 0.17%) 18.81s (± 0.20%) ~ 18.76s 18.87s p=0.748 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 523,765 523,765 ~ ~ ~ p=1.000 n=6
Types 178,055 178,055 ~ ~ ~ p=1.000 n=6
Memory used 462,123k (± 0.03%) 461,673k (± 0.07%) -450k (- 0.10%) 461,352k 462,119k p=0.045 n=6
Parse Time 3.16s (± 0.62%) 3.16s (± 0.99%) ~ 3.13s 3.21s p=0.558 n=6
Bind Time 1.17s (± 0.44%) 1.18s (± 1.03%) ~ 1.17s 1.20s p=0.923 n=6
Check Time 17.97s (± 0.53%) 17.93s (± 0.25%) ~ 17.85s 17.97s p=0.336 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 22.31s (± 0.45%) 22.27s (± 0.23%) ~ 22.17s 22.32s p=0.146 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,277ms (± 0.43%) 2,296ms (± 0.60%) ~ 2,279ms 2,310ms p=0.065 n=6
Req 2 - geterr 5,081ms (± 0.61%) 5,080ms (± 0.30%) ~ 5,065ms 5,101ms p=0.810 n=6
Req 3 - references 265ms (± 0.40%) 264ms (± 0.41%) ~ 262ms 265ms p=0.554 n=6
Req 4 - navto 228ms (± 0.18%) 227ms (± 1.40%) ~ 223ms 231ms p=0.445 n=6
Req 5 - completionInfo count 1,357 1,357 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 88ms (± 0.58%) 89ms (± 7.13%) ~ 77ms 94ms p=0.276 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,393ms (± 0.49%) 2,418ms (± 1.14%) ~ 2,369ms 2,450ms p=0.093 n=6
Req 2 - geterr 3,787ms (± 0.16%) 3,795ms (± 0.24%) ~ 3,783ms 3,807ms p=0.172 n=6
Req 3 - references 276ms (± 0.30%) 277ms (± 0.32%) +2ms (+ 0.54%) 276ms 278ms p=0.021 n=6
Req 4 - navto 226ms (± 0.23%) 226ms (± 0.33%) ~ 225ms 227ms p=0.784 n=6
Req 5 - completionInfo count 1,519 1,519 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 69ms (± 0.75%) 73ms (± 1.11%) 🔻+5ms (+ 6.80%) 72ms 74ms p=0.004 n=6
xstate-main-1-tsserver - node (v18.15.0, x64)
Req 1 - updateOpen 5,192ms (± 0.73%) 5,197ms (± 0.86%) ~ 5,168ms 5,287ms p=0.936 n=6
Req 2 - geterr 1,121ms (± 0.35%) 1,127ms (± 1.87%) ~ 1,111ms 1,168ms p=0.936 n=6
Req 3 - references 76ms 77ms (± 3.28%) ~ 75ms 82ms p=0.599 n=6
Req 4 - navto 450ms (± 0.70%) 455ms (± 0.72%) +5ms (+ 1.00%) 448ms 457ms p=0.036 n=6
Req 5 - completionInfo count 3,413 3,413 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 828ms (± 2.25%) 837ms (± 1.61%) ~ 816ms 856ms p=0.470 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstate-main-1-tsserver - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 157.02ms (± 0.17%) 156.97ms (± 0.19%) -0.05ms (- 0.03%) 156.03ms 162.11ms p=0.004 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 241.66ms (± 0.15%) 232.55ms (± 0.14%) 🟩-9.11ms (- 3.77%) 231.22ms 237.16ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 236.71ms (± 0.13%) 227.62ms (± 0.14%) 🟩-9.08ms (- 3.84%) 226.05ms 231.44ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 236.43ms (± 0.14%) 227.29ms (± 0.15%) 🟩-9.14ms (- 3.86%) 225.50ms 232.15ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

Yeah that's 4% startup time.

@DanielRosenwasser
Copy link
Member

@typescript-bot cherry-pick this to release-5.5

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jun 19, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
cherry-pick this to release-5.5 ✅ Started ✅ Results

@typescript-bot
Copy link
Collaborator

Hey, @DanielRosenwasser! I've created #58933 for you.

@jakebailey jakebailey merged commit 47bb090 into microsoft:main Jun 19, 2024
@jakebailey jakebailey deleted the defer-barebones-lib branch June 19, 2024 18:34
DanielRosenwasser pushed a commit that referenced this pull request Jun 19, 2024
@sandersn sandersn removed this from PR Backlog Apr 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants