High disk IO volume after upgrading to 3.0

[nur858]

We recently upgraded our containerized .net core app to 3.0. But we started to see high disk write volume in /tmp directory. We took a peek in one of the files and it looked like it was temporarily writing response data in a tmp file and unlinking after sending response. Is it by design? Would it be possible to use memory as temporary storage?

DotNet version: 3.0
Kestrel Transport: Sockets
Container base: CentOS
cpu cores/container: 2
memory/container: 4GB

[pid 2931533] open("/tmp/ASPNETCORE_d297fa6b-4150-4762-a06e-4f31823e2945.tmp", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 398

[pid 2931533] write(398, "191217 38|191217 39|191217 40|19"..., 1024) = 1024
[pid 2931533] write(398, "137|191217 138|191217 139|191217"..., 1024) = 1024
[pid 2931533] write(398, "30|191217 231|191217 232|191217 "..., 1024) = 1024
[pid 2931533] write(398, "4|191217 325|191217 326|191217 3"..., 1024) = 1024
[pid 2931533] write(398, "|191217 418|191217 419|191217 42"..., 1024) = 1024
[pid 2931533] write(398, "191217 511|191217 512|191217 513"..., 1024) = 1024
[pid 2931533] write(398, "91217 623|191217 624|191217 625|"..., 1024) = 1024
[pid 2931533] write(398, "1217 744|191217 745|191217 746|1"..., 1024) = 1024
[pid 2931533] write(398, "18 786|191218 789|191218 790|191"..., 1024) = 1024
[pid 2931533] write(398, "9 853|191219 854|191219 855|1912"..., 1024) = 1024
[pid 2931533] write(398, " 981|191219 982|191219 983|19121"..., 1024) = 1024
[pid 2931533] write(398, "191218 396|191218 397|191218 398"..., 1024) = 1024
[pid 2931533] write(398, "91218 570|191218 571|191218 590|"..., 1024) = 1024
[pid 2931533] write(398, "191218 74|191218 75|191218 76|19"..., 1024) = 1024
[pid 2931533] write(398, "18 212|191218 213|191218 214|191"..., 1024) = 1024
[pid 2931533] write(398, "8 306|191218 307|191218 308|1912"..., 1024) = 1024
[pid 2931533] write(398, " 449|191218 450|191218 451|19121"..., 16384) = 16384
[pid 2951520] unlink("/tmp/ASPNETCORE_d297fa6b-4150-4762-a06e-4f31823e2945.tmp") = 0
[pid 2951520] close(398)                = 0

[Pilchie]

@rynowak @richlander

[Pilchie]

Oh, 3.0, not 3.1

[Pilchie]

Starting with area-servers

[Tratcher]

Probably the buffering in the legacy MVC response serializers (e.g. Newtonsoft.Json). @pranavkm

[nur858]

We tried downgrading to 2.2 and the disk write came down to 0 bytes. And yes we are using Newtonsoft.Json. we used newtonsoft.json with both 2.2 and 3.0

Does this mean that using newtonsoft + 3.0 is likely to result in high disk write volume?

[rynowak]

Does this mean that using newtonsoft + 3.0 is likely to result in high disk write volume?

You can configure this behavior.

services.AddControllers(options =>
{
    options.SuppressOutputFormatterBuffering = true;
});

There's a tradeoff here.

In 2.2 and earlier the newtonsoft.json serializer would do synchronous writes to the output stream. For large payloads this could lead to thread starvation, and otherwise bad throughput performance.

In 3.0 and newer by default the newtonsoft.json serializer will buffer into memory (up to 32kb) and then to disk if the payload is larger than that. You can disable this behavior as shown above.

---

In 3.0 and newer if you use the System.Text.Json serializer, it's fully async, so it will work efficiently for any size payload.

[Tratcher]

Disabling SuppressOutputFormatterBuffering will also require enabling AllowSynchronousIO on the server. aspnet/Announcements#342

[nur858]

Thanks for the response.

In 3.0 and newer by default the newtonsoft.json serializer will buffer into memory (up to 32kb) and then to disk if the payload is larger than that.

is 32kb configurable? any downside in increasing it? response from my service is less than 2MB.

We were talking about switching to System.Text.Json. it requires quite a bit of work but we will get to the new json api.