kv/rangefeed: use snapshots instead of iterators for rangefeed catchup scans

[shailendra-patel]

On the drt-large cluster, node 12 was killed due to an Out of Memory (OOM) error. Prior to the OOM event, there was a noticeable increase in cgo memory usage. At the time of this event, 100 changefeeds were running and experiencing lag.

Memory and cpu profiles attached around the time of OOM.
memory-analysis.zip

Related discussion in slack thread.

Jira issue: CRDB-43801

[blathers-crl]

Hi @shailendra-patel, please add branch-* labels to identify which branch(es) this C-bug affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

Hi @shailendra-patel, please add branch-* labels to identify which branch(es) this GA-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

cc @cockroachdb/cdc

[asg0451]

just had a look from the cdc side. some observations

looking at (go) mem profiles, we have a lot more inuse mem in pebble compactions during the bad time, up to 25%. speaks to compaction issues

changefeeds were having trouble catching up -- 2 of them never finished before we canceled them -- which is odd because they were created with initial_scan=no 😂 here's what i think was happening:

• when a feed is created with initial_scan=no it sets its start time to the create stmt ts, then starts up a rangefeed from that time
• there's a limit on how many rangefeed catchup scans can be ongoing at once. default is 16, we had it set to 64 in drt-large. for reference we were trying to watch 300k ranges.
• as feeds are blocked waiting for that semaphore, their start time grows more and more noncurrent, effectively making them do some initial_scan work
• there were a lot of rangefeed restarts due to splits & merges (presumably operational chaos?)
• so changefeeds were just mostly waiting for their turn to run increasingly-long catchups, while those that were running would often fail due to operational chaos

i'm not sure if changefeeds are entirely to blame for the oomkill given the first point about pebble compactions. even if that's a non-issue, i think there is probably room for improvement here, but i don't see evidence of a ga-blocking bug.

@wenyihu6 / @rharding6373 do you concur?