Bookkeeper migration #8410

rustyrussell · 2025-07-11T06:39:10Z

This builds on #8445

We do all the things:

We put the coin movement notifications into the db, in separate tables.
We implement listchainmoves and listchannelmoves (and wait commands) on top of this.
We implement sql plugin support for these tables, and optimize it.
We migrate the internal tables & annotations that bookkeeper does from its local db into the datastore.
We add new (internal, for now) interfaces to inject "foreign" utxo deposits and spends.
We enhance libplugin to allow sync commands at any time, to avoid rewriting everything!
We move bookkeeper off the notifications onto the list commands, so no more replays!
We convert bookkeeper to use the sql plugin commands to query the db.
We migrate bookkeeper's accounts.db into our internal db

It needs some more testing, and I'm sure it dosn't pass CI, but it's ready for review!

Closes: #8393

cdecker · 2025-08-11T08:30:18Z

Cache cleared, restarting CI.

cdecker · 2025-08-11T08:31:14Z

Can't restart CI since it is too old, a rebase will start it again though.

niftynei

checkpoint submission #1

wallet/wallet.c

lightningd/coin_mvts.c

contrib/msggen/msggen/schema.json

tests/test_coinmoves.py

rustyrussell · 2025-08-15T05:15:33Z

I've done some more work:

Minor fixups from @niftynei review (mainly correcting tests and comments).
Rebased onto master now part 1 is merged.
Reworked final two commits (migrations) and implemented the full tests for them.

I appended the fixes needed, for later rebasing.

niftynei

checkpoint #2

plugins/bkpr/db.c

plugins/bkpr/onchain_fee.c

plugins/bkpr/descriptions.c

plugins/libplugin.c

plugins/bkpr/test/run-recorder.c

rustyrussell · 2025-08-17T23:49:15Z

I will now rebase, since CI won't run until I do. Folded fixes so far, too...

…hot. This way we can be sure they're the same after migration. Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

cdecker · 2025-08-15T10:27:38Z

wallet/db.c

@@ -1089,6 +1089,8 @@ static struct migration dbmigrations[] = {
 	 "  fees BIGINT NOT NULL,"
 	 "  PRIMARY KEY (id)"
 	 ")"), NULL},
+    /* We do a lookup before each append, to avoid duplicates */
+    {SQL("CREATE INDEX chain_moves_utxo_idx ON chain_moves (utxo)"), NULL},


An alternative would be to mark the utxo as a unique index, which causes conflicts on insert, could save a roundtrip, but then we sort of assume none of our statements can fail, as that'd tear down the transaction.

cdecker · 2025-08-15T10:29:39Z

wallet/wallet.c

@@ -3016,6 +3017,37 @@ void wallet_channel_close(struct wallet *w,
 	db_bind_u64(stmt, channel_state_in_db(CLOSED));
 	db_bind_u64(stmt, chan->dbid);
 	db_exec_prepared_v2(take(stmt));
+
+	/* Update all accouting records to use channel_id string, instead of


accouting -> accounting

niftynei

checkpoint #4

niftynei · 2025-08-16T02:09:05Z

wallet/account_migration.c

+	db_begin_transaction(account_db);
+	/* Last migration was 24.08.  Migrate there first if this happens. */
+	if (db_get_version(account_db) != 17)
+		fatal("Cannot migrate account database version %u",


nit: maybe suggest what version of CLN to use to update/migrate accounts.db to with first?

niftynei · 2025-08-16T02:09:50Z

wallet/account_migration.c

+		id = chain_mvt_index_created(ld, db, account, ev->credit, ev->debit);
+		db_bind_u64(stmt, id);
+		if (!mvt_tag_parse(ev->tag, strlen(ev->tag), &tag))
+			abort();


print missing tag name for easier debugging?

niftynei · 2025-08-16T17:11:04Z

tests/test_bookkeeper.py

-@unittest.skipIf(os.getenv('TEST_DB_PROVIDER', 'sqlite3') != 'sqlite3', "Depends on sqlite3 database location")
-def test_bookkeeping_external_withdraw_missing(node_factory, bitcoind):
-    """ Withdrawals to an external address turn up as
-    extremely large onchain_fees when they happen before


taking this out, along with the run-recorder.c tests, means almost all of the explicit onchain fee behavior checks have been removed.

niftynei · 2025-08-18T19:01:49Z

plugins/bkpr/bookkeeper.c

@@ -1616,7 +1513,7 @@ parse_and_log_chain_move(struct command *cmd,
 	e->spending_txid = tal_steal(e, spending_txid);

 	/* Now try to get out the optional parts */
-	err = json_scan(tmpctx, buf, params,
+	err = json_scan(tmpctx, buf, chainmove,
 			"{coin_movement:"


you deleted this portion of the scan above (L1606) but left here.

niftynei · 2025-08-18T19:13:59Z

plugins/bkpr/bookkeeper.c

 static struct command_result *inject_done(struct command *notif_cmd,
 					  const char *methodname,
 					  const char *buf,
 					  const jsmntok_t *result,
 					  void *unused)
 {
-	return notification_handled(notif_cmd);
+	/* We could do this lazily, but tests assume it happens now */


i think there's only one test that relies on logs printed by the bookkeeper and that's the test_bookkeeping_descriptions test.

the other tests (should?) rely on the coin_movements.py plugin in tests (iirc)

niftynei · 2025-08-18T19:26:08Z

plugins/bkpr/bookkeeper.c

-	e->debit = debit;
-	e->timestamp = timestamp;
-	e->tag = mvt_tag_str(tags[0]);
+	plugin_log(cmd->plugin, LOG_DBG, "coin_move 2 (%s) %s -%s %s %"PRIu64,


if i'm reading the code correctly, there's only one test that relies on these (test_bookkeeping_descriptions) and you can probably transition over to using the internal RPCs to assert that the coin moves are being correctly picked up there?

niftynei · 2025-08-18T19:30:54Z

tests/test_bookkeeper.py

    # Send l2 funds via the channel
    l1.pay(l2, invoice_msat)
-    l1.daemon.wait_for_log(r'coin movement:.*\'invoice\'')


if you remove this, you can also get rid of L353, the coin_mvt_plugin

niftynei · 2025-08-18T19:32:20Z

tests/test_bookkeeper.py

@@ -729,11 +648,15 @@ def test_bookkeeping_descriptions(node_factory, bitcoind, chainparams):
    # Send l2 funds via the channel
    bolt11_desc = 'test "bolt11" description, 🥰🪢'
    l1.pay(l2, 11000000, label=bolt11_desc)
+    wait_for(lambda: only_one(l1.rpc.listpeerchannels()['channels'])['htlcs'] == [])
+
+    # Need to call bookkeeper to trigger analysis!


these checks were added to make sure that the event completed. now that you've added the wait_for above, you can remove them.

tests/test_bookkeeper.py

The new access APIs are more symmetrical: 1. edit_utxo_description -> add_utxo_description 2. add_payment_hash_desc -> add_payment_hash_description And to read it, instead of accessing ->ev_desc (now removed) we use chain_event_description() & channel_event_description(), threading bkpr though as needed. Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

Remove the rebalance field from channel_event, and use the find_rebalance(bkpr, ev->db_id) to look it up instead. chain_event's also had a `rebalance` field, but it was only ever set (to false), never read. Note: list_rebalances() was only used by tests, not a public API. Signed-off-by: Rusty Russell <[email protected]>

We want to access it in stmt2chain_event, so plumb it through. Signed-off-by: Rusty Russell <[email protected]>

We won't be able to "UPDATE chain_events", so keep a separate record of these blockheights, and lookup that when the blockheight is 0. Signed-off-by: Rusty Russell <[email protected]>

Python's assert gives great analysis of what the differences are, making debugging much easier. So feed it dicts, not tuples, and simply do an assert. Signed-off-by: Rusty Russell <[email protected]>

…njection. For the moment, we'll continue to use bookkeeper to monitor the notifications to insert these (we don't have the internal infrastructure for that, and actually these commands are probably better than using notifications). We hoist param_outpoint() into common code, since there are already two uses. Signed-off-by: Rusty Russell <[email protected]>

…njected. This allows the bookkeeper plugin to know it's not actually a channel account. Remove the "ignored" tag from the schema too: we removed it previously. Signed-off-by: Rusty Russell <[email protected]>

… transfer_from. Before bkpr_listaccountevents() gave entries with origin like: {'account': "nifty's secret stash", 'blockheight': 111, 'credit_msat': 180000000, 'currency': 'bcrt', 'debit_msat': 0, 'origin': 'null', 'outpoint': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:0', 'tag': 'deposit', 'timestamp': 1679955976, 'type': 'chain'}, Changelog-Changed: Plugins: "utxo_deposit" is allows to have missing `transfer_from`, and null is not considered an account name. Signed-off-by: Rusty Russell <[email protected]>

…xodeposit / injectutxospend calls. And thus we absorb them as normal when they come back as "foreign" entries. Signed-off-by: Rusty Russell <[email protected]>

After much thought and mis-steps, I chose a simple solution: open another fd for sync comms. It's almost impossible to know what state the async one is in. jsonrpc_request_sync() is enhanced to return a valid tal object, as the current behaviour of returning a pointer to inside an array was surprising. Changelog-Changed: libplugin: you can now call the synchronous API functions at any time (not just in the init callback). Signed-off-by: Rusty Russell <[email protected]>

It's a great test, but it's very hard to simulate now we are going to be going from the internal db. Signed-off-by: Rusty Russell <[email protected]>

Rearrange all the JSON interfaces to call refresh_moves() (async) before doing anything. This does nothing for now, but it will be useful once we transition from notifications to using the list commands. Signed-off-by: Rusty Russell <[email protected]>

This is reliable, meaning we should never get replayed events. We have to reference count to make sure all commands are complete, before we return. In particular, annotating with descriptions can involve several calls to list commands. We need to give them the results *after* this is all complete. test_bookkeeping_descriptions() relied on log messages from notifications, which now only happen when a command is called. This changes the test a bit. Since we no longer subscribe to the balance_snapshot event, we need to create the wallet account at initialization, as callers expect it to exist. Signed-off-by: Rusty Russell <[email protected]>

We don't need it now bookkeeper uses the list commands. Signed-off-by: Rusty Russell <[email protected]>

We're going to be using this instead of our internal db. I also made json_out_obj() take the str arg, as it didn't and I expected it to. Signed-off-by: Rusty Russell <[email protected]>

Cleaner (I'm about to hand it a sha256 on the stack). Signed-off-by: Rusty Russell <[email protected]>

With some help (and hinderance!) from ChatGPT: the field names differ slightly from our internal db. The particilar wrinkle is that we have to restrict all queries to limit them to entries we've seen already. Our code expects this (we used to only enter it into the db when we processed it), and it would otherwise be confusing if a sql query returned inconsistent results because an event occurred while bookkeeper was processing. Signed-off-by: Rusty Russell <[email protected]>

There will be no more missing events (and at initialization time, we will do that as a migration). Signed-off-by: Rusty Russell <[email protected]>

Changelog-Changed: Plugins: `bookkeeper` now uses the lightningd database, not "accounts.db". Signed-off-by: Rusty Russell <[email protected]>

Now handles when we remove the db. Signed-off-by: Rusty Russell <[email protected]>

And gracefully fail for this case. There's no such thing for Postgres, but that's because dbs need to be set up by the admin. Signed-off-by: Rusty Russell <[email protected]>

We take over the --bookkeeper-dir and --bookkeeper-db options, and then if we can find the bookkeeper db we extract the records to initialize our chain_moves and channel_moves tables. Of course, bookkeeper now needs to not register those options. When bookkeeper gets invoked the first time, it will reconstruct everything from listchannelmoves and listcoinmoves. It cannot preserve manually-added descriptions, so we put those in the datastore for it ready to go. Note that the order of onchain_fee changes slightly from the original. But this is fine. Signed-off-by: Rusty Russell <[email protected]>

If we don't have an accountdb from bookkeeper: 1. Generate a deposit chain event for every confirmed UTXO. 2. Generate an open chain event for every open, confirmed channel. 3. Generate a push/lease event if necessary. 4. Generate a fixup "journal" entry if balance is different from initial. Signed-off-by: Rusty Russell <[email protected]>

This requires us to turn "sql" calls into calls to a local db, which means pulling in a lot of infrastructure. But it's possible. Signed-off-by: Rusty Russell <[email protected]>

ubsan complains that we declared a function not to take NULL. Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

rustyrussell added this to the v25.09 milestone Jul 11, 2025

madelinevibes added Status::Ready for Review The work has been completed and is now awaiting evaluation or approval. PLEASE clear CI 🫠 labels Aug 6, 2025

cdecker removed the PLEASE clear CI 🫠 label Aug 11, 2025

rustyrussell force-pushed the guilt/bookkeeper-migrate branch from be18b01 to f649a49 Compare August 12, 2025 05:49

rustyrussell requested a review from cdecker as a code owner August 12, 2025 05:49

rustyrussell requested a review from niftynei August 12, 2025 05:56

rustyrussell changed the title ~~Bookkeeper refactors and cleanups in preparation for migration~~ Bookkeeper migration Aug 12, 2025

niftynei reviewed Aug 15, 2025

View reviewed changes

rustyrussell force-pushed the guilt/bookkeeper-migrate branch 3 times, most recently from ef8cb22 to 5ab28b0 Compare August 15, 2025 06:07

niftynei reviewed Aug 16, 2025

View reviewed changes

rustyrussell force-pushed the guilt/bookkeeper-migrate branch 2 times, most recently from bd2e7a1 to d9a1b37 Compare August 18, 2025 01:44

rustyrussell enabled auto-merge (rebase) August 18, 2025 01:44

rustyrussell force-pushed the guilt/bookkeeper-migrate branch 4 times, most recently from 3b297c0 to c4d1280 Compare August 18, 2025 07:13

rustyrussell added 2 commits August 18, 2025 19:15

pytest: check current bookkeeper events in saved test_migration snaps…

3687d6d

…hot. This way we can be sure they're the same after migration. Signed-off-by: Rusty Russell <[email protected]>

db: helper for binding/reading bitcoin_outpoint.

bf936ac

Signed-off-by: Rusty Russell <[email protected]>

rustyrussell force-pushed the guilt/bookkeeper-migrate branch 2 times, most recently from 1245ed2 to a75f409 Compare August 18, 2025 12:08

cdecker reviewed Aug 18, 2025

View reviewed changes

niftynei reviewed Aug 18, 2025

View reviewed changes

rustyrussell added 27 commits August 19, 2025 10:30

bkpr: infrastructure to keep rebalance pairs in mem & datastore.

deef239

Signed-off-by: Rusty Russell <[email protected]>

bkpr: pass bkpr struct into all functions in recorder.c.

1ded980

We want to access it in stmt2chain_event, so plumb it through. Signed-off-by: Rusty Russell <[email protected]>

bkpr: add in-mem & datastore storage for external blockheights.

f765dc3

We won't be able to "UPDATE chain_events", so keep a separate record of these blockheights, and lookup that when the blockheight is 0. Signed-off-by: Rusty Russell <[email protected]>

pytest: use assert inside event testing.

8119b52

Python's assert gives great analysis of what the differences are, making debugging much easier. So feed it dicts, not tuples, and simply do an assert. Signed-off-by: Rusty Russell <[email protected]>

common: use "foreign" in extra_tags to indicate a chain movement is i…

a98835d

…njected. This allows the bookkeeper plugin to know it's not actually a channel account. Remove the "ignored" tag from the schema too: we removed it previously. Signed-off-by: Rusty Russell <[email protected]>

bkpr: forward utxo_deposit / utxo_spend notifications to new injectut…

f47d920

…xodeposit / injectutxospend calls. And thus we absorb them as normal when they come back as "foreign" entries. Signed-off-by: Rusty Russell <[email protected]>

plugins/bkpr/test/run-recorder: remove.

49b1c7d

It's a great test, but it's very hard to simulate now we are going to be going from the internal db. Signed-off-by: Rusty Russell <[email protected]>

wallet: remove notification on duplicate chain entries.

ce8766f

We don't need it now bookkeeper uses the list commands. Signed-off-by: Rusty Russell <[email protected]>

bkpr: helpers to query sql plugin for chainmoves and channelmoves.

3b68ed7

We're going to be using this instead of our internal db. I also made json_out_obj() take the str arg, as it didn't and I expected it to. Signed-off-by: Rusty Russell <[email protected]>

bkpr: take, don't steal in new_channel_event.

46a468e

Cleaner (I'm about to hand it a sha256 on the stack). Signed-off-by: Rusty Russell <[email protected]>

bkpr: remove missing event handling.

6116d85

There will be no more missing events (and at initialization time, we will do that as a migration). Signed-off-by: Rusty Russell <[email protected]>

bookkeeper: remove database.

b5ff099

Changelog-Changed: Plugins: `bookkeeper` now uses the lightningd database, not "accounts.db". Signed-off-by: Rusty Russell <[email protected]>

pytest: restore bookkeeper to recover tests.

60d6ae2

Now handles when we remove the db. Signed-off-by: Rusty Russell <[email protected]>

db: support mode where we don't want to create the db.

a00877d

And gracefully fail for this case. There's no such thing for Postgres, but that's because dbs need to be set up by the admin. Signed-off-by: Rusty Russell <[email protected]>

bkpr: restore run-recorder.

1742cf3

This requires us to turn "sql" calls into calls to a local db, which means pulling in a lot of infrastructure. But it's possible. Signed-off-by: Rusty Russell <[email protected]>

plugins/bkpr/test/run-recorder: don't hand NULL cmd.

b68c97b

ubsan complains that we declared a function not to take NULL. Signed-off-by: Rusty Russell <[email protected]>

pytest: wait to make sure moves are registered in coinmoves tests.

5a38d5c

Signed-off-by: Rusty Russell <[email protected]>

rustyrussell force-pushed the guilt/bookkeeper-migrate branch from 32ed994 to 5a38d5c Compare August 19, 2025 01:05

rustyrussell disabled auto-merge August 19, 2025 04:07

rustyrussell merged commit 94c1cf5 into ElementsProject:master Aug 19, 2025
109 of 115 checks passed

Bookkeeper migration #8410

Bookkeeper migration #8410

Uh oh!

Conversation

rustyrussell commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cdecker commented Aug 11, 2025

Uh oh!

cdecker commented Aug 11, 2025

Uh oh!

niftynei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rustyrussell commented Aug 15, 2025

Uh oh!

niftynei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rustyrussell commented Aug 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

niftynei left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rustyrussell commented Jul 11, 2025 •

edited

Loading

rustyrussell commented Aug 17, 2025 •

edited

Loading