cmd/geth: add db commands: stats, compact, put, get, delete #22014

holiman · 2020-12-14T08:21:59Z

This PR supersedes #21991 and #21887 (those PRs contain more info and examples.

This PR introduces

db.put to put a value into the database
db.get to read a value from the database
db.delete to delete a value from the database
db.stats to check compaction info from the database
db.compact to trigger a db compaction

It also moves inspectdb to db.inspect

holiman · 2021-02-01T12:58:45Z

Rebased + squashed, PTAL

holiman · 2021-02-02T09:38:22Z

Suggestions from triage:

geth db put xx
geth db get xx
geth db compact
geth db delete xx
geth db inspect
geth removedb

holiman · 2021-02-02T12:30:20Z

One thing that I'm having some problems with.. Three cases:

./build/bin/geth db stats --goerli  
./build/bin/geth db --goerli  stats 
./build/bin/geth --goerli db  stats

What should happen in these cases?

A) Only one works, the others show some kind of error + help (which one should work?)
B) All three works, with identical result, resolving the goerli datadir
C) Some of them just ignore the --goerli but otherwise "works"

RIght now, I'm at C, which is obviously incorrect. But should A or B be the target?

holiman · 2021-02-02T12:34:02Z

Ok, nevermind, I'll go with

A). Canon form is geth --goerli db stats, and the other variations show error + help

holiman · 2021-02-02T13:39:46Z

I added a commit which makes the ArgsUsage show:
Before:

[user@work go-ethereum]$ ./build/bin/geth --goerli export -h
export [command options] [arguments...]

After:

[user@work go-ethereum]$ ./build/bin/geth --goerli export -h
export [command options] <filename> [<blockNumFirst> <blockNumLast>]

And

[user@work go-ethereum]$ geth db put -h
put <hex-encoded key> <hex-encoded value>
This command sets a given database key to the given value. 
WARNING: This is a low-level operation which may cause database corruption!

holiman · 2021-02-08T12:58:24Z

Before this PR:

./build/bin/geth --goerli db inspect
...
INFO [02-08|13:30:14.738] Inspecting database                      count=56301000 elapsed=41.019s
+-----------------+--------------------+------------+----------+
|    DATABASE     |      CATEGORY      |    SIZE    |  ITEMS   |
+-----------------+--------------------+------------+----------+
| Key-Value store | Headers            | 70.02 MiB  |   113420 |
| Key-Value store | Bodies             | 212.04 MiB |   113420 |
| Key-Value store | Receipt lists      | 145.44 MiB |   113420 |
| Key-Value store | Difficulties       | 5.04 MiB   |   114067 |
| Key-Value store | Block number->hash | 4.59 MiB   |   113665 |
| Key-Value store | Block hash->number | 163.71 MiB |  4187014 |
| Key-Value store | Transaction index  | 294.63 MiB |  8581639 |
| Key-Value store | Bloombit index     | 146.44 MiB |  2093056 |
| Key-Value store | Contract codes     | 247.72 MiB |    43117 |
| Key-Value store | Trie nodes         | 4.27 GiB   | 42450160 |
| Key-Value store | Trie preimages     | 46.79 MiB  |   672798 |
| Key-Value store | Account snapshot   | 0.00 B     |        0 |
| Key-Value store | Storage snapshot   | 0.00 B     |        0 |
| Key-Value store | Clique snapshots   | 2.23 MiB   |     2433 |
| Key-Value store | Singleton metadata | 185.00 B   |        6 |
| Ancient store   | Headers            | 1.34 GiB   |  4096595 |
| Ancient store   | Bodies             | 1.94 GiB   |  4096595 |
| Ancient store   | Receipt lists      | 1.01 GiB   |  4096595 |
| Ancient store   | Difficulties       | 39.03 MiB  |  4096595 |
| Ancient store   | Block number->hash | 148.46 MiB |  4096595 |
| Light client    | CHT trie nodes     | 0.00 B     |        0 |
| Light client    | Bloom trie nodes   | 0.00 B     |        0 |
+-----------------+--------------------+------------+----------+
|                         TOTAL        | 10.04 GIB  |          |
+-----------------+--------------------+------------+----------+

After this PR:

INFO [02-08|13:40:44.535] Inspecting database                      count=55761000 elapsed=40.177s
+-----------------+--------------------+------------+----------+
|    DATABASE     |      CATEGORY      |    SIZE    |  ITEMS   |
+-----------------+--------------------+------------+----------+
| Key-Value store | Headers            | 70.02 MiB  |   113420 |
| Key-Value store | Bodies             | 212.04 MiB |   113420 |
| Key-Value store | Receipt lists      | 145.44 MiB |   113420 |
| Key-Value store | Difficulties       | 5.04 MiB   |   114067 |
| Key-Value store | Block number->hash | 4.59 MiB   |   113665 |
| Key-Value store | Block hash->number | 163.71 MiB |  4187014 |
| Key-Value store | Transaction index  | 294.63 MiB |  8581639 |
| Key-Value store | Bloombit index     | 146.44 MiB |  2093056 |
| Key-Value store | Contract codes     | 247.72 MiB |    43117 |
| Key-Value store | Trie nodes         | 4.27 GiB   | 42450160 |
| Key-Value store | Trie preimages     | 46.79 MiB  |   672798 |
| Key-Value store | Account snapshot   | 0.00 B     |        0 |
| Key-Value store | Storage snapshot   | 0.00 B     |        0 |
| Key-Value store | Clique snapshots   | 2.23 MiB   |     2433 |
| Key-Value store | Singleton metadata | 185.00 B   |        6 |
| Ancient store   | Headers            | 1.34 GiB   |  4096595 |
| Ancient store   | Bodies             | 1.94 GiB   |  4096595 |
| Ancient store   | Receipt lists      | 1.01 GiB   |  4096595 |
| Ancient store   | Difficulties       | 39.03 MiB  |  4096595 |
| Ancient store   | Block number->hash | 148.46 MiB |  4096595 |
| Light client    | CHT trie nodes     | 0.00 B     |        0 |
| Light client    | Bloom trie nodes   | 0.00 B     |        0 |
+-----------------+--------------------+------------+----------+
|                         TOTAL        | 10.04 GIB  |          |
+-----------------+--------------------+------------+----------+

Checking the c (contract codes):

[user@work go-ethereum]$ ./build/bin/geth --goerli db inspect 0x63 0x00
...
+-----------------+--------------------+------------+---------+
|    DATABASE     |      CATEGORY      |    SIZE    |  ITEMS  |
+-----------------+--------------------+------------+---------+
| Key-Value store | Headers            | 0.00 B     |       0 |
| Key-Value store | Bodies             | 0.00 B     |       0 |
| Key-Value store | Receipt lists      | 0.00 B     |       0 |
| Key-Value store | Difficulties       | 0.00 B     |       0 |
| Key-Value store | Block number->hash | 0.00 B     |       0 |
| Key-Value store | Block hash->number | 0.00 B     |       0 |
| Key-Value store | Transaction index  | 0.00 B     |       0 |
| Key-Value store | Bloombit index     | 0.00 B     |       0 |
| Key-Value store | Contract codes     | 247.72 MiB |   43117 |
| Key-Value store | Trie nodes         | 17.12 MiB  |  165946 |
| Key-Value store | Trie preimages     | 0.00 B     |       0 |
| Key-Value store | Account snapshot   | 0.00 B     |       0 |
| Key-Value store | Storage snapshot   | 0.00 B     |       0 |
| Key-Value store | Clique snapshots   | 2.23 MiB   |    2433 |
| Key-Value store | Singleton metadata | 0.00 B     |       0 |
| Ancient store   | Headers            | 1.34 GiB   | 4096595 |
| Ancient store   | Bodies             | 1.94 GiB   | 4096595 |
| Ancient store   | Receipt lists      | 1.01 GiB   | 4096595 |
| Ancient store   | Difficulties       | 39.03 MiB  | 4096595 |
| Ancient store   | Block number->hash | 148.46 MiB | 4096595 |
| Light client    | CHT trie nodes     | 0.00 B     |       0 |
| Light client    | Bloom trie nodes   | 0.00 B     |       0 |
+-----------------+--------------------+------------+---------+
|                         TOTAL        |  4.72 GIB  |         |
+-----------------+--------------------+------------+---------+

Checking clique- prefix:

[user@work go-ethereum]$  ./build/bin/geth --goerli db inspect 0x636c697175652d 0x00
...
+-----------------+--------------------+------------+---------+
|    DATABASE     |      CATEGORY      |    SIZE    |  ITEMS  |
+-----------------+--------------------+------------+---------+
| Key-Value store | Headers            | 0.00 B     |       0 |
| Key-Value store | Bodies             | 0.00 B     |       0 |
| Key-Value store | Receipt lists      | 0.00 B     |       0 |
| Key-Value store | Difficulties       | 0.00 B     |       0 |
| Key-Value store | Block number->hash | 0.00 B     |       0 |
| Key-Value store | Block hash->number | 0.00 B     |       0 |
| Key-Value store | Transaction index  | 0.00 B     |       0 |
| Key-Value store | Bloombit index     | 0.00 B     |       0 |
| Key-Value store | Contract codes     | 0.00 B     |       0 |
| Key-Value store | Trie nodes         | 0.00 B     |       0 |
| Key-Value store | Trie preimages     | 0.00 B     |       0 |
| Key-Value store | Account snapshot   | 0.00 B     |       0 |
| Key-Value store | Storage snapshot   | 0.00 B     |       0 |
| Key-Value store | Clique snapshots   | 2.23 MiB   |    2433 |
| Key-Value store | Singleton metadata | 0.00 B     |       0 |
| Ancient store   | Headers            | 1.34 GiB   | 4096595 |
| Ancient store   | Bodies             | 1.94 GiB   | 4096595 |
| Ancient store   | Receipt lists      | 1.01 GiB   | 4096595 |
| Ancient store   | Difficulties       | 39.03 MiB  | 4096595 |
| Ancient store   | Block number->hash | 148.46 MiB | 4096595 |
| Light client    | CHT trie nodes     | 0.00 B     |       0 |
| Light client    | Bloom trie nodes   | 0.00 B     |       0 |
+-----------------+--------------------+------------+---------+
|                         TOTAL        |  4.47 GIB  |         |
+-----------------+--------------------+------------+---------+

The last example brings down 41s to:

real    0m0.157s
user    0m0.119s
sys     0m0.029s

cmd/geth/dbcmd.go

holiman · 2021-02-09T09:59:10Z

cmd/geth/dbcmd.go

+	_, chainDb := utils.MakeChain(ctx, node, true)
+	defer chainDb.Close()


If possible, open in read-only

cmd/geth/dbcmd.go

…to dbcommands

holiman · 2021-02-10T08:42:22Z

Fixed most of the concerns, with the exception of not opening the db in readonly mode for inspect, which would be quite an invasive change.

put, get, delete

Check unclean shutdown marker (note: now in readonly)

[user@work go-ethereum]$ ./build/bin/geth --goerli  db get 0x756e636c65616e2d73687574646f776e
INFO [02-10|09:26:06.705] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:26:06.705] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:26:06.706] Set global gas cap                       cap=25000000
INFO [02-10|09:26:06.706] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=16.00MiB handles=500 readonly=true
key 0x756e636c65616e2d73687574646f776e:
        0xc780c58460217a8d

Overwrite unclean shutdown marker

[user@work go-ethereum]$ ./build/bin/geth --goerli  db put 0x756e636c65616e2d73687574646f776e 0x1337
INFO [02-10|09:27:41.871] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:27:41.871] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:27:41.871] Set global gas cap                       cap=25000000
INFO [02-10|09:27:41.871] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
INFO [02-10|09:27:41.913] Opened ancient database                  database=/home/user/.ethereum/goerli/geth/chaindata/ancient
Previous value:
0xc780c58460217a8d
INFO [02-10|09:27:41.914] Freezer shutting down

Check it

[user@work go-ethereum]$ ./build/bin/geth --goerli  db get 0x756e636c65616e2d73687574646f776e
INFO [02-10|09:28:14.451] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:28:14.451] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:28:14.452] Set global gas cap                       cap=25000000
INFO [02-10|09:28:14.452] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=16.00MiB handles=500 readonly=true
key 0x756e636c65616e2d73687574646f776e:
        0x1337

Put it back

[user@work go-ethereum]$ ./build/bin/geth --goerli  db put 0x756e636c65616e2d73687574646f776e 0xc780c58460217a8d
INFO [02-10|09:28:46.107] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:28:46.107] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:28:46.108] Set global gas cap                       cap=25000000
INFO [02-10|09:28:46.108] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
INFO [02-10|09:28:46.156] Opened ancient database                  database=/home/user/.ethereum/goerli/geth/chaindata/ancient
Previous value:
0x1337
INFO [02-10|09:28:46.156] Freezer shutting down

Delete it

[user@work go-ethereum]$ ./build/bin/geth --goerli  db delete 0x756e636c65616e2d73687574646f776e
INFO [02-10|09:33:02.186] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:33:02.186] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:33:02.186] Set global gas cap                       cap=25000000
INFO [02-10|09:33:02.188] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
INFO [02-10|09:33:02.222] Opened ancient database                  database=/home/user/.ethereum/goerli/geth/chaindata/ancient
INFO [02-10|09:33:02.223] Freezer shutting down

Verify delete:

[user@work go-ethereum]$ ./build/bin/geth --goerli  db get  0x756e636c65616e2d73687574646f776e
INFO [02-10|09:33:05.494] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:33:05.495] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:33:05.495] Set global gas cap                       cap=25000000
INFO [02-10|09:33:05.496] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=16.00MiB handles=500 readonly=true
INFO [02-10|09:33:05.508] Get operation failed                     error="leveldb: not found"
leveldb: not found

Restore it

[user@work go-ethereum]$ ./build/bin/geth --goerli  db put 0x756e636c65616e2d73687574646f776e 0xc780c58460217a8d
INFO [02-10|09:33:12.390] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:33:12.390] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:33:12.390] Set global gas cap                       cap=25000000
INFO [02-10|09:33:12.391] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
INFO [02-10|09:33:12.437] Opened ancient database                  database=/home/user/.ethereum/goerli/geth/chaindata/ancient
INFO [02-10|09:33:12.438] Freezer shutting down

inspect

Added uncleanShutdownMarker to the stats. This is still not readonly, since it's relying on the whole make utils.MakeChain(ctx, stack, true).

[user@work go-ethereum]$ ./build/bin/geth --goerli  db inspect
INFO [02-10|09:31:40.672] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:31:40.672] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:31:40.673] Set global gas cap                       cap=25000000
INFO [02-10|09:31:40.673] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
INFO [02-10|09:31:40.706] Opened ancient database                  database=/home/user/.ethereum/goerli/geth/chaindata/ancient
INFO [02-10|09:31:40.711] Persisted trie from memory database      nodes=361 size=51.17KiB time="742.775µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
...
| Key-Value store | Singleton metadata | 139.00 B   |       4 |
| Key-Value store | Shutdown metadata  | 24.00 B    |       1 |
| Ancient store   | Headers            | 1.34 GiB   | 4095953 |

stats

Note readonly

[user@work go-ethereum]$ ./build/bin/geth --goerli  db stats
INFO [02-10|09:32:28.114] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:32:28.114] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:32:28.115] Set global gas cap                       cap=25000000
INFO [02-10|09:32:28.115] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=16.00MiB handles=500 readonly=true
Compactions
 Level |   Tables   |    Size(MB)   |    Time(sec)  |    Read(MB)   |   Write(MB)
-------+------------+---------------+---------------+---------------+---------------
   0   |          2 |       0.00060 |       0.00000 |       0.00000 |       0.00000
   1   |          2 |       0.00059 |       0.00000 |       0.00000 |       0.00000
   2   |        320 |     655.45603 |       0.00000 |       0.00000 |       0.00000
-------+------------+---------------+---------------+---------------+---------------
 Total |        324 |     655.45723 |       0.00000 |       0.00000 |       0.00000

Read(MB):0.03220 Write(MB):0.00000

compact

[user@work go-ethereum]$ ./build/bin/geth --goerli  db compact
INFO [02-10|09:37:32.313] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:37:32.313] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:37:32.313] Set global gas cap                       cap=25000000
INFO [02-10|09:37:32.313] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
Compactions
 Level |   Tables   |    Size(MB)   |    Time(sec)  |    Read(MB)   |   Write(MB)
-------+------------+---------------+---------------+---------------+---------------
   0   |          3 |       0.00101 |       0.00000 |       0.00000 |       0.00000
   1   |          2 |       0.00059 |       0.00000 |       0.00000 |       0.00000
   2   |        320 |     655.45603 |       0.00000 |       0.00000 |       0.00000
-------+------------+---------------+---------------+---------------+---------------
 Total |        325 |     655.45763 |       0.00000 |       0.00000 |       0.00000

Read(MB):0.03209 Write(MB):0.03208
INFO [02-10|09:37:32.348] Triggering compaction
Compactions
 Level |   Tables   |    Size(MB)   |    Time(sec)  |    Read(MB)   |   Write(MB)
-------+------------+---------------+---------------+---------------+---------------
   1   |          0 |       0.00000 |       0.01392 |       0.00160 |       0.00060
   2   |        318 |     655.34744 |       2.93187 |     186.54778 |     186.43859
-------+------------+---------------+---------------+---------------+---------------
 Total |        318 |     655.34744 |       2.94579 |     186.54938 |     186.43919

Read(MB):183.20840 Write(MB):186.48059
INFO [02-10|09:37:35.314] Closing db
INFO [02-10|09:37:35.316] Exiting

…22014)

This was referenced Dec 14, 2020

cmd/geth: add ability to get/put raw db values #21991

Closed

cmd/geth: add dbstats command #21887

Closed

holiman force-pushed the dbcommands branch from b6ada7e to 321c315 Compare December 14, 2020 08:56

holiman force-pushed the dbcommands branch from cde4bce to 1718fa1 Compare December 26, 2020 17:49

Olo23 approved these changes Dec 26, 2020

View reviewed changes

gballet added the status:triage label Jan 19, 2021

holiman changed the title ~~cmd/geth: add db commands: stats,compact,put,get~~ cmd/geth: add db commands: stats, compact, put, get, delete Jan 26, 2021

holiman force-pushed the dbcommands branch from 1718fa1 to b479f53 Compare February 1, 2021 12:57

cmd/geth: add db commands: stats, compact, put, get, delete

23d9b0f

holiman force-pushed the dbcommands branch from b479f53 to 23d9b0f Compare February 1, 2021 16:57

holiman added 2 commits February 2, 2021 14:24

cmd/geth: move db commands into a subcommand

17fa796

internal/flags: show argsusage

fc40e84

cmd/geth: make db inspect accept prefix+start arguments

2a993fe

holiman requested review from karalabe and rjl493456442 as code owners February 8, 2021 12:57