upmap-remapped.py erroring on loading existing upmaps

[carlilek]

I am experiencing an issue with this tool on Rocky 9.4, with Python 3.9.18 and Reef 18.2.4. I have the rados 2.0.0 module. When I run upmap-remapped.py, I get this:

# ./upmap-remapped.py
Error loading existing upmaps

Digging around a bit, it appears to be erroring specifically on line 162:
upmaps = json.loads(osd_dump_json)['pg_upmap_items']

If I print the exception, I get:
Error loading existing upmaps: Expecting value: line 1 column 2345 (char 2344)

Now, if I put the data from line 161:
osd_dump_json = output.decode('utf-8').strip()
into a file and run jq .pg_upmap_items on it, I get valid data back.

So I'm really at a loss as to how to modify the code.

[dvanders]

hi @carlilek could you trying add your line just above upmaps = ... ?

+ osd_dump_json = output.decode('utf-8').strip()
upmaps = json...

[carlilek]

That already appears to be in the code (and it's where the error seems to originate):

# discover existing upmaps
try:
  if use_shell:
    osd_dump_json = subprocess.getoutput('ceph osd dump -f json | jq -r .')
  else:
    cmd = {"prefix": "osd dump", "format": "json"}
    ret, output, errs = cluster.mon_command(json.dumps(cmd), b'', timeout=5)
    osd_dump_json = output.decode('utf-8').strip()
  upmaps = json.loads(osd_dump_json)['pg_upmap_items']
except ValueError:
  eprint('Error loading existing upmaps')
  sys.exit(1)

[JoshuaGabriel]

@carlilek hello, can we get your ceph osd dump --f json to see if we can get a fix for this

[carlilek]

Here you go,
The pythondump.json is the osd_dump_json variable from the script, while the dump.json is the ceph osd dump -f json output.

pythondump.json
dump.json

[JoshuaGabriel]

with Error loading existing upmaps: Expecting value: line 1 column 2345 (char 2344)

root@ceph-test-2:~/testing# cut -c 2288-2348 pythondump.json
"read_balance":{"score_acting":inf,"score_stable":inf,"optima

the inf characters coming from the read balancer seems to be the offender here I think. Could you try with this regex solution to parse it before feeding to json.loads() @carlilek

+ import re 
...
...

+ osd_dump_json = re.sub(r':\s*inf\b', ': "inf"', osd_dump_json) # parse inf to string
upmaps = json.loads(osd_dump_json)['pg_upmap_items']

[carlilek]

Yup, that made it run through. Thank you!

[phineas42]

I experienced this same issue, with a ".mgr" pool reporting "read_balance_score inf", reported as DBL_MAX in the jq output, and unable to be loaded through json.loads.

[dvanders]

I'm testing a solution for this using the yaml library as a more tolerant json parser.

[JoshuaGabriel]

There is also the jq library for parsing, https://pypi.org/project/jq/. IIRC jq on the cli handles the inf keyword correctly.

[afortiorama]

I experience the same the DBL_MAX is unquoted and rejected. It is also in portion of the json that is loaded but not necesary. Perhaps restricting from the start to what is used is a good strategy to minimize problems from json that haven't been sanitized.