BUG: validator tries to resolve $id although there are no outside references

[joooeey]

Looking at the following MCVE:

import jsonschema

null = None

schema = {
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "tag:example.com,2022:test.schema.json",
  # "$id": "worksfinewithoutcolon",
  "$ref": "#/$defs/foo",

  "$defs": {

    "foo": {
      "properties": {
        "bar": {
          "$ref": "#/$defs/bar"
        }
      }
    },

    "bar": { }
  }
}

instance = {
  "bar": 42
}

jsonschema.validate(instance, schema)

This raises:

Full Traceback

Traceback (most recent call last):

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/validators.py", line 898, in resolve_from_url
    document = self.store[url]

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/_utils.py", line 28, in __getitem__
    return self.store[self.normalize(uri)]

KeyError: ''


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/validators.py", line 901, in resolve_from_url
    document = self.resolve_remote(url)

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/validators.py", line 1007, in resolve_remote
    with urlopen(uri) as url:

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/urllib/request.py", line 503, in open
    req = Request(fullurl, data)

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/urllib/request.py", line 322, in __init__
    self.full_url = url

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/urllib/request.py", line 348, in full_url
    self._parse()

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/urllib/request.py", line 377, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)

ValueError: unknown url type: ''


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/spyder_kernels/py3compat.py", line 356, in compat_exec
    exec(code, globals, locals)

  File "/home/lukas/Desktop/scratch/schemaref.py", line 41, in <module>
    jsonschema.validate(instance, schema, resolver=resolver)

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/validators.py", line 1108, in validate
    error = exceptions.best_match(validator.iter_errors(instance))

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/exceptions.py", line 382, in best_match
    best = next(errors, None)

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/validators.py", line 278, in iter_errors
    for error in errors:

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/_validators.py", line 332, in properties
    yield from validator.descend(

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/validators.py", line 295, in descend
    for error in self.evolve(schema=schema).iter_errors(instance):

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/validators.py", line 278, in iter_errors
    for error in errors:

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/_validators.py", line 298, in ref
    yield from validator.descend(instance, resolved)

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/validators.py", line 295, in descend
    for error in self.evolve(schema=schema).iter_errors(instance):

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/validators.py", line 278, in iter_errors
    for error in errors:

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/_validators.py", line 294, in ref
    scope, resolved = validator.resolver.resolve(ref)

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/validators.py", line 887, in resolve
    return url, self._remote_cache(url)

  File "/home/lukas/anaconda3/envs/ifos_test/lib/python3.10/site-packages/jsonschema/validators.py", line 903, in resolve_from_url
    raise exceptions.RefResolutionError(exc)

RefResolutionError: unknown url type: ''

RefResolutionError: unknown url type: ''

According to my reading of the spec, this should validate just fine. Hyperjump also finds these JSONs valid (after removing the # comment line).

However, jsonschema fails here. It looks like two things are needed for the error to happen:

• A URI in the "$id" field where jsonschema doesn't know the scheme (or a string that looks like a URI).
• Nested "$ref" statements.

According to the spec:

The $id keyword defines a URI for the schema, and the base URI that other URI references within the schema are resolved against.

jsonschema validates the file just fine if the "$id" doesn't contain a colon. I don't even understand why it tries to resolve the "$id".

Related: #313

[joooeey]

Workaround:

resolver = jsonschema.RefResolver.from_schema(schema=schema, store={"": schema})
jsonschema.validate(instance, schema, resolver=resolver)

That workaround looks problematic to me because "" is ambiguous and will stay in the store all the time. But it works for my use case.

[joooeey]

It looks like the issue lies in jsonschema.validators.create/Validator.push_scope. The function call self._urljoin_cache(self.resolution_scope, scope) reduces to:

urljoin("tag:example.com,2022:test.schema.json", "#/defs/foos")
Out[20]: '#/defs/foos'

which strips the "$id".

[Julian]

Hello there!

This, along with many many other $ref-related issues, is now finally being handled in #1049 with the introduction of a new referencing library which is fully compliant and has APIs which I hope are a lot easier to understand and customize.

The next release of jsonschema (v4.18.0) will contain a merged version of that PR, and should be released shortly in beta, and followed quickly by a regular release, assuming no critical issues are reported.

It looks from my testing like indeed this specific example works there! If you still care to, I'd love it if you tried out the beta once it is released, or certainly it'd be hugely helpful to immediately install the branch containing this work (https://github.com/python-jsonschema/jsonschema/tree/referencing) and confirm. You can in the interim find documentation for the change in a preview page here.

I'm going to close this given it indeed seems like it is addressed by #1049, but feel free to follow up with any comments. Sorry for the delay in getting to these, but hopefully this new release will bring lots of benefit!

[joooeey]

This issue does indeed appear solved in 4.18.0a4 for my code.

However, the new version throws new RefResolutionErrors in my code now that weren't there before. I'm investigating. EDIT: Looks like Python doesn't have an internet connection this time. I have no idea why.

[joooeey]

The ref resolution regression is discussed in another issue: #1089