CP-307865 accept SHA512 for custom server certs

[lindig]

A user can install server certificates to secure the connection between xapi and their API clients. So far we demanded SHA256 certificates. Accept SHA512 in addition.

The patch renames the predicate to no longer imply that we only accept SHA256.

Tested this:

openssl req -x509 -sha512 -nodes -days 365 -newkey rsa:2048 -keyout mycert.key -out mycert.crt -subj "/CN=localhost"
xe host-server-certificate-install certificate=mycert.crt private-key=mycert.key

[psafont]

The string for server_certificate_signature_not_supported in ocaml/idl/datamodel_errors.ml needs to be changed as well to mention SHA512.

A unit test needs to be added to ocaml/gencert/test_lib.mlto ensure that SHA512 signatures work.

Do we want to update the default generated certs to use SHA512 signatures as well?

[edwintorok]

Do we want to update the default generated certs to use SHA512 signatures as well?

I don't think we want to change the defaults at this time. AFAICT there is nothing wrong with SHA-256.

[stormi]

We've had two separate customers with certs issued by Sectigo who could not import them in XAPI because they were SHA-384. I'm sure they would appreciate if that were supported too.

[andyhhp]

because they were SHA-384

SHA(any)-384 is one of the idiotic digest sizes; You're doing all the work with the larger internal state size, then truncating the result.

But, it is a standard, and people do use it, so it should be accepted.

[edwintorok]

SHA(any)-384 is one of the idiotic digest sizes; You're doing all the work with the larger internal state size, then truncating the result.

But, it is a standard, and people do use it, so it should be accepted.

Wikipedia says "length extension attacks on SHA-224 and SHA-384 succeed with probability 2^−(256−224) = 2^−32 > 2^−224 and 2^−(512−384) = 2^−128 > 2^−384 respectively.", so in some contexts SHA-384 only has 128 bits of security or less. I don't know whether in the context of TLS cert signatures those attacks would be applicable or not.
But we definitely shouldn't support SHA-224, that seems too risky (luckily noone proposes to support that).

Also we do currently use SHA-384 in the TLS cipher, our 2 ciphers are ECDHE-RSA-AES256-GCM-SHA384 and ECDHE-RSA-AES128-GCM-SHA256, so it is hard to say why SHA-384 is acceptable there but not as a certificate signature.

I'm not entirely convinced that those asking for SHA-384 know it is from the SHA-2 family, as you say it is very easy to confuse it with SHA-3 (which is an entirely different algorithm).

[stormi]

I'm not entirely convinced that those asking for SHA-384 know it is from the SHA-2 family

I don't think they do. They just bought certs from a provider that did not provide SHA-256 and were stuck because of that.