core/vm: add subgroup checks for mul/mulexp for G1/G2

[MariusVanDerWijden]

Adds the subgroup checks as required by the current spec: https://github.com/ethereum/EIPs/pull/8456/files

[holiman]

Generally, this seems un-optimal to me (all these cases). We perform both these calls, with IsOnCurve done from inside decodePointGX.

// IsOnCurve returns true if p in on the curve
func (p *G1Affine) IsOnCurve() bool {
	var point G1Jac
	point.FromAffine(p)
	return point.IsOnCurve() // call this function to handle infinity point
}

// IsInSubGroup returns true if p is in the correct subgroup, false otherwise
func (p *G1Affine) IsInSubGroup() bool {
	var _p G1Jac
	_p.FromAffine(p)
	return _p.IsInSubGroup()
}

Seems like it should be faster to do

func (p *G1Affine) IsOnCurveAndInSubGroup() bool {
	var point G1Jac
	point.FromAffine(p)
	point.IsOnCurve() && point.IsInSubGroup()
}

?

[holiman]

Maybe (probably?) it doesn't make any difference

[MariusVanDerWijden]

It makes very little difference I think. The subgroup check is the heaviest, moving from affine to jac is very quick and the curve checks are very quick as well

[holiman]

LGTM