[strongSwan] IKEv1 Pubkey Auth Fails from Windows to Linux

[Tobias Brunner]

Hi Quinn,

>> Anyway, doing this is definitely not recommended.
> 
> Understood.  I generated another key pair to reproduce the test, and
> I am getting the same results even when a different key pair is used
> on the initiating system (e.g. Windows initiation fails, Red Hat
> initiation succeeds).

OK, I found the issue.  The problem is that Windows, at least as
initiator, creates IKEv1 ECDSA signatures incorrectly if the negotiated
integrity algorithm does not match the one associated with the ECDSA
authentication method.  RFC 4754 defines three ECDSA authentication
methods for use in IKEv1 (also listed at [1]):

   9  ECDSA with SHA-256 on the P-256 curve
  10  ECDSA with SHA-384 on the P-384 curve
  11  ECDSA with SHA-512 on the P-521 curve

So each authentication method is tied to a specific hash algorithm.  RFC
2409, section 5.1 says the following about signature algorithms with
this property:

  In general the signature will be over HASH_I and HASH_R as above
  using the negotiated prf, or the HMAC version of the negotiated hash
  function (if no prf is negotiated). However, this can be overridden
  for construction of the signature if the signature algorithm is tied
  to a particular hash algorithm (e.g. DSS is only defined with SHA's
  160 bit output). In this case, the signature will be over HASH_I and
  HASH_R as above, except using the HMAC version of the hash algorithm
  associated with the signature method.  The negotiated prf and hash
  function would continue to be used for all other prescribed pseudo-
  random functions.

So when using ECDSA-384, as in your case, HASH_I|R are to be created
with HMAC-SHA2-384, the negotiated hash/PRF algorithm (in your case
SHA2-256/HMAC-SHA2-256) is only used for other purposes within IKEv1.

Unfortunately, that's not what Windows does as initiator.  Instead, it
always uses the negotiated PRF to create HASH_I|R even if it
authenticates itself with ECDSA, which obviously causes the verification
by strongSwan to fail.  For some reason Windows seems to do this
correctly if it acts as responder (I have not been able to verify this,
though).

You can fix this by changing the Main Mode proposals configured via
`netsh advfirewall set global mainmode mmsecmethods`.  The proposal
that's finally selected has to include the matching hash algorithm
(SHA2-256 for ECDSA-256 or SHA2-384 for ECDSA-384).  So to authenticate
with ECDSA-384 either configure a single proposal with SHA2-384 e.g.
ecdhp384:aes192-sha384 or make sure the proposal list includes one and
then enforce that via e.g. ike=aes192-sha384-ecp384! in ipsec.conf.

Regards,
Tobias

[1] https://www.iana.org/assignments/ipsec-registry/ipsec-registry.xhtml