Certificate verification fails with RSAE-PSS, SHA512 and a 1024-bit RSA key. #16167

dwmw2 · 2021-07-27T14:13:21Z

When RSA-PSS is used with a 1024-bit RSA key and SHA512, the salt length ends up being 62 bytes.

However, we end up calling RSA_verify_PKCS1_PSS_mgf1() with the sLen argument set to -1 which means it expects the salt length to be exactly the hash length. Thus it complains:

apps/openssl s_server -accept 8443 -cert ~/git/openconnect/tests/certs/server-cert.pem -key ~/git/openconnect/tests/certs/server-key.pem  -tls1_3 -Verify 1
verify depth is 1, must return a certificate
Using default temp DH parameters
ACCEPT
depth=0 CN = A user, UID = test
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = A user, UID = test
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = A user, UID = test
verify return:1
ERROR
verify error:unable to verify the first certificate
C0C1927B127F0000:error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:crypto/store/store_register.c:237:scheme=file
C0C1927B127F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:269:calling stat(/usr/local/ssl/certs)
C0C1927B127F0000:error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:crypto/store/store_register.c:237:scheme=file
C0C1927B127F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:269:calling stat(/usr/local/ssl/certs)
C0C1927B127F0000:error:0200006D:rsa routines:RSA_verify_PKCS1_PSS_mgf1:data too large:crypto/rsa/rsa_pss.c:90:
C0C1927B127F0000:error:1C880004:Provider routines:rsa_verify:RSA lib:providers/implementations/signature/rsa_sig.c:807:
C0C1927B127F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:ssl/statem/statem_lib.c:536:

The client in this case is OpenConnect, and because all crypto libraries hate their users I actually had to write my own PSS padding in order to get keys from a TPM to work with TLSv1.3: https://gitlab.com/openconnect/openconnect/-/commit/ff367965fcc13f6c1ba7fbda7a49d1467f1b39de

This client does authenticate to gnutls-serv using TLSv1.3 and RSA-PSS-RSAE-SHA512.

I had to hack my client to pretend the key doesn't support SHA256 or SHA384 in order to force SHA512.

The text was updated successfully, but these errors were encountered:

dwmw2 · 2021-07-27T15:30:44Z

Trying to reproduce with just openssl:

$ yes '' | openssl req -newkey rsa:1024 -nodes   -x509 -keyout rsa1k.pem -out rsa1k.pem
$ yes '' | openssl req -newkey rsa:2048 -nodes   -x509 -keyout rsa2k.pem -out rsa2k.pem
$ openssl s_client -cert rsa2k.pem -key rsa2k.pem -tls1_3 -sigalgs RSA-PSS+SHA512 localhost:8443 
$ openssl s_client -cert rsa1k.pem -key rsa1k.pem -tls1_3 -sigalgs RSA-PSS+SHA512 localhost:8443

The client doesn't even try.

$ LD_LIBRARY_PATH=`pwd` apps/openssl s_client -cert ~/rsa1k.pem -key ~/rsa1k.pem -tls1_3 -sigalgs RSA-PSS+SHA512 -debug  localhost:8443
CONNECTED(00000003)
write to 0x22388a0 [0x2352610] (199 bytes => 199 (0xC7))
0000 - 16 03 01 00 c2 01 00 00-be 03 03 b5 4a e1 06 4a   ............J..J
0010 - 81 3e 23 1d 7b 6c 7c 51-cf c8 4e 02 fc 3a 7f 51   .>#.{l|Q..N..:.Q
0020 - 0d 82 6f 25 67 b0 41 a7-56 1d 83 20 70 52 b3 d9   ..o%g.A.V.. pR..
0030 - b9 a1 05 dc e0 a6 5f d1-96 e4 93 9d 52 cc d5 4b   ......_.....R..K
0040 - d1 34 fa 62 82 a9 e7 24-a6 d1 91 44 00 08 13 02   .4.b...$...D....
0050 - 13 03 13 01 00 ff 01 00-00 6d 00 0b 00 04 03 00   .........m......
0060 - 01 02 00 0a 00 16 00 14-00 1d 00 17 00 1e 00 19   ................
0070 - 00 18 01 00 01 01 01 02-01 03 01 04 00 23 00 00   .............#..
0080 - 00 16 00 00 00 17 00 00-00 0d 00 04 00 02 08 06   ................
0090 - 00 2b 00 03 02 03 04 00-2d 00 02 01 01 00 33 00   .+......-.....3.
00a0 - 26 00 24 00 1d 00 20 53-e1 64 1e 0a a9 a1 11 bd   &.$... S.d......
00b0 - f9 7f 14 bc 69 df c4 b9-8d 20 1c 81 01 6b 14 b6   ....i.... ...k..
00c0 - a1 37 c3 b8 a9 20 3d                              .7... =
read from 0x22388a0 [0x23493f3] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 7a                                    ....z
read from 0x22388a0 [0x23493f8] (122 bytes => 122 (0x7A))
0000 - 02 00 00 76 03 03 bb 52-c9 62 f1 09 a8 06 a1 0c   ...v...R.b......
0010 - aa ff 23 d9 b2 93 50 bb-3e f5 72 f8 00 0e db 2a   ..#...P.>.r....*
0020 - c0 b6 69 86 2e 56 20 70-52 b3 d9 b9 a1 05 dc e0   ..i..V pR.......
0030 - a6 5f d1 96 e4 93 9d 52-cc d5 4b d1 34 fa 62 82   ._.....R..K.4.b.
0040 - a9 e7 24 a6 d1 91 44 13-02 00 00 2e 00 33 00 24   ..$...D......3.$
0050 - 00 1d 00 20 70 b3 ae 6d-b8 be fc 08 17 c7 59 90   ... p..m......Y.
0060 - 4e bd dc e6 79 ff 8b 59-11 82 d6 aa f5 b7 3b 07   N...y..Y......;.
0070 - c2 f0 90 34 00 2b 00 02-03 04                     ...4.+....
read from 0x22388a0 [0x23493f3] (5 bytes => 5 (0x5))
0000 - 14 03 03 00 01                                    .....
read from 0x22388a0 [0x23493f8] (1 bytes => 1 (0x1))
0000 - 01                                                .
read from 0x22388a0 [0x23493f3] (5 bytes => 5 (0x5))
0000 - 17 03 03 00 17                                    .....
read from 0x22388a0 [0x23493f8] (23 bytes => 23 (0x17))
0000 - 2c 97 13 97 d7 8c dd 04-29 03 77 30 a6 1c 93 9a   ,.......).w0....
0010 - 70 e2 01 3b 48 9c 29                              p..;H.)
Can't use SSL_get_servername
read from 0x22388a0 [0x23493f3] (5 bytes => 5 (0x5))
0000 - 17 03 03 00 44                                    ....D
read from 0x22388a0 [0x23493f8] (68 bytes => 68 (0x44))
0000 - 42 1c 70 f8 5c 54 d1 65-5b e3 e2 88 7e 58 71 91   B.p.\T.e[...~Xq.
0010 - f7 5a 61 6a 4a 8f 39 71-9b a7 39 46 4e 98 98 21   .ZajJ.9q..9FN..!
0020 - 1e 12 89 52 aa fa 53 43-61 98 69 a0 1a 41 13 dd   ...R..SCa.i..A..
0030 - 28 05 3b b4 d9 e5 c9 dd-e3 b0 7c 34 47 48 32 74   (.;.......|4GH2t
0040 - 6b 75 e5 e2                                       ku..
read from 0x22388a0 [0x23493f3] (5 bytes => 5 (0x5))
0000 - 17 03 03 03 b3                                    .....
read from 0x22388a0 [0x23493f8] (947 bytes => 947 (0x3B3))
0000 - 42 46 c2 9f bf 00 40 da-a4 22 65 1b 7b 65 ca 49   BF....@.."e.{e.I
0010 - 9f 55 fa 2d b2 00 1b da-58 44 8c 76 d3 2e fc 35   .U.-....XD.v...5
0020 - eb 95 cc 0d f2 12 b2 17-c6 6d 9c 74 7a fa 18 bf   .........m.tz...
0030 - 03 93 07 34 43 91 2a 24-f4 f8 11 db 03 d0 2a 97   ...4C.*$......*.
0040 - 03 26 0f 74 02 d3 91 37-8f bb c8 7a 70 0c de b1   .&.t...7...zp...
0050 - 4c d9 8c 87 d7 ba 07 3d-94 71 02 13 f1 06 b2 76   L......=.q.....v
0060 - 5c 56 37 16 c6 36 e4 5a-5e aa 68 81 f2 52 55 3c   \V7..6.Z^.h..RU<
0070 - ab d7 c5 ed 32 dd a2 e8-b8 d5 6d 11 c6 8c 57 a0   ....2.....m...W.
0080 - 0f 5a 30 53 48 9d c1 b4-e3 e0 56 db fc 48 77 e6   .Z0SH.....V..Hw.
0090 - 3e 7d 0e ec 27 2f 71 8a-61 f9 1d 5c c4 06 32 52   >}..'/q.a..\..2R
00a0 - b3 c0 90 1d 94 e1 d5 73-cb a9 29 de 57 09 dc 06   .......s..).W...
00b0 - 25 0b dd ad 1b f2 41 21-0a 19 cc 0a 6e d2 b7 c8   %.....A!....n...
00c0 - d1 b5 62 2b 8e 7d 43 ea-92 d2 12 c1 36 97 aa 0f   ..b+.}C.....6...
00d0 - e5 65 41 a6 5a ab a2 47-cb 3f 05 27 e0 e0 25 06   .eA.Z..G.?.'..%.
00e0 - 45 a5 ec 01 65 d8 95 8e-83 a4 4c 7c 20 5c c6 30   E...e.....L| \.0
00f0 - b1 ce 8a 73 c0 c4 39 b3-e1 9d 34 13 35 48 8f c5   ...s..9...4.5H..
0100 - 0c 3b ff bf d0 52 80 73-f5 1f 21 3c 87 80 e3 79   .;...R.s..!<...y
0110 - db 01 f6 8c 60 fd 6f dd-b5 45 b9 ec 0e fb fb a4   ....`.o..E......
0120 - 5b 4b 7f 67 b7 26 65 18-4b 5c cc 4b d2 e5 ce dd   [K.g.&e.K\.K....
0130 - b2 f9 bc 72 2d ed f3 a9-c8 96 22 90 ff c6 a2 69   ...r-....."....i
0140 - 9a 50 aa e5 c1 52 f0 dc-f7 b7 76 86 86 c3 ea dd   .P...R....v.....
0150 - 50 b4 ee e0 05 42 37 35-5d f6 53 3c 6d 78 d7 d8   P....B75].S<mx..
0160 - 1f 11 1d ee a9 e6 3e f6-ed b0 69 bb f6 1b 58 d8   ......>...i...X.
0170 - 1b 4a 40 b9 07 53 7a 68-5e fc 33 2b 7d cd 93 93   .J@..Szh^.3+}...
0180 - e7 e5 1f 35 25 9c 02 3c-ea d2 bd 9f 95 03 23 29   ...5%..<......#)
0190 - ad 60 62 cb 95 21 77 b9-ab 9a 39 53 08 45 64 42   .`b..!w...9S.EdB
01a0 - 86 f1 0d 4b 7e 8b 39 02-e8 86 bf 33 53 0d d3 28   ...K~.9....3S..(
01b0 - 44 0d 10 58 94 ce 5c fb-56 3d 21 9c 67 27 86 79   D..X..\.V=!.g'.y
01c0 - 4e d1 b5 28 9f 96 ce 03-a4 cc 83 16 0d 64 7d d1   N..(.........d}.
01d0 - 53 5b 55 a7 83 9a b0 b2-bd f1 d5 45 db 3f ab 76   S[U........E.?.v
01e0 - 56 d4 93 00 3a 5f e2 2b-64 e3 84 71 4c 5a c0 d8   V...:_.+d..qLZ..
01f0 - c6 58 e1 0c 4b 54 c8 f2-1c 8c 0c 75 f2 05 da 6e   .X..KT.....u...n
0200 - f9 fe 12 de fd 19 5c 22-5d c9 8b 53 7d 11 d2 d9   ......\"]..S}...
0210 - 80 5b 2b b6 8b e6 79 a3-6d 0e b1 77 8e 4a 7d 95   .[+...y.m..w.J}.
0220 - 98 d7 14 e0 df 14 50 3d-31 f3 2b 3e c5 12 64 ae   ......P=1.+>..d.
0230 - 9f 92 f0 06 82 e8 26 26-bd e5 3b c4 55 e1 be f1   ......&&..;.U...
0240 - 77 73 4f dc 47 32 f2 26-e9 96 3b 8e b0 d6 cd 42   wsO.G2.&..;....B
0250 - 1b 64 c3 8e af ed b6 5a-2e e2 c0 66 0a 8e 66 f4   .d.....Z...f..f.
0260 - af 7e a0 49 6f 2f fd 5d-cb e0 e8 56 1c 1a 3d 73   .~.Io/.]...V..=s
0270 - 17 2a 62 59 65 eb a9 77-1c bc 8d 72 e2 98 85 9e   .*bYe..w...r....
0280 - 69 a7 a6 a6 e8 6e e1 26-ac ba 3e 87 c9 05 cf 22   i....n.&..>...."
0290 - a5 45 ef 73 93 04 f5 c7-7d 2a 01 1c cd be 64 53   .E.s....}*....dS
02a0 - eb c5 58 e7 47 fe ea 43-5c ab fc 5c 09 f3 12 1f   ..X.G..C\..\....
02b0 - b5 4a 2d 03 21 af 4b 3f-b3 23 51 ac fc a6 69 12   .J-.!.K?.#Q...i.
02c0 - d9 08 c6 05 a4 a6 86 76-d4 b7 02 13 74 1b cb 46   .......v....t..F
02d0 - fd c9 06 9c 42 d7 0f cd-dc 0a ee d7 69 00 e0 e0   ....B.......i...
02e0 - 16 b6 c2 8b 80 ab 0c 33-7e 77 3a ac 3f 2c 8c 27   .......3~w:.?,.'
02f0 - 89 af 7f dd 6e 4c 87 73-0a 04 aa 7f c8 ef 9b b0   ....nL.s........
0300 - a2 4e 98 96 c6 eb d0 c6-e8 0f ec bb 1d 29 3b 02   .N...........);.
0310 - 64 e2 23 f9 ed 8d 78 0c-44 66 12 13 f9 ac 78 f4   d.#...x.Df....x.
0320 - 4d 1c b5 34 e1 c1 ba f1-60 a6 a7 75 83 57 5b aa   M..4....`..u.W[.
0330 - 01 85 c9 96 57 bc f6 1a-eb 35 02 b1 2a 48 18 6c   ....W....5..*H.l
0340 - ac 06 43 21 95 21 03 b3-db aa ca 29 7b 66 13 23   ..C!.!.....){f.#
0350 - ef 78 e1 36 31 2f fd 16-43 9d 4a 9c ff 2a 5b 89   .x.61/..C.J..*[.
0360 - 8c 51 e4 41 5b 89 be 8f-de 6d b5 a5 99 7e be 5b   .Q.A[....m...~.[
0370 - c0 90 7c fb ab 9f 8d 57-ef af 8d f9 ca e3 5a 9a   ..|....W......Z.
0380 - 6f be ee 31 fc 5f 55 d4-7a 2f 91 e2 80 be bb a1   o..1._U.z/......
0390 - d1 d6 53 32 aa f1 ec fc-01 c9 0d 66 e7 ba 7c cd   ..S2.......f..|.
03a0 - 32 b0 08 64 92 a7 94 47-cf 6b 7c 69 10 f2 dc ad   2..d...G.k|i....
03b0 - 4e ac 0f                                          N..
depth=0 CN = localhost
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = localhost
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = localhost
verify return:1
read from 0x22388a0 [0x23493f3] (5 bytes => 5 (0x5))
0000 - 17 03 03 01 49                                    ....I
read from 0x22388a0 [0x23493f8] (329 bytes => 329 (0x149))
0000 - a6 11 01 2b f9 f8 4f 14-04 af 05 bb af a7 34 33   ...+..O.......43
0010 - f1 ff ca e9 a9 a2 9c 54-64 7a 61 9f 21 42 f8 c5   .......Tdza.!B..
0020 - 28 dc 2a 69 fd 3c 8d 45-e5 ed 3b b4 8c 8a 09 70   (.*i.<.E..;....p
0030 - 0b ec c3 6e 9e 01 76 98-12 c1 56 8b 5f 4f 7f cd   ...n..v...V._O..
0040 - e8 7d b9 b3 ec 49 c2 df-5d 78 c0 f5 94 e6 96 d0   .}...I..]x......
0050 - 37 f1 f5 ed 5a 6d fb f0-54 f0 5d 0c 9f 4c 26 93   7...Zm..T.]..L&.
0060 - 2b eb cf fc 2e 89 f1 9b-69 69 86 08 ef 43 b8 cc   +.......ii...C..
0070 - cc ef 88 eb d0 4a 4d 45-ae a2 28 71 44 43 48 9a   .....JME..(qDCH.
0080 - 87 4a ca 5c d1 bb 9d 16-dd 80 03 71 44 f1 bd ad   .J.\.......qD...
0090 - 2c 77 11 4b 0c fd 67 b2-be fd f9 9f 7c 43 8b 44   ,w.K..g.....|C.D
00a0 - c6 b1 c5 91 c8 42 e9 7f-af a4 13 a3 37 14 d7 f6   .....B......7...
00b0 - 53 cf 60 eb 56 b6 b6 18-d0 19 64 a9 b9 9e 14 ac   S.`.V.....d.....
00c0 - b8 12 6e b1 28 7b e1 1a-dc d4 62 1f 75 af 2e 1e   ..n.({....b.u...
00d0 - 12 ed 36 f2 9d 20 ff 94-5c 49 3a 19 ee d6 02 65   ..6.. ..\I:....e
00e0 - 27 2f 48 3f 59 20 23 0e-fc e3 e4 03 24 6f 52 b7   '/H?Y #.....$oR.
00f0 - 57 01 03 a2 f3 95 53 76-f6 e9 0d f9 5c aa 54 d2   W.....Sv....\.T.
0100 - 91 cf a2 d6 c1 c5 2d d1-1d 73 da 8d af 26 b0 2e   ......-..s...&..
0110 - 77 65 66 8c 4e 9d 80 eb-a5 19 e3 9a ac c9 ef e8   wef.N...........
0120 - af fa d5 98 d7 65 ff b5-b6 64 33 8c c6 35 7b e7   .....e...d3..5{.
0130 - c3 f8 37 33 cb 8e a4 9f-33 7f c4 00 34 95 07 be   ..73....3...4...
0140 - 1f 2d 2b a9 31 b2 7d f0-13                        .-+.1.}..
read from 0x22388a0 [0x23493f3] (5 bytes => 5 (0x5))
0000 - 17 03 03 00 45                                    ....E
read from 0x22388a0 [0x23493f8] (69 bytes => 69 (0x45))
0000 - 4a 98 ae fb 25 2e 76 1c-d4 81 17 6f 00 ff 4a 00   J...%.v....o..J.
0010 - ff 0f 9c f8 12 85 39 f9-70 a1 7e f2 05 d4 05 1b   ......9.p.~.....
0020 - d1 30 8b 86 38 01 24 cc-d9 22 fb 6f b4 b8 0c 8d   .0..8.$..".o....
0030 - c4 ac 99 a6 d1 3c ac ce-1c 60 a0 6f c9 63 aa 2f   .....<...`.o.c./
0040 - 61 89 64 6e 26                                    a.dn&
write to 0x22388a0 [0x2352610] (110 bytes => 110 (0x6E))
0000 - 14 03 03 00 01 01 17 03-03 00 19 aa 0e 1b 9d 13   ................
0010 - fc 4b 16 4c 73 6e 2a c9-6c 52 59 5e 4c bd c8 5b   .K.Lsn*.lRY^L..[
0020 - e5 7f 75 22 17 03 03 00-45 01 ff 83 e7 2a 57 dc   ..u"....E....*W.
0030 - b5 6d 22 56 4a b8 50 64-d5 ec 7f d4 88 82 59 a3   .m"VJ.Pd......Y.
0040 - 39 56 76 dd 7e 4f e4 83-34 ae aa 76 8d 28 db 2d   9Vv.~O..4..v.(.-
0050 - 7e d0 96 a1 9e 31 40 61-b9 45 a4 5a 29 29 8a f0   ~....1@a.E.Z))..
0060 - 05 b7 8b ac 6d 9f 92 9e-85 dc 03 9e e5 7c         ....m........|
---
Certificate chain
 0 s:CN = localhost
   i:CN = CA
   a:PKEY: rsaEncryption, 2432 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jul  6 14:51:29 2013 GMT; NotAfter: May 15 14:51:29 2023 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDkTCCAkmgAwIBAgIEUdgu8DANBgkqhkiG9w0BAQsFADANMQswCQYDVQQDEwJD
QTAiGA8yMDEzMDcwNjE0NTEyOVoYDzIwMjMwNTE1MTQ1MTI5WjAUMRIwEAYDVQQD
Ewlsb2NhbGhvc3QwggFSMA0GCSqGSIb3DQEBAQUAA4IBPwAwggE6AoIBMQCnOivs
PxSwLBn28W6QHb+OqfbpcIQJh/NQ81/DlFD6LGTWV4BY4Zb87tC9BBV+X3+lM/j8
u5HvN3nDWtv4Ge0DryLW6Tcs6FPCt4srEfCkh5l54LrMmWbhFgkVlN5fTqoY0lnd
YJx2X8WWldRjeL+8E7nFUcFStWrgi9AzgMFrjsL4pql97YAZRXcMoQXVjbRmzVLZ
IVumQy7c+tl7Eqz8lx/xS/5Fx9tIRunqNS5jEUs8Nn5E6FvraAcy+eI0gXTGk759
KNPYisSqAuFAmmt/XDTTvvOo6dpAseXqtR2/LjZJWOlXdiZ/yjHg5+RKQ5dt3dk5
7lAIWER9egIOo/+GAkyek0ZJ5GWU6VxTsFcIl6oy3S7EtB0NCIM7hvhy32QrJ5ZU
yNncTSf6qMVoedgdAgMBAAGjgY0wgYowDAYDVR0TAQH/BAIwADAUBgNVHREEDTAL
gglsb2NhbGhvc3QwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0PAQH/BAUDAweg
ADAdBgNVHQ4EFgQUqCVH9o9E1jUb72ys0de5boT536MwHwYDVR0jBBgwFoAUSCM0
UwqJMThKWurKttKm3s4dKxgwDQYJKoZIhvcNAQELBQADggExAK7dBCSwM/OJw+6s
9MJAb7Ygi9xhHSq30Hg3M7DaPC7J9rZB6+IAVb3poOZAtDDtyTqvXH7qY5UMjJC9
GsFmHPI/OSk2xuJJpG+ZJaP54b7kzTtUD6UCHETsgBk2aNuqNhjXR2fYnR9QME0C
zZWIDV+5DFEBI97ln30N6PcXvIxp7Rsac3qwzvwt3zL+23kTwgM+DoRPoPO0PHr/
eQ9hvRU5wA2Vc47zhUXIFy1Jmx7Sf//pw0/wq46VUAjDZ5B09EoCpzBNvOD7P+cF
FQQ7SId8h8OQ2uOWxT2baeJX0pVbVv+qwOOB1F0q3sjx0dZa/2rxOUZ3wnHG9j8j
LZSUkZxGpPQffCSpSPma5RhYff8/BncdA8soT0dyEfXIX5V91IXnrlI8XZrADvJM
zzJKdNg=
-----END CERTIFICATE-----
subject=CN = localhost

issuer=CN = CA

---
No client certificate CA names sent
Requested Signature Algorithms: RSA+SHA256:RSA-PSS+SHA256:RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA+SHA384:RSA-PSS+SHA384:RSA-PSS+SHA384:ECDSA+SHA384:Ed448:RSA+SHA512:RSA-PSS+SHA512:RSA-PSS+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA512
Peer signing digest: SHA512
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1594 bytes and written 309 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2432 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
read from 0x22388a0 [0x23493f3] (5 bytes => 5 (0x5))
0000 - 17 03 03 00 13                                    .....
read from 0x22388a0 [0x23493f8] (19 bytes => 19 (0x13))
0000 - 34 64 fc b3 92 73 5a 0d-ef c7 2f 1d b6 b7 36 35   4d...sZ.../...65
0010 - 72 27 40                                          r'@
C011B2B4147F0000:error:0A00045C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:ssl/record/rec_layer_s3.c:1584:SSL alert number 116
read from 0x22388a0 [0x21774b0] (8192 bytes => 0)

The 1024-bit key works with RSA-PSS+SHA384, and the 2048-bit key works with RSA-PSS+SHA512.

slontis · 2021-07-28T06:06:30Z

The RSA PSS code does this:
emLen is set to RSA_size(rsa).
Fail if emLen < hashLen + saltLen + 2
So if hashLen = emLen/2 then you are in trouble if saltlen = hashlen

So unless this particular case was special cased to pass in either sLen = RSA_PSS_SALTLEN_MAX or (sLen = RSA_size(rsa) - hashLen - 2) it will not work.

dwmw2 · 2021-07-28T11:31:16Z

Hm, RFC8446 §4.2.3 says:

      The length of the Salt MUST be equal to the length of the output
      of the digest algorithm.

So I think truncating to 62 bytes is wrong. Instead of filing this ticket against OpenSSL, I should be filing one against GnuTLS for the fact that this does work there. (https://gitlab.com/gnutls/gnutls/-/issues/1258).

And fixing my own code not to truncate either. (https://gitlab.com/openconnect/openconnect/-/commit/6c2266deb189a55a00be8a8f8448d879c3faff6a).

There is perhaps an argument that the client shouldn't have tried TLSv1.3 in this case. This one works:

openssl s_client -cert ~/rsa1k.pem -key ~/rsa1k.pem  -sigalgs RSA-PSS+SHA512:RSA+SHA256 -tls1_2  localhost:8443

And this one doesn't:

openssl s_client -cert ~/rsa1k.pem -key ~/rsa1k.pem  -sigalgs RSA-PSS+SHA512:RSA+SHA256   localhost:8443

Admittedly that's a bit of a contrived test, but -sigalgs RSA+SHA256 isn't that unrealistic a scenario at all, with some hardware keys that simply don't support PSS (or raw decryption, I suppose, since that would suffice). Those really should limit us to trying TLSv1.2.

RFC8446 forbids this, and it looks like it was a bug that it ever worked against GnuTLS. • https://gitlab.com/gnutls/gnutls/-/issues/1258 • openssl/openssl#16167 Signed-off-by: David Woodhouse <dwmw2@infradead.org>

slontis · 2021-07-30T04:24:38Z

This is a duplicate of a closed issue that was discussed by the OTC previously. See #12713
For this reason I will close this issue...

dwmw2 added the issue: bug report The issue was opened to report a bug label Jul 27, 2021

slontis closed this as completed Jul 30, 2021

jborean93 mentioned this issue Dec 11, 2021

Support macOS M1 jborean93/omi#47

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Certificate verification fails with RSAE-PSS, SHA512 and a 1024-bit RSA key. #16167

Certificate verification fails with RSAE-PSS, SHA512 and a 1024-bit RSA key. #16167

dwmw2 commented Jul 27, 2021

dwmw2 commented Jul 27, 2021

slontis commented Jul 28, 2021

dwmw2 commented Jul 28, 2021

slontis commented Jul 30, 2021

Certificate verification fails with RSAE-PSS, SHA512 and a 1024-bit RSA key. #16167

Certificate verification fails with RSAE-PSS, SHA512 and a 1024-bit RSA key. #16167

Comments

dwmw2 commented Jul 27, 2021

dwmw2 commented Jul 27, 2021

slontis commented Jul 28, 2021

dwmw2 commented Jul 28, 2021

slontis commented Jul 30, 2021