Does OpenSSL "verify" command really validate the digital signature in a self-signed certificate?

No, OpenSSL "verify" command does not validate the digital signature in a self-signed certificate.

In the following test, a CSR with an RSA public key was "self-signed" by the OpenSSL "req -x509" command with a DSA private key:

C:\Users\fyicenter>\local\openssl\openssl.exe

OpenSSL> x509 -in rsa_dsa_test.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            f6:7d:11:23:20:b6:f1:77
    Signature Algorithm: dsa_with_SHA256
        Issuer: C=us, ST=NY, L=New York, O=Donald Inc., OU=IT, 
                CN=www.donald.inc/emailAddress=john@donald.inc
        Validity
            Not Before: Aug 21 13:02:46 2016 GMT
            Not After : Sep 20 13:02:46 2016 GMT
        Subject: C=us, ST=NY, L=New York, O=Donald Inc., OU=IT, 
                 CN=www.donald.inc/emailAddress=john@donald.inc
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (512 bit)
                Modulus:
                    00:f6:d5:d3:79:87:8d:9d:83:49:6f:fb:08:67:08:
                    fb:0f:ab:b4:7f:51:55:7b:49:fa:e3:47:8e:6e:22:
                    d7:ba:ad:dc:10:56:e9:b3:42:f7:25:20:9d:a5:e3:
                    5f:5e:7c:95:cb:5a:22:f3:8f:3d:e1:b2:0a:fa:15:
                    c5:16:64:17:03
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                99:FB:5B:B6:BE:B4:E2:2B:4D:46:75:3F:0E:5E:52:36:F1:0E:A4:DB
            X509v3 Authority Key Identifier:
                keyid:99:FB:5B:B6:BE:B4:E2:2B:4D:46:75:3F:0E:5E:52:36:F1:0E:A4:DB

            X509v3 Basic Constraints:
                CA:TRUE
    Signature Algorithm: dsa_with_SHA256
         r:
             1c:c1:8c:6a:0f:79:b6:68:33:ec:59:30:aa:e9:f1:
             19:03:5a:c0:1f
         s:
             00:91:d5:16:e3:8c:8f:71:57:0c:c8:e1:69:04:4a:
             d1:a5:62:b8:29:91

OpenSSL> verify -CAfile rsa_dsa_test.crt rsa_dsa_test.crt
rsa_dsa_test.crt: OK

OpenSSL> verify -trusted rsa_dsa_test.crt rsa_dsa_test.crt
rsa_dsa_test.crt: OK

As you can see from the output, both "verify -CAfile" and "verify -trusted" commands passed the validation of the given "self-signed" certificate. Apparently, OpenSSL did not verify the signature with the public key in the certificate. If it did, validation would fail. The public key will never verify the signature, since the public key is using RSA algorithm and the signature is using DSA algorithm.

It looks like OpenSSL only checks to see if the "subject" matches "issuer" or not in the case of self-signed certificate validation.

⇒ OpenSSL "req -config" - Using Configuration File

⇐ OpenSSL "req -x509" - Sign CSR with Different Key

⇑ OpenSSL "req" Command

⇑⇑ OpenSSL Tutorials