STARTTLS and stream encryption are discussed in detail in Section 10.10. Among the items you might need to set up to employ them is a key file that corresponds to a certificate file. That is the key used by sendmail when it is acting in the roll of a server (receiving inbound email). A server key is contained in a file, the location of which is set with this ServerKeyFile option:

O ServerKeyFile=path           configuration file (V8.11 and later) 
-OServerKeyFile=path           command line (V8.11 and later) 
define(`confSERVER_KEY',`path')    mc configuration (V8.11 and later 

Here, path is a full path specification of the file containing the key. The path might contain sendmail macros, which will be expanded (their values used) when the configuration file, or command line, is read:

define(`confSERVER_KEY', `${MyCERTPath}/SrvrKey.pem')

The path must be a full pathname (must begin with a slash), and must live in a directory that is safe (every component of which is writable only by root or the trusted user specified in the TrustedUser option), and must itself be safe (mode 0600, owned by, readable, and writable only by root or the trusted user specified in the TrustedUser option). If it is not, it will be rejected and the following error logged:

STARTTLS=server: file path unsafe: reason 

But, even if all goes well, there is still a chance that the SSL software will reject the certificate. If it does, the following will be logged:

STARTTLS=server, error: SSL_CTX_use_PrivateKey_file(path=) failed

The ServerKeyFile option is not safe. If specified from the command line, it can cause sendmail to relinquish its special privileges.