Since OPENSSL_malloc() and friends report ERR_R_MALLOC_FAILURE, and
at least handle the file name and line number they are called from,
there's no need to report ERR_R_MALLOC_FAILURE where they are called
directly, or when SSLfatal() and RLAYERfatal() is used, the reason
`ERR_R_MALLOC_FAILURE` is changed to `ERR_R_CRYPTO_LIB`.
There were a number of places where `ERR_R_MALLOC_FAILURE` was reported
even though it was a function from a different sub-system that was
called. Those places are changed to report ERR_R_{lib}_LIB, where
{lib} is the name of that sub-system.
Some of them are tricky to get right, as we have a lot of functions
that belong in the ASN1 sub-system, and all the `sk_` calls or from
the CRYPTO sub-system.
Some extra adaptation was necessary where there were custom OPENSSL_malloc()
wrappers, and some bugs are fixed alongside these changes.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19301)
When assigning pointers to functions in an OSSL_DISPATCH table, we try
to ensure that those functions are properly defined or declared with
an extra declaration using the corresponding function typedefs that
are defined by include/openssl/core_dispatch.h.
For the core dispatch table, found in crypto/provider_core.c, it seems
we forgot this habit, and thus didn't ensure well enough that the
function pointers that are assigned in the table can actually be used
for those dispatch table indexes.
This change adds all the missing declarations, and compensates for
differences with functions that do the necessary casting, making those
explicit rather than implicit, thereby trying to assure that we know
what we're doing.
One function is not fixed in this change, because there's a controversy,
a clash between the signature of BIO_ctrl() and OSSL_FUNC_BIO_ctrl_fn.
They have different return types.
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18198)
Richard Levitte
Matt Caswell
Shane Lontis
Pauli