crypto/rand/randfile.c: avoid signed integer overflow in RAND_load_file

If a file supplied to RAND_load_file is too big (more than INT_MAX bytes),
it is possible to trigger a signer integer overflow during ret calculation.
Avoid it by returning early when we are about to hit it on the next
iteration.

Reported-by: Liu-Ermeng <liuermeng2@huawei.com>
Resolves: https://github.com/openssl/openssl/issues/28375
Signed-off-by: Eugene Syromiatnikov <esyr@openssl.org>

Reviewed-by: Paul Dale <ppzgs1@gmail.com>
Reviewed-by: Saša Nedvědický <sashan@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/28379)

(cherry picked from commit 35db6a15d4)

crypto/rand/randfile.c

@@ -167,6 +167,10 @@ int RAND_load_file(const char *file, long bytes)
         /* If given a bytecount, and we did it, break. */
         if (bytes > 0 && (bytes -= i) <= 0)
             break;
+
+        /* We can hit a signed integer overflow on the next iteration */
+        if (ret > INT_MAX - RAND_LOAD_BUF_SIZE)
+            break;
     }
 
     OPENSSL_cleanse(buf, sizeof(buf));

doc/man3/RAND_load_file.pod

@@ -20,6 +20,8 @@ RAND_load_file() reads a number of bytes from file B<filename> and
 adds them to the PRNG. If B<max_bytes> is nonnegative,
 up to B<max_bytes> are read;
 if B<max_bytes> is -1, the complete file is read.
+RAND_load_file() can read less than the complete file or the requested number
+of bytes if it doesn't fit in the return value type.
 Do not load the same file multiple times unless its contents have
 been updated by RAND_write_file() between reads.
 Also, note that B<filename> should be adequately protected so that an