Restrict the size of OBJECT IDENTIFIERs that OBJ_obj2txt will translate

OBJ_obj2txt() would translate any size OBJECT IDENTIFIER to canonical
numeric text form.  For gigantic sub-identifiers, this would take a very
long time, the time complexity being O(n^2) where n is the size of that
sub-identifier.

To mitigate this, a restriction on the size that OBJ_obj2txt() will
translate to canonical numeric text form is added, based on RFC 2578
(STD 58), which says this:

> 3.5. OBJECT IDENTIFIER values
>
> An OBJECT IDENTIFIER value is an ordered list of non-negative numbers.
> For the SMIv2, each number in the list is referred to as a sub-identifier,
> there are at most 128 sub-identifiers in a value, and each sub-identifier
> has a maximum value of 2^32-1 (4294967295 decimal).

Fixes otc/security#96
Fixes CVE-2023-2650

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>

CHANGES

@@ -9,6 +9,32 @@
 
  Changes between 1.1.1t and 1.1.1u [xx XXX xxxx]
 
+  *) Mitigate for the time it takes for `OBJ_obj2txt` to translate gigantic
+     OBJECT IDENTIFIER sub-identifiers to canonical numeric text form.
+
+     OBJ_obj2txt() would translate any size OBJECT IDENTIFIER to canonical
+     numeric text form.  For gigantic sub-identifiers, this would take a very
+     long time, the time complexity being O(n^2) where n is the size of that
+     sub-identifier.  (CVE-2023-2650)
+
+     To mitigitate this, `OBJ_obj2txt()` will only translate an OBJECT
+     IDENTIFIER to canonical numeric text form if the size of that OBJECT
+     IDENTIFIER is 586 bytes or less, and fail otherwise.
+
+     The basis for this restriction is RFC 2578 (STD 58), section 3.5. OBJECT
+     IDENTIFIER values, which stipulates that OBJECT IDENTIFIERS may have at
+     most 128 sub-identifiers, and that the maximum value that each sub-
+     identifier may have is 2^32-1 (4294967295 decimal).
+
+     For each byte of every sub-identifier, only the 7 lower bits are part of
+     the value, so the maximum amount of bytes that an OBJECT IDENTIFIER with
+     these restrictions may occupy is 32 * 128 / 7, which is approximately 586
+     bytes.
+
+     Ref: https://datatracker.ietf.org/doc/html/rfc2578#section-3.5
+
+     [Richard Levitte]
+
   *) Reworked the Fix for the Timing Oracle in RSA Decryption (CVE-2022-4304).
      The previous fix for this timing side channel turned out to cause
      a severe 2-3x performance regression in the typical use case

NEWS

@@ -7,6 +7,8 @@
 
   Major changes between OpenSSL 1.1.1t and OpenSSL 1.1.1u [under development]
 
+      o Mitigate for very slow `OBJ_obj2txt()` performance with gigantic
+        OBJECT IDENTIFIER sub-identities.  (CVE-2023-2650)
       o Fixed documentation of X509_VERIFY_PARAM_add0_policy() (CVE-2023-0466)
       o Fixed handling of invalid certificate policies in leaf certificates
         (CVE-2023-0465)

crypto/objects/obj_dat.c

@@ -428,6 +428,25 @@ int OBJ_obj2txt(char *buf, int buf_len, const ASN1_OBJECT *a, int no_name)
     first = 1;
     bl = NULL;
 
+    /*
+     * RFC 2578 (STD 58) says this about OBJECT IDENTIFIERs:
+     *
+     * > 3.5. OBJECT IDENTIFIER values
+     * >
+     * > An OBJECT IDENTIFIER value is an ordered list of non-negative
+     * > numbers. For the SMIv2, each number in the list is referred to as a
+     * > sub-identifier, there are at most 128 sub-identifiers in a value,
+     * > and each sub-identifier has a maximum value of 2^32-1 (4294967295
+     * > decimal).
+     *
+     * So a legitimate OID according to this RFC is at most (32 * 128 / 7),
+     * i.e. 586 bytes long.
+     *
+     * Ref: https://datatracker.ietf.org/doc/html/rfc2578#section-3.5
+     */
+    if (len > 586)
+        goto err;
+
     while (len > 0) {
         l = 0;
         use_bn = 0;