If an invalid key size is passed in for RSA (I haven't tested others) then the keygen will fail and will be retried. Over and over, never really failing.

The journal logs:
May 06 18:11:28 ipa.example.test certmonger[1086797]: 2026-05-06 18:11:28 [1086797] Overriding requested key size of 2046 with -1.
May 06 18:11:28 ipa.example.test certmonger[1086797]: 2026-05-06 18:11:28 [1086797] Overriding requested key size of 2046 with -1.
May 06 18:11:28 ipa.example.test certmonger[1086797]: 2026-05-06 18:11:28 [1086797] Overriding requested key size of 2046 with -1.
....

It doesn't fail gracefully here in keygen-o.c:

    /* Retry with the optimum key size. */
    if (privkey == NULL) {
        cm_key_size = PK11_GetBestKeyLength(slot, pmech);
        if (cm_key_size != cm_requested_key_size) {
            cm_log(1,
                   "Overriding requested key size of %d with %d.\n",
                   cm_requested_key_size, cm_key_size);
            goto retry_gen;
        }

I can see two obvious paths forward:

• Add a check for -1 and exit with CM_SUB_STATUS_INTERNAL_ERROR
• Act like keygen-o where this is not checked at all and will fail with CM_SUB_STATUS_INTERNAL_ERROR if the keygen is not successful.