Fix a bug optimizing out br_on_cast (#5403)

We were considering casts between unrelated types as unconditionally failing,
but in the case where the unrelated types are nullable, the cast could still
succeed if the value is null.

This bug was introduced in #5397.

1 files changed, 7 insertions, 3 deletions

diff --git a/src/ir/gc-type-utils.h b/src/ir/gc-type-utils.h
index 6ac604ad2..8832d4d0f 100644
--- a/src/ir/gc-type-utils.h
+++ b/src/ir/gc-type-utils.h
@@ -61,9 +61,13 @@ inline EvaluationResult evaluateKindCheck(Expression* curr) {
         if (Type::isSubType(br->ref->type, br->castType)) {
           return flip ? Failure : Success;
         }
-        // If the cast type is unrelated to the type we have, the cast will
-        // certainly fail.
-        if (!Type::isSubType(br->castType, br->ref->type)) {
+        // If the cast type is unrelated to the type we have and it's not
+        // possible for the cast to succeed anyway because the value is null,
+        // then the cast will certainly fail. TODO: This is essentially the same
+        // as `canBeCastTo` in OptimizeInstructions. Find a way to deduplicate
+        // this logic.
+        if (!Type::isSubType(br->castType, br->ref->type) &&
+            (br->castType.isNonNullable() || br->ref->type.isNonNullable())) {
           return flip ? Success : Failure;
         }
         return Unknown;