Make `GlobalTypeRewriter` work for isorecursive types (#4829)

There are two new potential problems that `GlobalTypeRewriter` can run into when
working with isorecursive types instead of nominal types. First, the refined
types may have replaced generic references with references to specific other
types, potentially creating new recursions and making the existing recursion
groups insufficient. Second, distinct types may be refined to structurally
identical types and those distinct input types may map the same output type,
potentially changing cast behavior.

Both of these problems are solved by putting all the new types in a single large
recursion group.

We do not currently account for the fact that types may be used in the external
interface of the module, but when we do, externalized types will be excluded
from optimizations and will not be affected by the creation of this single large
rec group.

Fixes #4816.

1 files changed, 3 insertions, 2 deletions

diff --git a/src/passes/SignatureRefining.cpp b/src/passes/SignatureRefining.cpp
index 8a8c45872..11df83f4d 100644
--- a/src/passes/SignatureRefining.cpp
+++ b/src/passes/SignatureRefining.cpp
@@ -51,8 +51,9 @@ struct SignatureRefining : public Pass {
     if (!module->features.hasGC()) {
       return;
     }
-    if (getTypeSystem() != TypeSystem::Nominal) {
-      Fatal() << "SignatureRefining requires nominal typing";
+    if (getTypeSystem() != TypeSystem::Nominal &&
+        getTypeSystem() != TypeSystem::Isorecursive) {
+      Fatal() << "SignatureRefining requires nominal/hybrid typing";
     }
 
     if (!module->tables.empty()) {