Disallow --nominal with GC (#4758)

Nominal types don't make much sense without GC, and in particular trying to emit
them with typed function references but not GC enabled can result in invalid
binaries because nominal types do not respect the type ordering constraints
required by the typed function references proposal. Making this change was
mostly straightforward, but required fixing the fuzzer to use --nominal only
when GC is enabled and required exiting early from nominal-only optimizations
when GC was not enabled.

Fixes #4756.

1 files changed, 6 insertions, 0 deletions

diff --git a/src/tools/tool-options.h b/src/tools/tool-options.h
index bc948fc97..c15291c8d 100644
--- a/src/tools/tool-options.h
+++ b/src/tools/tool-options.h
@@ -176,6 +176,12 @@ struct ToolOptions : public Options {
   void applyFeatures(Module& module) const {
     module.features.enable(enabledFeatures);
     module.features.disable(disabledFeatures);
+    // Non-default type systems only make sense with GC enabled. TODO: Error on
+    // non-GC equirecursive types as well once we make isorecursive the default
+    // if we don't remove equirecursive types entirely.
+    if (!module.features.hasGC() && getTypeSystem() == TypeSystem::Nominal) {
+      Fatal() << "Nominal typing is only allowed when GC is enabled";
+    }
   }
 
 private: