[Wasm GC] Do not merge supertypes into subtypes (#5439)

In TypeMerging we previously merged all subsequent types in a refined partition
into whichever type happened to be first in the partition, but when that first
type happened to be a subtype of one of the other types in the partition, that
would cause type-updating.cpp to try to update that subtype's supertype to be
itself, causing an assertion failure.

Fix the problem by ensuring that the merge target is not a subtype of any other
types in the partition.

1 files changed, 8 insertions, 3 deletions

diff --git a/src/passes/TypeMerging.cpp b/src/passes/TypeMerging.cpp
index 6f39d586f..5623ecdb9 100644
--- a/src/passes/TypeMerging.cpp
+++ b/src/passes/TypeMerging.cpp
@@ -280,10 +280,15 @@ void TypeMerging::run(Module* module_) {
   // The types we can merge mapped to the type we are merging them into.
   TypeUpdates merges;
 
-  // Merge each refined partition into a single type.
+  // Merge each refined partition into a single type. We should only merge into
+  // supertypes or siblings because if we try to merge into a subtype then we
+  // will accidentally set that subtype to be its own supertype.
   for (const auto& partition : refinedPartitions) {
-    for (size_t i = 1; i < partition.size(); ++i) {
-      merges[partition[i]] = partition[0];
+    auto target = *HeapTypeOrdering::SupertypesFirst(partition).begin();
+    for (auto type : partition) {
+      if (type != target) {
+        merges[type] = target;
+      }
     }
   }