[GC] Do not crash on unreasonable GC array allocations in interpreter; trap (#3559)

The spec does not mention traps here, but this is like a JS VM trapping on
OOM - a runtime limitation is reached.

As these are not specced traps, I did not add them to effects.h. Note how
as a result the optimizer happily optimizes into a nop an unused allocation of an
array of size unsigned(-1), which is the behavior we want.

2 files changed, 22 insertions, 2 deletions

diff --git a/src/ir/effects.h b/src/ir/effects.h
index de228c3f4..3d71a8433 100644
--- a/src/ir/effects.h
+++ b/src/ir/effects.h
@@ -22,8 +22,7 @@
 
 namespace wasm {
 
-// Look for side effects, including control flow
-// TODO: optimize
+// Analyze various possible effects.
 
 class EffectAnalyzer {
 public:
@@ -77,9 +76,23 @@ public:
   bool writesHeap = false;
   // A trap, either from an unreachable instruction, or from an implicit trap
   // that we do not ignore (see below).
+  //
   // Note that we ignore trap differences, so it is ok to reorder traps with
   // each other, but it is not ok to remove them or reorder them with other
   // effects in a noticeable way.
+  //
+  // Note also that we ignore runtime-dependent traps, such as hitting a
+  // recursion limit or running out of memory. Such traps are not part of wasm's
+  // official semantics, and they can occur anywhere: *any* instruction could in
+  // theory be implemented by a VM call (as will be the case when running in an
+  // interpreter), and such a call could run out of stack or memory in
+  // principle. To put it another way, an i32 division by zero is the program
+  // doing something bad that causes a trap, but the VM running out of memory is
+  // the VM doing something bad - and therefore the VM behaving in a way that is
+  // not according to the wasm semantics - and we do not model such things. Note
+  // that as a result we do *not* mark things like GC allocation instructions as
+  // having side effects, which has the nice benefit of making it possible to
+  // eliminate an allocation whose result is not captured.
   bool trap = false;
   // A trap from an instruction like a load or div/rem, which may trap on corner
   // cases. If we do not ignore implicit traps then these are counted as a trap.
diff --git a/src/wasm-interpreter.h b/src/wasm-interpreter.h
index 9118eaae1..1c8377127 100644
--- a/src/wasm-interpreter.h
+++ b/src/wasm-interpreter.h
@@ -1617,6 +1617,13 @@ public:
     }
     const auto& element = curr->rtt->type.getHeapType().getArray().element;
     Index num = size.getSingleValue().geti32();
+    // Arbitrary deterministic limit on size. If we need to allocate a Literals
+    // vector that takes around 1-2GB of memory then we are likely to hit memory
+    // limits on 32-bit machines, and in particular on wasm32 VMs that do not
+    // have 4GB support, so give up there.
+    if (num >= (1 << 30) / sizeof(Literal)) {
+      trap("allocation failure");
+    }
     Literals data(num);
     if (curr->isWithDefault()) {
       for (Index i = 0; i < num; i++) {