Fix TypeBuilder canonicalization (#3578)

When types or heap types were used multiple times in a TypeBuilder instance, it
was possible for the canonicalization algorithm to canonicalize a parent type
before canonicalizing all of its component child types, leaking the temporary
types into globally interned types. This bug led to incorrect canonicalization
results and use-after free bugs.

The cause of the bug was that types were canonicalized in the reverse of the
order that they were visited in, but children were visited after the first
occurrence of their parents, not necessarily after the last occurrence of their
parents. One fix could have been to remove the logic that prevented types from
being visited multiple times so that children would always be visited after
their parents. That simple fix, however, would not scale gracefully to handle
recursive types because it would require some way to detect recursions without
accidentally reintroducing these bugs.

This PR implements a more robust solution: topologically sorting the traversed
types to ensure that children are canonicalized before their parents. This
solution will be trivial to adapt for recursive types because recursive types
are trivial to detect from the reachability graph used to perform the
topological sort.

0 files changed, 0 insertions, 0 deletions