[spirv-remap] Fix undefined behavior in hashing (#2403)

There's a statement that intends to generate a 32-bit hashcode, but due
to integer promotion, the intermediate values can trigger signed integer
overflow, which is undefined behavior.

To avoid this, cast at least one operand to unsigned int before
multiplying, which will cause the result to be promoted to unsigned int
instead of signed int.

With this patch, I'm able to build core for qemu-x64 with host_asan-ubsan.

Fixed: 60128
Change-Id: Idd644e534116bf29dca8013936ac39901bbe68fc
Reviewed-on: https://fuchsia-review.googlesource.com/c/third_party/glslang/+/428254
Reviewed-by: John Bauman <jbauman@google.com>

Co-authored-by: Drew Fisher <zarvox@google.com>
This commit is contained in:
craig stout 2020-09-26 20:43:30 -04:00 committed by GitHub
parent 1815d86a22
commit d0e7ed37fc
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -830,7 +830,15 @@ namespace spv {
[&](spv::Id& id) {
if (thisOpCode != spv::OpNop) {
++idCounter;
const std::uint32_t hashval = opCounter[thisOpCode] * thisOpCode * 50047 + idCounter + fnId * 117;
const std::uint32_t hashval =
// Explicitly cast operands to unsigned int to avoid integer
// promotion to signed int followed by integer overflow,
// which would result in undefined behavior.
static_cast<unsigned int>(opCounter[thisOpCode])
* thisOpCode
* 50047
+ idCounter
+ static_cast<unsigned int>(fnId) * 117;
if (isOldIdUnmapped(id))
localId(id, nextUnusedId(hashval % softTypeIdLimit + firstMappedID));