mewin/glslang
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

SPIR-V compression: Requires rerunning CMake. Adds a standalone tool for running the SPV compression.

Browse Source 
git-svn-id: https://cvs.khronos.org/svn/repos/ogl/trunk/ecosystem/public/sdk/tools/glslang@31232 e7fa87d3-cd2b-0410-9028-fcbf551c1848
This commit is contained in:
John Kessenich 2015-05-19 21:07:04 +00:00
parent 40e391184c
commit 3c4a276282
4 changed files with 1635 additions and 1155 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

137
README-spirv-remap.txt Normal file
View File

@ -0,0 +1,137 @@
VERSION
--------------------------------------------------------------------------------
spirv-remap 0.97
INTRO:
--------------------------------------------------------------------------------
spirv-remap is a utility to improve compression of SPIRV binary files via
entropy reduction, plus optional stripping of debug information and
load/store optimization. It transforms SPIRV to SPIRV, remapping IDs. The
resulting modules have an increased ID range (IDs are not as tightly packed
around zero), but will compress better when multiple modules are compressed
together, since compressor's dictionary can find better cross module
commonality.
Remapping is accomplished via canonicalization. Thus, modules can be
compressed one at a time with no loss of quality relative to operating on
many modules at once. The command line tool operates on multiple modules
only in the trivial repetition sense, for ease of use. The remapper API
only accepts a single module at a time.
There are two modes of use: command line, and a C++11 API. Both are
described below.
spirv-remap is currently in an alpha state. Although there are no known
remapping defects, it has only been exercised on one real world game shader
workload.
FEEDBACK
--------------------------------------------------------------------------------
Report defects, enhancements requests, code improvements, etc to:
spvremapper@lunarg.com
COMMAND LINE USAGE:
--------------------------------------------------------------------------------
Examples are given with a verbosity of one (-v), but more verbosity can be
had via -vv, -vvv, etc, or an integer parameter to --verbose, such as
"--verbose 4". With no verbosity, the command is silent and returns 0 on
success, and a positive integer error on failure.
Pre-built binaries for several OSs are available. Examples presented are
for Linux. Command line arguments can be provided in any order.
1. Basic ID remapping
Perform ID remapping on all shaders in "*.spv", writing new files with
the same basenames to /tmp/out_dir.
spirv-remap --map all --input *.spv --output /tmp/out_dir
2. Perform all possible size reductions
spirv-remap-linux-64 --do-everything --input *.spv --output /tmp/out_dir
Note that --do-everything is a synonym for:
--map all --dce all --opt all --strip all
API USAGE:
--------------------------------------------------------------------------------
The public interface to the remapper is defined in SPIRV/SPVRemapper.h as follows:
namespace spv {
class spirvbin_t
{
public:
enum Options { ... };
spirvbin_t(int verbose = 0); // construct
// remap an existing binary in memory
void remap(std::vector<std::uint32_t>& spv, std::uint32_t opts = Options::DO_EVERYTHING);
// Type for error/log handler functions
typedef std::function<void(const std::string&)> errorfn_t;
typedef std::function<void(const std::string&)> logfn_t;
// Register error/log handling functions (can be c/c++ fn, lambda fn, or functor)
static void registerErrorHandler(errorfn_t handler) { errorHandler = handler; }
static void registerLogHandler(logfn_t handler) { logHandler = handler; }
};
} // namespace spv
The class definition is in SPVRemapper.cpp.
remap() accepts an std::vector of SPIRV words, modifies them per the
request given in 'opts', and leaves the 'spv' container with the result.
It is safe to instantiate one spirvbin_t per thread and process a different
SPIRV in each.
The "opts" parameter to remap() accepts a bit mask of desired remapping
options. See REMAPPING AND OPTIMIZATION OPTIONS.
On error, the function supplied to registerErrorHandler() will be invoked.
This can be a standard C/C++ function, a lambda function, or a functor.
The default handler simply calls exit(5); The error handler is a static
members, so need only be set up once, not once per spirvbin_t instance.
Log messages are supplied to registerLogHandler(). By default, log
messages are eaten silently. The log handler is also a static member.
BUILD DEPENDENCIES:
--------------------------------------------------------------------------------
1. C++11 compatible compiler
2. cmake
3. glslang
BUILDING
--------------------------------------------------------------------------------
The standalone remapper is built along side glslangValidator through its
normal build process.
REMAPPING AND OPTIMIZATION OPTIONS
--------------------------------------------------------------------------------
API:
These are bits defined under spv::spirvbin_t::Options::, and can be
bitwise or-ed together as desired.
MAP_TYPES = canonicalize type IDs
MAP_NAMES = canonicalize named data
MAP_FUNCS = canonicalize function bodies
DCE_FUNCS = remove dead functions
DCE_VARS = remove dead variables
DCE_TYPES = remove dead types
OPT_LOADSTORE = optimize unneeded load/stores
MAP_ALL = (MAP_TYPES | MAP_NAMES | MAP_FUNCS)
DCE_ALL = (DCE_FUNCS | DCE_VARS | DCE_TYPES)
OPT_ALL = (OPT_LOADSTORE)
ALL_BUT_STRIP = (MAP_ALL | DCE_ALL | OPT_ALL)
DO_EVERYTHING = (STRIP | ALL_BUT_STRIP)

297
SPIRV/SPVRemapper.cpp
View File

@ -36,8 +36,6 @@
#include "SPVRemapper.h" #include "SPVRemapper.h"
#include "doc.h" #include "doc.h"
/* -*-mode:c++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 3 -*- */
#if !defined (use_cpp11) #if !defined (use_cpp11)
// ... not supported before C++11 // ... not supported before C++11
#else // defined (use_cpp11) #else // defined (use_cpp11)
@ -47,21 +45,21 @@
namespace spv { namespace spv {
// By default, just abort on error. Can be overridden via RegisterErrorHandler // By default, just abort on error. Can be overridden via RegisterErrorHandler
spirvbin_t::errorfn_t spirvbin_t::errorHandler = [](const std::string&) { exit(5); }; spirvbin_t::errorfn_t spirvbin_t::errorHandler = [](const std::string&) { exit(5); };
// By default, eat log messages. Can be overridden via RegisterLogHandler // By default, eat log messages. Can be overridden via RegisterLogHandler
spirvbin_t::logfn_t spirvbin_t::logHandler = [](const std::string&) { }; spirvbin_t::logfn_t spirvbin_t::logHandler = [](const std::string&) { };
// This can be overridden to provide other message behavior if needed // This can be overridden to provide other message behavior if needed
void spirvbin_t::msg(int minVerbosity, int indent, const std::string& txt) const void spirvbin_t::msg(int minVerbosity, int indent, const std::string& txt) const
{ {
if (verbose >= minVerbosity) if (verbose >= minVerbosity)
logHandler(std::string(indent, ' ') + txt); logHandler(std::string(indent, ' ') + txt);
} }
// hash opcode, with special handling for OpExtInst // hash opcode, with special handling for OpExtInst
std::uint32_t spirvbin_t::asOpCodeHash(int word) std::uint32_t spirvbin_t::asOpCodeHash(int word)
{ {
const spv::Op opCode = asOpCode(word); const spv::Op opCode = asOpCode(word);
std::uint32_t offset = 0; std::uint32_t offset = 0;
@ -74,10 +72,10 @@ std::uint32_t spirvbin_t::asOpCodeHash(int word)
} }
return opCode * 19 + offset; // 19 = small prime return opCode * 19 + offset; // 19 = small prime
} }
spirvbin_t::range_t spirvbin_t::literalRange(spv::Op opCode) const spirvbin_t::range_t spirvbin_t::literalRange(spv::Op opCode) const
{ {
static const int maxCount = 1<<30; static const int maxCount = 1<<30;
switch (opCode) { switch (opCode) {
@ -91,10 +89,10 @@ spirvbin_t::range_t spirvbin_t::literalRange(spv::Op opCode) const
case spv::OpConstant: return range_t(3, maxCount); case spv::OpConstant: return range_t(3, maxCount);
default: return range_t(0, 0); default: return range_t(0, 0);
} }
} }
spirvbin_t::range_t spirvbin_t::typeRange(spv::Op opCode) const spirvbin_t::range_t spirvbin_t::typeRange(spv::Op opCode) const
{ {
static const int maxCount = 1<<30; static const int maxCount = 1<<30;
if (isConstOp(opCode)) if (isConstOp(opCode))
@ -112,10 +110,10 @@ spirvbin_t::range_t spirvbin_t::typeRange(spv::Op opCode) const
case spv::OpTypePointer: return range_t(3, 4); case spv::OpTypePointer: return range_t(3, 4);
default: return range_t(0, 0); default: return range_t(0, 0);
} }
} }
spirvbin_t::range_t spirvbin_t::constRange(spv::Op opCode) const spirvbin_t::range_t spirvbin_t::constRange(spv::Op opCode) const
{ {
static const int maxCount = 1<<30; static const int maxCount = 1<<30;
switch (opCode) { switch (opCode) {
@ -124,11 +122,11 @@ spirvbin_t::range_t spirvbin_t::constRange(spv::Op opCode) const
case spv::OpConstantComposite: return range_t(3, maxCount); case spv::OpConstantComposite: return range_t(3, maxCount);
default: return range_t(0, 0); default: return range_t(0, 0);
} }
} }
// Is this an opcode we should remove when using --strip? // Is this an opcode we should remove when using --strip?
bool spirvbin_t::isStripOp(spv::Op opCode) const bool spirvbin_t::isStripOp(spv::Op opCode) const
{ {
switch (opCode) { switch (opCode) {
case spv::OpSource: case spv::OpSource:
case spv::OpSourceExtension: case spv::OpSourceExtension:
@ -137,28 +135,28 @@ bool spirvbin_t::isStripOp(spv::Op opCode) const
case spv::OpLine: return true; case spv::OpLine: return true;
default: return false; default: return false;
} }
} }
bool spirvbin_t::isFlowCtrlOpen(spv::Op opCode) const bool spirvbin_t::isFlowCtrlOpen(spv::Op opCode) const
{ {
switch (opCode) { switch (opCode) {
case spv::OpBranchConditional: case spv::OpBranchConditional:
case spv::OpSwitch: return true; case spv::OpSwitch: return true;
default: return false; default: return false;
} }
} }
bool spirvbin_t::isFlowCtrlClose(spv::Op opCode) const bool spirvbin_t::isFlowCtrlClose(spv::Op opCode) const
{ {
switch (opCode) { switch (opCode) {
case spv::OpLoopMerge: case spv::OpLoopMerge:
case spv::OpSelectionMerge: return true; case spv::OpSelectionMerge: return true;
default: return false; default: return false;
} }
} }
bool spirvbin_t::isTypeOp(spv::Op opCode) const bool spirvbin_t::isTypeOp(spv::Op opCode) const
{ {
switch (opCode) { switch (opCode) {
case spv::OpTypeVoid: case spv::OpTypeVoid:
case spv::OpTypeBool: case spv::OpTypeBool:
@ -181,10 +179,10 @@ bool spirvbin_t::isTypeOp(spv::Op opCode) const
case spv::OpTypePipe: return true; case spv::OpTypePipe: return true;
default: return false; default: return false;
} }
} }
bool spirvbin_t::isConstOp(spv::Op opCode) const bool spirvbin_t::isConstOp(spv::Op opCode) const
{ {
switch (opCode) { switch (opCode) {
case spv::OpConstantNullObject: error("unimplemented constant type"); case spv::OpConstantNullObject: error("unimplemented constant type");
case spv::OpConstantSampler: error("unimplemented constant type"); case spv::OpConstantSampler: error("unimplemented constant type");
@ -196,28 +194,28 @@ bool spirvbin_t::isConstOp(spv::Op opCode) const
case spv::OpConstant: return true; case spv::OpConstant: return true;
default: return false; default: return false;
} }
} }
const auto inst_fn_nop = [](spv::Op, int) { return false; }; const auto inst_fn_nop = [](spv::Op, int) { return false; };
const auto op_fn_nop = [](spv::Id&) { }; const auto op_fn_nop = [](spv::Id&) { };
// g++ doesn't like these defined in the class proper in an anonymous namespace. // g++ doesn't like these defined in the class proper in an anonymous namespace.
// Dunno why. Also MSVC doesn't like the constexpr keyword. Also dunno why. // Dunno why. Also MSVC doesn't like the constexpr keyword. Also dunno why.
// Defining them externally seems to please both compilers, so, here they are. // Defining them externally seems to please both compilers, so, here they are.
const spv::Id spirvbin_t::unmapped = spv::Id(-10000); const spv::Id spirvbin_t::unmapped = spv::Id(-10000);
const spv::Id spirvbin_t::unused = spv::Id(-10001); const spv::Id spirvbin_t::unused = spv::Id(-10001);
const int spirvbin_t::header_size = 5; const int spirvbin_t::header_size = 5;
spv::Id spirvbin_t::nextUnusedId(spv::Id id) spv::Id spirvbin_t::nextUnusedId(spv::Id id)
{ {
while (isNewIdMapped(id)) // search for an unused ID while (isNewIdMapped(id)) // search for an unused ID
++id; ++id;
return id; return id;
} }
spv::Id spirvbin_t::localId(spv::Id id, spv::Id newId) spv::Id spirvbin_t::localId(spv::Id id, spv::Id newId)
{ {
assert(id != spv::NoResult && newId != spv::NoResult); assert(id != spv::NoResult && newId != spv::NoResult);
if (id >= idMapL.size()) if (id >= idMapL.size())
@ -240,12 +238,12 @@ spv::Id spirvbin_t::localId(spv::Id id, spv::Id newId)
} }
return idMapL[id] = newId; return idMapL[id] = newId;
} }
// Parse a literal string from the SPIR binary and return it as an std::string // Parse a literal string from the SPIR binary and return it as an std::string
// Due to C++11 RValue references, this doesn't copy the result string. // Due to C++11 RValue references, this doesn't copy the result string.
std::string spirvbin_t::literalString(int word) const std::string spirvbin_t::literalString(int word) const
{ {
std::string literal; std::string literal;
literal.reserve(16); literal.reserve(16);
@ -256,11 +254,11 @@ std::string spirvbin_t::literalString(int word) const
literal += *bytes++; literal += *bytes++;
return literal; return literal;
} }
void spirvbin_t::applyMap() void spirvbin_t::applyMap()
{ {
msg(3, 2, std::string("Applying map: ")); msg(3, 2, std::string("Applying map: "));
// Map local IDs through the ID map // Map local IDs through the ID map
@ -270,12 +268,12 @@ void spirvbin_t::applyMap()
assert(id != unused && id != unmapped); assert(id != unused && id != unmapped);
} }
); );
} }
// Find free IDs for anything we haven't mapped // Find free IDs for anything we haven't mapped
void spirvbin_t::mapRemainder() void spirvbin_t::mapRemainder()
{ {
msg(3, 2, std::string("Remapping remainder: ")); msg(3, 2, std::string("Remapping remainder: "));
spv::Id unusedId = 1; // can't use 0: that's NoResult spv::Id unusedId = 1; // can't use 0: that's NoResult
@ -297,10 +295,10 @@ void spirvbin_t::mapRemainder()
} }
bound(maxBound); // reset header ID bound to as big as it now needs to be bound(maxBound); // reset header ID bound to as big as it now needs to be
} }
void spirvbin_t::stripDebug() void spirvbin_t::stripDebug()
{ {
if ((options & Options::STRIP) == 0) if ((options & Options::STRIP) == 0)
return; return;
@ -313,10 +311,10 @@ void spirvbin_t::stripDebug()
return true; return true;
}, },
op_fn_nop); op_fn_nop);
} }
void spirvbin_t::buildLocalMaps() void spirvbin_t::buildLocalMaps()
{ {
msg(2, 2, std::string("build local maps: ")); msg(2, 2, std::string("build local maps: "));
mapped.clear(); mapped.clear();
@ -377,11 +375,11 @@ void spirvbin_t::buildLocalMaps()
[this](spv::Id& id) { localId(id, unmapped); } [this](spv::Id& id) { localId(id, unmapped); }
); );
} }
// Validate the SPIR header // Validate the SPIR header
void spirvbin_t::validate() const void spirvbin_t::validate() const
{ {
msg(2, 2, std::string("validating: ")); msg(2, 2, std::string("validating: "));
if (spv.size() < header_size) if (spv.size() < header_size)
@ -396,11 +394,11 @@ void spirvbin_t::validate() const
if (schemaNum() != 0) if (schemaNum() != 0)
error("bad schema, must be 0"); error("bad schema, must be 0");
} }
int spirvbin_t::processInstruction(int word, instfn_t instFn, idfn_t idFn) int spirvbin_t::processInstruction(int word, instfn_t instFn, idfn_t idFn)
{ {
const auto instructionStart = word; const auto instructionStart = word;
const unsigned wordCount = asWordCount(instructionStart); const unsigned wordCount = asWordCount(instructionStart);
const spv::Op opCode = asOpCode(instructionStart); const spv::Op opCode = asOpCode(instructionStart);
@ -500,11 +498,11 @@ int spirvbin_t::processInstruction(int word, instfn_t instFn, idfn_t idFn)
} }
return nextInst; return nextInst;
} }
// Make a pass over all the instructions and process them given appropriate functions // Make a pass over all the instructions and process them given appropriate functions
spirvbin_t& spirvbin_t::process(instfn_t instFn, idfn_t idFn, int begin, int end) spirvbin_t& spirvbin_t::process(instfn_t instFn, idfn_t idFn, int begin, int end)
{ {
// For efficiency, reserve name map space. It can grow if needed. // For efficiency, reserve name map space. It can grow if needed.
nameMap.reserve(32); nameMap.reserve(32);
@ -519,11 +517,11 @@ spirvbin_t& spirvbin_t::process(instfn_t instFn, idfn_t idFn, int begin, int end
nextInst = processInstruction(word, instFn, idFn); nextInst = processInstruction(word, instFn, idFn);
return *this; return *this;
} }
// Apply global name mapping to a single module // Apply global name mapping to a single module
void spirvbin_t::mapNames() void spirvbin_t::mapNames()
{ {
static const std::uint32_t softTypeIdLimit = 3011; // small prime. TODO: get from options static const std::uint32_t softTypeIdLimit = 3011; // small prime. TODO: get from options
static const std::uint32_t firstMappedID = 3019; // offset into ID space static const std::uint32_t firstMappedID = 3019; // offset into ID space
@ -535,11 +533,11 @@ void spirvbin_t::mapNames()
if (isOldIdUnmapped(name.second)) if (isOldIdUnmapped(name.second))
localId(name.second, nextUnusedId(hashval % softTypeIdLimit + firstMappedID)); localId(name.second, nextUnusedId(hashval % softTypeIdLimit + firstMappedID));
} }
} }
// Map fn contents to IDs of similar functions in other modules // Map fn contents to IDs of similar functions in other modules
void spirvbin_t::mapFnBodies() void spirvbin_t::mapFnBodies()
{ {
static const std::uint32_t softTypeIdLimit = 19071; // small prime. TODO: get from options static const std::uint32_t softTypeIdLimit = 19071; // small prime. TODO: get from options
static const std::uint32_t firstMappedID = 6203; // offset into ID space static const std::uint32_t firstMappedID = 6203; // offset into ID space
@ -658,12 +656,12 @@ void spirvbin_t::mapFnBodies()
localId(id, nextUnusedId(hashval % softTypeIdLimit + firstMappedID)); localId(id, nextUnusedId(hashval % softTypeIdLimit + firstMappedID));
} }
}); });
} }
#ifdef NOTDEF #ifdef NOTDEF
// remove bodies of uncalled functions // remove bodies of uncalled functions
void spirvbin_t::offsetIds() void spirvbin_t::offsetIds()
{ {
// Count of how many functions each ID appears within // Count of how many functions each ID appears within
std::unordered_map<spv::Id, int> idFnCount; std::unordered_map<spv::Id, int> idFnCount;
std::unordered_map<spv::Id, int> idDefinedLoc; std::unordered_map<spv::Id, int> idDefinedLoc;
@ -710,14 +708,14 @@ void spirvbin_t::offsetIds()
if (idFnCount[id] == 1 && (instCount - idDefinedLoc[id]) < relOffsetLimit) if (idFnCount[id] == 1 && (instCount - idDefinedLoc[id]) < relOffsetLimit)
id = idDefinedLoc[id] - instCount; id = idDefinedLoc[id] - instCount;
}); });
} }
#endif #endif
// EXPERIMENTAL: forward IO and uniform load/stores into operands // EXPERIMENTAL: forward IO and uniform load/stores into operands
// This produces invalid Schema-0 SPIRV // This produces invalid Schema-0 SPIRV
void spirvbin_t::forwardLoadStores() void spirvbin_t::forwardLoadStores()
{ {
idset_t fnLocalVars; // set of function local vars idset_t fnLocalVars; // set of function local vars
idmap_t idMap; // Map of load result IDs to what they load idmap_t idMap; // Map of load result IDs to what they load
@ -772,11 +770,11 @@ void spirvbin_t::forwardLoadStores()
strip(); // strip out data we decided to eliminate strip(); // strip out data we decided to eliminate
buildLocalMaps(); // rebuild ID mapping data buildLocalMaps(); // rebuild ID mapping data
} }
// remove bodies of uncalled functions // remove bodies of uncalled functions
void spirvbin_t::optLoadStore() void spirvbin_t::optLoadStore()
{ {
idset_t fnLocalVars; idset_t fnLocalVars;
// Map of load result IDs to what they load // Map of load result IDs to what they load
idmap_t idMap; idmap_t idMap;
@ -858,11 +856,11 @@ void spirvbin_t::optLoadStore()
strip(); // strip out data we decided to eliminate strip(); // strip out data we decided to eliminate
buildLocalMaps(); // rebuild ID mapping data buildLocalMaps(); // rebuild ID mapping data
} }
// remove bodies of uncalled functions // remove bodies of uncalled functions
void spirvbin_t::dceFuncs() void spirvbin_t::dceFuncs()
{ {
msg(3, 2, std::string("Removing Dead Functions: ")); msg(3, 2, std::string("Removing Dead Functions: "));
// TODO: There are more efficient ways to do this. // TODO: There are more efficient ways to do this.
@ -905,11 +903,11 @@ void spirvbin_t::dceFuncs()
} else ++fn; } else ++fn;
} }
} }
} }
// remove unused function variables + decorations // remove unused function variables + decorations
void spirvbin_t::dceVars() void spirvbin_t::dceVars()
{ {
msg(3, 2, std::string("DCE Vars: ")); msg(3, 2, std::string("DCE Vars: "));
std::unordered_map<spv::Id, int> varUseCount; std::unordered_map<spv::Id, int> varUseCount;
@ -935,11 +933,11 @@ void spirvbin_t::dceVars()
return true; return true;
}, },
op_fn_nop); op_fn_nop);
} }
// remove unused types // remove unused types
void spirvbin_t::dceTypes() void spirvbin_t::dceTypes()
{ {
std::vector<bool> isType(bound(), false); std::vector<bool> isType(bound(), false);
// for speed, make O(1) way to get to type query (map is log(n)) // for speed, make O(1) way to get to type query (map is log(n))
@ -967,12 +965,12 @@ void spirvbin_t::dceTypes()
stripInst(typeStart); stripInst(typeStart);
} }
} }
} }
#ifdef NOTDEF #ifdef NOTDEF
bool spirvbin_t::matchType(const spirvbin_t::globaltypes_t& globalTypes, spv::Id lt, spv::Id gt) const bool spirvbin_t::matchType(const spirvbin_t::globaltypes_t& globalTypes, spv::Id lt, spv::Id gt) const
{ {
// Find the local type id "lt" and global type id "gt" // Find the local type id "lt" and global type id "gt"
const auto lt_it = typeConstPosR.find(lt); const auto lt_it = typeConstPosR.find(lt);
if (lt_it == typeConstPosR.end()) if (lt_it == typeConstPosR.end())
@ -1027,35 +1025,35 @@ bool spirvbin_t::matchType(const spirvbin_t::globaltypes_t& globalTypes, spv::Id
case spv::OpTypeSampler: return cmpLiteral() && cmpConst() && cmpSubType() && wordCount == 8; case spv::OpTypeSampler: return cmpLiteral() && cmpConst() && cmpSubType() && wordCount == 8;
default: return cmpLiteral() && cmpConst() && cmpSubType(); default: return cmpLiteral() && cmpConst() && cmpSubType();
} }
} }
// Look for an equivalent type in the globalTypes map // Look for an equivalent type in the globalTypes map
spv::Id spirvbin_t::findType(const spirvbin_t::globaltypes_t& globalTypes, spv::Id lt) const spv::Id spirvbin_t::findType(const spirvbin_t::globaltypes_t& globalTypes, spv::Id lt) const
{ {
// Try a recursive type match on each in turn, and return a match if we find one // Try a recursive type match on each in turn, and return a match if we find one
for (const auto& gt : globalTypes) for (const auto& gt : globalTypes)
if (matchType(globalTypes, lt, gt.first)) if (matchType(globalTypes, lt, gt.first))
return gt.first; return gt.first;
return spv::NoType; return spv::NoType;
} }
#endif // NOTDEF #endif // NOTDEF
// Return start position in SPV of given type. error if not found. // Return start position in SPV of given type. error if not found.
int spirvbin_t::typePos(spv::Id id) const int spirvbin_t::typePos(spv::Id id) const
{ {
const auto tid_it = typeConstPosR.find(id); const auto tid_it = typeConstPosR.find(id);
if (tid_it == typeConstPosR.end()) if (tid_it == typeConstPosR.end())
error("type ID not found"); error("type ID not found");
return tid_it->second; return tid_it->second;
} }
// Hash types to canonical values. This can return ID collisions (it's a bit // Hash types to canonical values. This can return ID collisions (it's a bit
// inevitable): it's up to the caller to handle that gracefully. // inevitable): it's up to the caller to handle that gracefully.
std::uint32_t spirvbin_t::hashType(int typeStart) const std::uint32_t spirvbin_t::hashType(int typeStart) const
{ {
const unsigned wordCount = asWordCount(typeStart); const unsigned wordCount = asWordCount(typeStart);
const spv::Op opCode = asOpCode(typeStart); const spv::Op opCode = asOpCode(typeStart);
@ -1130,10 +1128,10 @@ std::uint32_t spirvbin_t::hashType(int typeStart) const
error("unknown type opcode"); error("unknown type opcode");
return 0; return 0;
} }
} }
void spirvbin_t::mapTypeConst() void spirvbin_t::mapTypeConst()
{ {
globaltypes_t globalTypeMap; globaltypes_t globalTypeMap;
msg(3, 2, std::string("Remapping Consts & Types: ")); msg(3, 2, std::string("Remapping Consts & Types: "));
@ -1148,12 +1146,12 @@ void spirvbin_t::mapTypeConst()
if (isOldIdUnmapped(resId)) if (isOldIdUnmapped(resId))
localId(resId, nextUnusedId(hashval % softTypeIdLimit + firstMappedID)); localId(resId, nextUnusedId(hashval % softTypeIdLimit + firstMappedID));
} }
} }
// Strip a single binary by removing ranges given in stripRange // Strip a single binary by removing ranges given in stripRange
void spirvbin_t::strip() void spirvbin_t::strip()
{ {
if (stripRange.empty()) // nothing to do if (stripRange.empty()) // nothing to do
return; return;
@ -1175,13 +1173,16 @@ void spirvbin_t::strip()
spv.resize(strippedPos); spv.resize(strippedPos);
stripRange.clear(); stripRange.clear();
} }
// Strip a single binary by removing ranges given in stripRange // Strip a single binary by removing ranges given in stripRange
void spirvbin_t::remap(std::uint32_t opts) void spirvbin_t::remap(std::uint32_t opts)
{ {
options = opts; options = opts;
// Set up opcode tables from SpvDoc
spv::Parameterize();
validate(); // validate header validate(); // validate header
buildLocalMaps(); buildLocalMaps();
@ -1203,18 +1204,18 @@ void spirvbin_t::remap(std::uint32_t opts)
#define EXPERIMENT3 0 #define EXPERIMENT3 0
#if (EXPERIMENT3) #if (EXPERIMENT3)
// TODO: ... shortcuts for simple single-const access chains and constants, // TODO: ... shortcuts for simple single-const access chains and constants,
// folded into high half of the ID space. // folded into high half of the ID space.
#endif #endif
} }
// remap from a memory image // remap from a memory image
void spirvbin_t::remap(std::vector<std::uint32_t>& in_spv, std::uint32_t opts) void spirvbin_t::remap(std::vector<std::uint32_t>& in_spv, std::uint32_t opts)
{ {
spv.swap(in_spv); spv.swap(in_spv);
remap(opts); remap(opts);
spv.swap(in_spv); spv.swap(in_spv);
} }
} // namespace SPV } // namespace SPV

6
StandAlone/CMakeLists.txt
View File

@ -10,8 +10,10 @@ else(WIN32)
endif(WIN32) endif(WIN32)
set(SOURCES StandAlone.cpp) set(SOURCES StandAlone.cpp)
set(REMAPPER_SOURCES spirv-remap.cpp)
add_executable(glslangValidator ${SOURCES}) add_executable(glslangValidator ${SOURCES})
add_executable(spirv-remap ${REMAPPER_SOURCES})
set(LIBRARIES set(LIBRARIES
glslang glslang
@ -26,6 +28,7 @@ elseif(UNIX)
endif(WIN32) endif(WIN32)
target_link_libraries(glslangValidator ${LIBRARIES}) target_link_libraries(glslangValidator ${LIBRARIES})
target_link_libraries(spirv-remap ${LIBRARIES})
if(WIN32) if(WIN32)
source_group("Source" FILES ${SOURCES}) source_group("Source" FILES ${SOURCES})
@ -33,3 +36,6 @@ endif(WIN32)
install(TARGETS glslangValidator install(TARGETS glslangValidator
RUNTIME DESTINATION bin) RUNTIME DESTINATION bin)
install(TARGETS spirv-remap
RUNTIME DESTINATION bin)

336
StandAlone/spirv-remap.cpp Normal file
View File

@ -0,0 +1,336 @@
//
//Copyright (C) 2015 LunarG, Inc.
//
//All rights reserved.
//
//Redistribution and use in source and binary forms, with or without
//modification, are permitted provided that the following conditions
//are met:
//
// Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.
//
// Redistributions in binary form must reproduce the above
// copyright notice, this list of conditions and the following
// disclaimer in the documentation and/or other materials provided
// with the distribution.
//
// Neither the name of 3Dlabs Inc. Ltd. nor the names of its
// contributors may be used to endorse or promote products derived
// from this software without specific prior written permission.
//
//THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
//"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
//LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
//FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
//COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
//INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
//BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
//LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
//CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
//LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
//ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
//POSSIBILITY OF SUCH DAMAGE.
//
#include <iostream>
#include <fstream>
#include <cstring>
#include <stdexcept>
#include "../SPIRV/SPVRemapper.h"
namespace {
typedef unsigned int SpvWord;
// Poor man's basename: given a complete path, return file portion.
// E.g:
// Linux: /foo/bar/test -> test
// Win: c:\foo\bar\test -> test
// It's not very efficient, but that doesn't matter for our minimal-duty use.
// Using boost::filesystem would be better in many ways, but want to avoid that dependency.
// OS dependent path separator (avoiding boost::filesystem dependency)
#if defined(_WIN32)
char path_sep_char() { return '\\'; }
#else
char path_sep_char() { return '/'; }
#endif
std::string basename(const std::string filename)
{
const size_t sepLoc = filename.find_last_of(path_sep_char());
return (sepLoc == filename.npos) ? filename : filename.substr(sepLoc+1);
}
void errHandler(const std::string& str) {
std::cout << str << std::endl;
exit(5);
}
void logHandler(const std::string& str) {
std::cout << str << std::endl;
}
// Read word stream from disk
void read(std::vector<SpvWord>& spv, const std::string& inFilename)
{
std::ifstream fp;
std::cout << " reading: " << inFilename << std::endl;
spv.clear();
fp.open(inFilename, std::fstream::in | std::fstream::binary);
if (fp.fail())
errHandler("error opening file for read: ");
// Reserve space (for efficiency, not for correctness)
fp.seekg(0, fp.end);
spv.reserve(size_t(fp.tellg()) / sizeof(SpvWord));
fp.seekg(0, fp.beg);
while (!fp.eof()) {
SpvWord inWord;
fp.read((char *)&inWord, sizeof(inWord));
if (!fp.eof()) {
spv.push_back(inWord);
if (fp.fail())
errHandler(std::string("error reading file: ") + inFilename);
}
}
}
void write(std::vector<SpvWord>& spv, const std::string& outFile)
{
if (outFile.empty())
errHandler("missing output filename.");
std::ofstream fp;
std::cout << " writing: " << outFile << std::endl;
fp.open(outFile, std::fstream::out | std::fstream::binary);
if (fp.fail())
errHandler(std::string("error opening file for write: ") + outFile);
for (auto word : spv) {
fp.write((char *)&word, sizeof(word));
if (fp.fail())
errHandler(std::string("error writing file: ") + outFile);
}
// file is closed by destructor
}
// Print helpful usage message to stdout, and exit
void usage(const char* const name, const char* const msg = 0)
{
if (msg)
std::cout << msg << std::endl << std::endl;
std::cout << "Usage: " << std::endl;
std::cout << " " << basename(name)
<< " [-v[v[...]] | --verbose [int]]"
<< " [--map (all|types|names|funcs)]"
<< " [--dce (all|types|funcs)]"
<< " [--opt (all|loadstore)]"
<< " [--strip-all | --strip all | -s]"
<< " [--do-everything]"
<< " --input | -i file1 [file2...] --output|-o DESTDIR"
<< std::endl;
std::cout << " " << basename(name) << " [--version | -V]" << std::endl;
std::cout << " " << basename(name) << " [--help | -?]" << std::endl;
exit(5);
}
// grind through each SPIR in turn
void execute(const std::vector<std::string>& inputFile, const std::string& outputDir,
int opts, int verbosity)
{
for (const auto& filename : inputFile) {
std::vector<SpvWord> spv;
read(spv, filename);
spv::spirvbin_t(verbosity).remap(spv, opts);
const std::string outfile = outputDir + path_sep_char() + basename(filename);
write(spv, outfile);
}
if (verbosity > 0)
std::cout << "Done: " << inputFile.size() << " file(s) processed" << std::endl;
}
// Parse command line options
void parseCmdLine(int argc, char** argv, std::vector<std::string>& inputFile,
std::string& outputDir,
int& options,
int& verbosity)
{
if (argc < 2)
usage(argv[0]);
verbosity = 0;
options = spv::spirvbin_t::Options::NONE;
// Parse command line.
// boost::program_options would be quite a bit nicer, but we don't want to
// introduce a dependency on boost.
for (int a=1; a<argc; ) {
const std::string arg = argv[a];
if (arg == "--output" || arg == "-o") {
// Output directory
if (++a >= argc)
usage(argv[0], "--output requires an argument");
if (!outputDir.empty())
usage(argv[0], "--output can be provided only once");
outputDir = argv[a++];
// Remove trailing directory separator characters
while (!outputDir.empty() && outputDir.back() == path_sep_char())
outputDir.pop_back();
}
else if (arg == "-vv") { verbosity = 2; ++a; } // verbosity shortcuts
else if (arg == "-vvv") { verbosity = 3; ++a; } // ...
else if (arg == "-vvvv") { verbosity = 4; ++a; } // ...
else if (arg == "-vvvvv") { verbosity = 5; ++a; } // ...
else if (arg == "--verbose" || arg == "-v") {
++a;
verbosity = 1;
if (a < argc) {
try {
verbosity = std::stoi(argv[a]);
++a;
} catch (const std::invalid_argument&) { } // ok to have no numeric value
}
}
else if (arg == "--version" || arg == "-V") {
std::cout << basename(argv[0]) << " version 0.97 " << __DATE__ << " " << __TIME__ << std::endl;
exit(0);
} else if (arg == "--input" || arg == "-i") {
// Collect input files
for (++a; a < argc && argv[a][0] != '-'; ++a)
inputFile.push_back(argv[a]);
} else if (arg == "--do-everything") {
++a;
options = options | spv::spirvbin_t::Options::DO_EVERYTHING;
} else if (arg == "--strip-all" || arg == "-s") {
++a;
options = options | spv::spirvbin_t::Options::STRIP;
} else if (arg == "--strip") {
++a;
if (strncmp(argv[a], "all", 3) == 0) {
options = options | spv::spirvbin_t::Options::STRIP;
++a;
}
} else if (arg == "--dce") {
// Parse comma (or colon, etc) separated list of things to dce
++a;
for (const char* c = argv[a]; *c; ++c) {
if (strncmp(c, "all", 3) == 0) {
options = (options | spv::spirvbin_t::Options::DCE_ALL);
c += 3;
} else if (strncmp(c, "*", 1) == 0) {
options = (options | spv::spirvbin_t::Options::DCE_ALL);
c += 1;
} else if (strncmp(c, "funcs", 5) == 0) {
options = (options | spv::spirvbin_t::Options::DCE_FUNCS);
c += 5;
} else if (strncmp(c, "types", 5) == 0) {
options = (options | spv::spirvbin_t::Options::DCE_TYPES);
c += 5;
}
}
++a;
} else if (arg == "--map") {
// Parse comma (or colon, etc) separated list of things to map
++a;
for (const char* c = argv[a]; *c; ++c) {
if (strncmp(c, "all", 3) == 0) {
options = (options | spv::spirvbin_t::Options::MAP_ALL);
c += 3;
} else if (strncmp(c, "*", 1) == 0) {
options = (options | spv::spirvbin_t::Options::MAP_ALL);
c += 1;
} else if (strncmp(c, "types", 5) == 0) {
options = (options | spv::spirvbin_t::Options::MAP_TYPES);
c += 5;
} else if (strncmp(c, "names", 5) == 0) {
options = (options | spv::spirvbin_t::Options::MAP_NAMES);
c += 5;
} else if (strncmp(c, "funcs", 5) == 0) {
options = (options | spv::spirvbin_t::Options::MAP_FUNCS);
c += 5;
}
}
++a;
} else if (arg == "--opt") {
++a;
for (const char* c = argv[a]; *c; ++c) {
if (strncmp(c, "all", 3) == 0) {
options = (options | spv::spirvbin_t::Options::OPT_ALL);
c += 3;
} else if (strncmp(c, "*", 1) == 0) {
options = (options | spv::spirvbin_t::Options::OPT_ALL);
c += 1;
} else if (strncmp(c, "loadstore", 9) == 0) {
options = (options | spv::spirvbin_t::Options::OPT_LOADSTORE);
c += 9;
}
}
++a;
} else if (arg == "--help" || arg == "-?") {
usage(argv[0]);
} else {
usage(argv[0], "Unknown command line option");
}
}
}
} // namespace
int main(int argc, char** argv)
{
#ifdef use_cpp11
std::vector<std::string> inputFile;
std::string outputDir;
int opts;
int verbosity;
// handle errors by exiting
spv::spirvbin_t::registerErrorHandler(errHandler);
// Log messages to std::cout
spv::spirvbin_t::registerLogHandler(logHandler);
if (argc < 2)
usage(argv[0]);
parseCmdLine(argc, argv, inputFile, outputDir, opts, verbosity);
if (outputDir.empty())
usage(argv[0], "Output directory required");
std::string errmsg;
// Main operations: read, remap, and write.
execute(inputFile, outputDir, opts, verbosity);
#endif
// If we get here, everything went OK! Nothing more to be done.
}