TheRock/rocm-libraries/shared/rocroller/docs/src/DesignOverview.md

Design Overview

• Design Overview
  • Introduction
    • Overview
  • Command
    • Command Class
    • Operations
    • CommandArgument
  • Intermediate Graph Representation
    • Kernel Graph
    • Expression
    • Generating a Kernel Graph from a Command
  • Machine Code Representation
    • Instructions
    • Registers
    • Generating Instructions
    • Loading Arguments
    • Generating Instructions from a KernelGraph
    • Context
    • Scheduling Instructions
    • Wait Count
    • Assembly Kernel
    • Scheduled Instructions
  • Kernel Executer
    • Assembler
    • Executable Kernel
    • Command Kernel
  • GPUArchitectureInformation
    • GPUArchitectureGenerator
    • GPUArchitectureLibrary
  • Other Topics
    • Components
    • Serialization
    • Generator

Introduction

The rocRoller Library provides functionality for expressing a collection of operations that will be executed on a GPU. This is referred to as a Command. That Command can then be converted into highly performant GPU assembly code.

The rocRoller Library also provides methods for executing the generated code on an AMD GPU with arguments provided by the user.

Overview

A user can represent a series of operations to be performed on a GPU using the Command class. This class makes it easy to represent operations on tensors and scalars.

Once a Command object has been created, it can be translated into a graph representation called a KernelGraph. Transformations can be performed on this graph to optimize the performance of the kernel.

After optimizations have been performed on the KernelGraph, it can be translated into a series of Instructions. An Instruction is a representation of a machine code instruction.

The series of Instructions can then be converted into an assembly file, which is passed to LLVM to create object code for the kernel.

The user can execute this kernel with their own arguments by using the CommandKernel class.

Command

A command object is a high level representation of the operations that will be performed on a GPU. These operations could be things such as: loading/storing data, adding two vectors, or multiplying two vectors together.

The definitions for everything needed to create a Command object is found within the lib/include/rocRoller/Operations directory.

Command Class

A Command object consists of a collection of operations and runtime arguments to those operations. When an operation is added to a Command, a tag is created for that operation. Other operations can then refer to that tag to reference the result of the operation. Some operations (such as Load/Store operations) can also allocate runtime arguments of type CommandArgument.

Operations

All of the different operations that can be added to a Command should be listed within the Operations.hpp file. An operation should, at the bare minimum, have a collection of inputs and outputs.

The T_Execute operation can contain a list of other operations that start with E_. These operations perform simple operations upon vectors or scalars. These operations could be things such as Add or Subtract.

Load and Store operations will also allocate various CommandArguments describing the data that should be either loaded or stored. The actual values for these CommandArguments will be provided by the user when the kernel is executed.

CommandArgument

A CommandArgument represents a runtime argument that a user will provide when the kernel is executed. These can be pointers to data on the GPU, or scalar values.

Intermediate Graph Representation

A Command object can be represented as a graph. This internal representation can make it easier to reason about the Command, as well as perform optimizations upon it.

Kernel Graph

The high-level operations in a Command are translated into a KernelGraph. The KernelGraph is iteratively rewritten (or lowered) until it becomes a low-level representation of the original Command. GPU code is then generated by traversing the graph and emitting Instructions.

A KernelGraph encodes:

1. Coordinate transforms: how vector/matrix/tensor indexes are computed from each other.
2. Control flow and operations: operations and their dependencies.
3. Data flow and distribution: how data flows through the GPU and how it is distributed.

Nodes in the coordinate transform graph are instances of the Dimension variant. A Dimension can represent a few different things: from simple dimensions (with size and stride), to indexes of a for loop.

Edges in the coordinate transform graph describe how indexes of dimensions are transformed. For example, the Flatten edge takes multiple source indexes and computes a row-major, contiguous "flat" destination index. Edges connect multiple source nodes (tail set) to multiple destination nodes (head set), and hence the coordination transform graph is a hyper-graph.

Nodes in the control flow graph are instances of the Operation variant. An Operation can represent operations like LoadVGPR or Multiply; or control flow constructs like ForLoop or If. Control flow nodes typically contain another instance of a control graph that represent their bodies.

Graphs are implemented using C++'s variant. Coordinate transforms are implemented in CoordinateTransform.cpp, and graph translation, rewriting, and code generation are implemented in KernelGraph.cpp.

Expression

An Expression is used to represent a mathematical expression upon arguments or literal values. The operations that are supported within an Expression are listed within Expression.hpp.

Expressions are used in various locations throughout the code base. But, they can only contain certain types of values at certain points. The EvaluationTime enum can be used to describe when a certain expression can be used.

Expressions are represented using C++'s variant. Because of this, the std::visit function is normally used to view the Expression tree and make transformations.

Transformations can be performed on an Expression to improve performance. A list of transformations can be found within ExpressionTransformations.hpp.

Generating a Kernel Graph from a Command

A KernelGraph can be created from a Command by using the KernelGraph::translate function. The translate function will visit each Operation within a Command and add the appropriate edges into the graph.

Machine Code Representation

After a KernelGraph has been created, it can be converted into a low level representation of machine code instructions. These are represented with the Instruction class. These instructions are then scheduled in an attempt to achieve the best performance. The list of scheduled instructions can then be turned into an assembly file that contains the entire kernel.

Instructions

An Instruction object usually represents a single assembly instruction. A standard instruction contains a name, a list of destination registers, a list of source registers, a list of modifiers, and a comment. Other types of Instruction objects include:

• Waitcount instructions, e.g. s_waitcnt 0
• No-op instructions, e.g. s_nop 2
• Labels, e.g. 'MyKernel:`
• Register allocations
• Assembler directives, e.g. .amdgcn_target "amdgcn-amd-amdhsa--gfx906"
• Comments

Waitcounts, no-ops, labels, and allocations can be attached to a standard instruction manually, or by Observer objects in the Context.

An Instruction object can be easily converted into a string that can appear in an assembly file.

The Instruction.hpp file contains a list of static constructors for creating new instances of specific instructions, such as Comment or Wait.

Registers

A Register::Value object represents a single register on the GPU. It can be provided to an Instruction as an input or output. A Register::Value can represent a single value that is stored within one or more actual registers, or an array of registers.

A Register::Value has a Register::Allocation object associated with it. This is used to select the actual registers that will be used for a Register::Value. For example, if a Register::Value is a scalar 64-bit integer, the Register::Allocation might pick to use registers s4 and s5. The Allocator object (owned by the Context) uses the Allocation object's lifetime to manage allocating and freeing registers.

A Register::Value can also be used to represent a literal value or label that can be used as an argument to an Instruction. The Register.hpp file provides a collection of constructors for creating different types of Register::Values.

Generating Instructions

In general, it is best not to directly create an Instruction object. Instead, different classes are provided that will generate Instruction objects. The generateOp<Expression> function can be used to generate Instruction objects for different Expression operations.

Instructions to load and store data to and from memory can be generated using the MemoryInstructions class. For a list of available load and store instructions, see MemoryInstructions.hpp.

Loading Arguments

The ArgumentLoader class provides methods for generating instructions to load arguments to a kernel into registers. The ArgumentLoader keeps track of which arguments have already been loaded into registers, so that they aren't loaded twice.

The ArgumentLoader class provides a method for allocating all arguments at once. This can be useful to do at the beginning of a kernel. If this isn't called, all arguments will be loaded as needed.

Generating Instructions from a KernelGraph

A list of Instructions can be generated for a particular KernelGraph by calling the KernelGraph::generate function. The KernelGraph::generate function will visit each edge within the graph and attempt to generate a series of Instructions. Ideally, the Instruction constructor will never be used here. Instead, there should be calls to classes such as ArithmeticGenerator, which will generate the appropriate instructions for a particular architecture.

Edges within a graph might include Expressions. Instructions might need to be generated for these as well. The Expression::generate() function can be used to generate instructions for an expression. It will visit each node within an Expression tree and attempt to generate instructions for the node.

Context

A Context object is used to generate and schedule instructions for a particular architecture. A Context also contains a collection of Register:Allocators to keep track of which registers are available to be used within a kernel. A single Context object should be created for generating instructions for a single kernel on a specific architecture.

A GPUArchitecture object is used to specify which AMD GPU the Context should be generating instructions for. A particular GPUArchitecture will have information on which instructions and features are available. It will also have information on the costs of executing specific instructions on that architecture. This information is stored within GPUInstructionInfo objects.

Scheduling Instructions

A Context object has a schedule method that can be called to schedule a series of Instructions. Instructions are added to a kernel by calling a Context's schedule method. The schedule method will go through a list of observers that could potentially modify an Instruction. It will then add the modified instruction to a list of scheduled instructions.

An IObserver class provides three methods: peek, modify and observe.

The peek method will return information about the state of the machine if this instruction were to be scheduled. This could include things like stall cycles needing to be added or errors that may have occurred.

The modify method will modify an Instruction by adding things such as wait counts to it.

The observe method will modify any state after the Instruction has been scheduled. This could include keeping track of the state of various queues that the Instruction has affected.

The MetaObserver class holds a collection of IObserver types. It will call each IObserver's peek, modify and observe methods. A Context contains a MetaObserver object that will be called by the schedule method.

All of the available IObserver types can be found in the lib/include/rocRoller/Scheduling/Observers directory.

Wait Count

Some instructions on AMD GPUs need to wait for certain conditions until their results can be used. A waitcnt instruction needs to be generated in these situations. There are different queues on the machine, and a waitcnt instruction can specify to wait for the different queues to reach a certain size before continuing. A WaitCount object represents these wait conditions and can be added to an Instruction.

The WaitcntObserver is an IObserver that adds a WaitCount to scheduled instructions based on the instructions that have already been scheduled. The WaitcntObserver will get information from a Context's GPUArchitecture to determine wait counts needed for different instructions.

Assembly Kernel

An AssemblyKernel object contains all of the information needed to generate an assembly file for a kernel. It provides methods for generating instructions and information that need to be added to the beginning and end of the file.

Arguments need to be added to the AssemblyKernel before generating the initial setup instructions. The instructions for the body of the kernel need to be generated before generating the instructions at the end of the file.

It is also necessary to set the Work Group Size, Work Item Count and the Dynamic Shared Memory Bytes.

Scheduled Instructions

All of the instructions scheduled by a Context, including those that came from the AssemblyKernel, are stored within a ScheduleInstructions object. This object can be converted into a string that contains the entire contents of an assembly file representing a single kernel.

Kernel Executer

Assembler

In order to execute the kernel, the assembly instructions that are stored within a Context's scheduled instructions must be converted into object code. This is performed by the Assembler class.

The Assembler class uses AMD Comgr's API to assemble the instructions without having to write an assembly file to disk.

Executable Kernel

After object code has been created for a kernel, it can be executed on a GPU with provided arguments. An ExecutableKernel object will load object code generated by the Assembler class using HIP's hipModuleLoadData function. Once the kernel has been loaded, it can be executed on the GPU using HIP's hipExtModuleLaunchKernel function. Arguments are passed to the kernel using a KernelArguments object.

Command Kernel

In general, the ExecutableKernel object should not be called directly. Instead, a CommandKernel object should be created, which will contain an ExecutableKernel. A CommandKernel object can take a Command object and create an ExecutableKernel from it. It does this by generating a KernelGraph from a Command, then generating ScheduledInstructions from the KernelGraph and finally generating an ExecutableKernel from those instructions.

Once a CommandKernel object has been created, it can be launched on a GPU with unique arguments by using the launchKernel method. The arguments are provided with a RuntimeArguments object.

A RuntimeArguments object is a std::span<uint8_t const> that provides a view into a block of memory containing all of the arguments that will be passed to the kernel in the correct order. The actual memory is typically managed by a KernelArguments object, which provides methods for constructing and manipulating the argument buffer.

GPUArchitectureInformation

GPUArchitectureGenerator

The main ideas behind the GPUArchitectureGenerator are the following:

• minimize the runtime overhead of looking up architecture and instruction information
• simplify the specification of architecture and instruction information

To keep runtime overhead low we're storing this information in dictionaries (std::map<> and std::unordered_map<>). However, explicitly specifying these dictionaries in source files is unwieldy and difficult to maintain, so we'd like to be able to specify this info in a more programmatic way, but that would negatively impact runtime performance.

To satisfy both of these requirements we have the GPUArchitectureGenerator, which allows us to populate architecture and instruction dictionaries in a programmatic method at compile time and serialize it. Those serialized dictionaries are read in and deserialized at runtime.

The GPUArchitectureGenerator::FillArchitectures() function fills in GPUArchitectureGenerator::GPUArchitectures using various predicates, lists, and dictionaries defined in GPUArchitectureGenerator/include/GPUArchitectureGenerator/GPUArchitectureGenerator_defs.hpp.

There is a custom cmake rule, GPUArchitecture_def, which uses the GPUArchitectureGenerator to generate GPUArchitecture_def.yaml at compile time.

GPUArchitectureLibrary

At runtime, the GPUArchitectureLibrary class can be used to query architecture and instruction information. For example, the following determines if GFX90a supports DirectToLDS:

GPUArchitectureLibrary::GetArch("GFX90a").HasCapability(GPUCapability::HasDirectToLds)

And the following gets the wait count for the global_store_byte instruction on GFX90a:

GPUArchitectureLibrary::GetArch("GFX90a").GetInstructionInfo("global_store_byte").getWaitCount()

Other Topics

Components

The Component system provides a flexible plugin architecture for swapping implementations based on runtime conditions. Components are organized into base classes (defining interfaces) and concrete implementations (providing specific functionality).

A ComponentBase defines:

• An Argument type used to select the appropriate implementation
• A Basename string identifier

Concrete Component implementations provide:

• A Match function to determine if this component is appropriate for given arguments
• A Build factory function to create instances
• A Name identifier

The ComponentFactory manages registration and selection of components. When Component::Get<Base>(args) is called:

1. The factory searches registered components for a matching implementation
2. For non-single-use components, instances are cached and reused
3. For single-use components, new instances are created each time

Components are thread-safe, using reader-writer locks for concurrent access. This pattern is used throughout rocRoller for architecture-specific code generation (e.g., different assembler implementations).

Serialization

rocRoller provides a flexible serialization framework for converting objects to and from YAML format. The serialization system supports both LLVM's YAML library and yaml-cpp.

Key serialization traits:

• ScalarTraits<T>: For basic types and enums, providing output() and input() methods
• MappingTraits<T, IO, Context>: For structured types with named fields
• SequenceTraits<T, IO>: For sequential collections
• EnumTraits<T, IO>: For enumeration types

The main API functions are:

• toYAML(obj): Converts an object to a YAML string
• fromYAML<T>(yaml): Parses a YAML string into an object
• writeYAML(stream, obj): Writes YAML to an output stream

Many rocRoller types implement serialization traits, including Command, KernelGraph, Expression, and Operation types. This enables saving and loading kernel definitions, intermediate representations, and configuration data. Serialization is particularly useful for debugging, caching compiled kernels, and inspecting internal structures.

Generator

The Generator<T> class template implements lazy sequences using C++20 coroutines. It provides an efficient way to produce sequences of values on-demand without storing the entire collection in memory.

Key features:

• Implements std::ranges::input_range for compatibility with range-based algorithms
• Supports co_yield for single values ranges
• Provides convenience methods: map(), filter(), take(), only(), empty()
• Can create standard collections with .to<Container>()

Generators are movable but not copyable. The prefix increment operator (++iter) is more efficient than postfix (iter++). Generators don't begin executing until the first value is requested.

Example usage pattern:

Generator<int> numbers() {
    for(int i = 0; i < 10; i++)
        co_yield i;
}

// Use with range-based for
for(int n : numbers().filter([](int x) { return x % 2 == 0; }))
    std::cout << n << std::endl;

// Convert to vector
auto vec = numbers().take(5).to<std::vector>();

Generators are used internally in rocRoller for producing sequences of instructions during code generation, allowing lazy evaluation and efficient memory usage.