# Design Overview - [Design Overview](#design-overview) - [Introduction](#introduction) - [Overview](#overview) - [Command](#command) - [Command Class](#command-class) - [Operations](#operations) - [CommandArgument](#commandargument) - [Intermediate Graph Representation](#intermediate-graph-representation) - [Kernel Graph](#kernel-graph) - [Expression](#expression) - [Generating a Kernel Graph from a Command](#generating-a-kernel-graph-from-a-command) - [Machine Code Representation](#machine-code-representation) - [Instructions](#instructions) - [Registers](#registers) - [Generating Instructions](#generating-instructions) - [Loading Arguments](#loading-arguments) - [Generating Instructions from a KernelGraph](#generating-instructions-from-a-kernelgraph) - [Context](#context) - [Scheduling Instructions](#scheduling-instructions) - [Wait Count](#wait-count) - [Assembly Kernel](#assembly-kernel) - [Scheduled Instructions](#scheduled-instructions) - [Kernel Executer](#kernel-executer) - [Assembler](#assembler) - [Executable Kernel](#executable-kernel) - [Command Kernel](#command-kernel) - [GPUArchitectureInformation](#gpuarchitectureinformation) - [GPUArchitectureGenerator](#gpuarchitecturegenerator) - [GPUArchitectureLibrary](#gpuarchitecturelibrary) - [Other Topics](#other-topics) - [Components](#components) - [Serialization](#serialization) - [Generator](#generator) ## Introduction The rocRoller Library provides functionality for expressing a collection of operations that will be executed on a GPU. This is referred to as a `Command`. That `Command` can then be converted into highly performant GPU assembly code. The rocRoller Library also provides methods for executing the generated code on an AMD GPU with arguments provided by the user. ### Overview A user can represent a series of operations to be performed on a GPU using the `Command` class. This class makes it easy to represent operations on tensors and scalars. Once a `Command` object has been created, it can be translated into a graph representation called a `KernelGraph`. Transformations can be performed on this graph to optimize the performance of the kernel. After optimizations have been performed on the `KernelGraph`, it can be translated into a series of `Instruction`s. An `Instruction` is a representation of a machine code instruction. The series of `Instruction`s can then be converted into an assembly file, which is passed to LLVM to create object code for the kernel. The user can execute this kernel with their own arguments by using the `CommandKernel` class. ## Command A command object is a high level representation of the operations that will be performed on a GPU. These operations could be things such as: loading/storing data, adding two vectors, or multiplying two vectors together. The definitions for everything needed to create a Command object is found within the `lib/include/rocRoller/Operations` directory. ### Command Class A `Command` object consists of a collection of operations and runtime arguments to those operations. When an operation is added to a `Command`, a tag is created for that operation. Other operations can then refer to that tag to reference the result of the operation. Some operations (such as Load/Store operations) can also allocate runtime arguments of type `CommandArgument`. ### Operations All of the different operations that can be added to a `Command` should be listed within the `Operations.hpp` file. An operation should, at the bare minimum, have a collection of inputs and outputs. The `T_Execute` operation can contain a list of other operations that start with `E_`. These operations perform simple operations upon vectors or scalars. These operations could be things such as `Add` or `Subtract`. Load and Store operations will also allocate various `CommandArgument`s describing the data that should be either loaded or stored. The actual values for these `CommandArgument`s will be provided by the user when the kernel is executed. ### CommandArgument A `CommandArgument` represents a runtime argument that a user will provide when the kernel is executed. These can be pointers to data on the GPU, or scalar values. ## Intermediate Graph Representation A `Command` object can be represented as a graph. This internal representation can make it easier to reason about the `Command`, as well as perform optimizations upon it. ### Kernel Graph The high-level operations in a `Command` are translated into a `KernelGraph`. The `KernelGraph` is iteratively rewritten (or lowered) until it becomes a low-level representation of the original `Command`. GPU code is then generated by traversing the graph and emitting `Instruction`s. A `KernelGraph` encodes: 1. Coordinate transforms: how vector/matrix/tensor indexes are computed from each other. 2. Control flow and operations: operations and their dependencies. 3. Data flow and distribution: how data flows through the GPU and how it is distributed. Nodes in the coordinate transform graph are instances of the `Dimension` variant. A `Dimension` can represent a few different things: from simple dimensions (with size and stride), to indexes of a `for` loop. Edges in the coordinate transform graph describe how indexes of dimensions are transformed. For example, the `Flatten` edge takes multiple source indexes and computes a row-major, contiguous "flat" destination index. Edges connect multiple source nodes (tail set) to multiple destination nodes (head set), and hence the coordination transform graph is a hyper-graph. Nodes in the control flow graph are instances of the `Operation` variant. An `Operation` can represent operations like `LoadVGPR` or `Multiply`; or control flow constructs like `ForLoop` or `If`. Control flow nodes typically contain another instance of a control graph that represent their bodies. Graphs are implemented using C++'s `variant`. Coordinate transforms are implemented in `CoordinateTransform.cpp`, and graph translation, rewriting, and code generation are implemented in `KernelGraph.cpp`. ### Expression An `Expression` is used to represent a mathematical expression upon arguments or literal values. The operations that are supported within an `Expression` are listed within `Expression.hpp`. Expressions are used in various locations throughout the code base. But, they can only contain certain types of values at certain points. The `EvaluationTime` enum can be used to describe when a certain expression can be used. Expressions are represented using C++'s `variant`. Because of this, the `std::visit` function is normally used to view the Expression tree and make transformations. Transformations can be performed on an `Expression` to improve performance. A list of transformations can be found within `ExpressionTransformations.hpp`. ### Generating a Kernel Graph from a Command A `KernelGraph` can be created from a `Command` by using the `KernelGraph::translate` function. The `translate` function will visit each `Operation` within a `Command` and add the appropriate edges into the graph. ## Machine Code Representation After a `KernelGraph` has been created, it can be converted into a low level representation of machine code instructions. These are represented with the `Instruction` class. These instructions are then scheduled in an attempt to achieve the best performance. The list of scheduled instructions can then be turned into an assembly file that contains the entire kernel. ### Instructions An `Instruction` object usually represents a single assembly instruction. A standard instruction contains a name, a list of destination registers, a list of source registers, a list of modifiers, and a comment. Other types of `Instruction` objects include: - Waitcount instructions, e.g. `s_waitcnt 0` - No-op instructions, e.g. `s_nop 2` - Labels, e.g. 'MyKernel:` - Register allocations - Assembler directives, e.g. `.amdgcn_target "amdgcn-amd-amdhsa--gfx906"` - Comments Waitcounts, no-ops, labels, and allocations can be attached to a standard instruction manually, or by Observer objects in the Context. An `Instruction` object can be easily converted into a string that can appear in an assembly file. The `Instruction.hpp` file contains a list of static constructors for creating new instances of specific instructions, such as `Comment` or `Wait`. ### Registers A `Register::Value` object represents a single register on the GPU. It can be provided to an `Instruction` as an input or output. A `Register::Value` can represent a single value that is stored within one or more actual registers, or an array of registers. A `Register::Value` has a `Register::Allocation` object associated with it. This is used to select the actual registers that will be used for a `Register::Value`. For example, if a `Register::Value` is a scalar 64-bit integer, the `Register::Allocation` might pick to use registers `s4` and `s5`. The `Allocator` object (owned by the `Context`) uses the `Allocation` object's lifetime to manage allocating and freeing registers. A `Register::Value` can also be used to represent a literal value or label that can be used as an argument to an `Instruction`. The `Register.hpp` file provides a collection of constructors for creating different types of `Register::Value`s. ### Generating Instructions In general, it is best not to directly create an `Instruction` object. Instead, different classes are provided that will generate `Instruction` objects. The `generateOp` function can be used to generate `Instruction` objects for different `Expression` operations. Instructions to load and store data to and from memory can be generated using the `MemoryInstructions` class. For a list of available load and store instructions, see `MemoryInstructions.hpp`. ### Loading Arguments The `ArgumentLoader` class provides methods for generating instructions to load arguments to a kernel into registers. The `ArgumentLoader` keeps track of which arguments have already been loaded into registers, so that they aren't loaded twice. The `ArgumentLoader` class provides a method for allocating all arguments at once. This can be useful to do at the beginning of a kernel. If this isn't called, all arguments will be loaded as needed. ### Generating Instructions from a KernelGraph A list of `Instruction`s can be generated for a particular `KernelGraph` by calling the `KernelGraph::generate` function. The `KernelGraph::generate` function will visit each edge within the graph and attempt to generate a series of `Instruction`s. Ideally, the `Instruction` constructor will never be used here. Instead, there should be calls to classes such as `ArithmeticGenerator`, which will generate the appropriate instructions for a particular architecture. Edges within a graph might include `Expression`s. Instructions might need to be generated for these as well. The `Expression::generate()` function can be used to generate instructions for an expression. It will visit each node within an `Expression` tree and attempt to generate instructions for the node. ### Context A `Context` object is used to generate and schedule instructions for a particular architecture. A `Context` also contains a collection of `Register:Allocator`s to keep track of which registers are available to be used within a kernel. A single `Context` object should be created for generating instructions for a single kernel on a specific architecture. A `GPUArchitecture` object is used to specify which AMD GPU the `Context` should be generating instructions for. A particular `GPUArchitecture` will have information on which instructions and features are available. It will also have information on the costs of executing specific instructions on that architecture. This information is stored within `GPUInstructionInfo` objects. ### Scheduling Instructions A `Context` object has a `schedule` method that can be called to schedule a series of `Instruction`s. Instructions are added to a kernel by calling a `Context`'s `schedule` method. The `schedule` method will go through a list of observers that could potentially modify an `Instruction`. It will then add the modified instruction to a list of scheduled instructions. An `IObserver` class provides three methods: `peek`, `modify` and `observe`. The `peek` method will return information about the state of the machine if this instruction were to be scheduled. This could include things like stall cycles needing to be added or errors that may have occurred. The `modify` method will modify an `Instruction` by adding things such as wait counts to it. The `observe` method will modify any state after the `Instruction` has been scheduled. This could include keeping track of the state of various queues that the `Instruction` has affected. The `MetaObserver` class holds a collection of `IObserver` types. It will call each `IObserver`'s `peek`, `modify` and `observe` methods. A `Context` contains a `MetaObserver` object that will be called by the `schedule` method. All of the available `IObserver` types can be found in the `lib/include/rocRoller/Scheduling/Observers` directory. ### Wait Count Some instructions on AMD GPUs need to wait for certain conditions until their results can be used. A `waitcnt` instruction needs to be generated in these situations. There are different queues on the machine, and a `waitcnt` instruction can specify to wait for the different queues to reach a certain size before continuing. A `WaitCount` object represents these wait conditions and can be added to an `Instruction`. The `WaitcntObserver` is an `IObserver` that adds a `WaitCount` to scheduled instructions based on the instructions that have already been scheduled. The `WaitcntObserver` will get information from a `Context`'s `GPUArchitecture` to determine wait counts needed for different instructions. ### Assembly Kernel An `AssemblyKernel` object contains all of the information needed to generate an assembly file for a kernel. It provides methods for generating instructions and information that need to be added to the beginning and end of the file. Arguments need to be added to the `AssemblyKernel` before generating the initial setup instructions. The instructions for the body of the kernel need to be generated before generating the instructions at the end of the file. It is also necessary to set the Work Group Size, Work Item Count and the Dynamic Shared Memory Bytes. ### Scheduled Instructions All of the instructions scheduled by a `Context`, including those that came from the `AssemblyKernel`, are stored within a `ScheduleInstructions` object. This object can be converted into a string that contains the entire contents of an assembly file representing a single kernel. ## Kernel Executer ### Assembler In order to execute the kernel, the assembly instructions that are stored within a `Context`'s scheduled instructions must be converted into object code. This is performed by the `Assembler` class. The `Assembler` class uses AMD Comgr's API to assemble the instructions without having to write an assembly file to disk. ### Executable Kernel After object code has been created for a kernel, it can be executed on a GPU with provided arguments. An `ExecutableKernel` object will load object code generated by the `Assembler` class using HIP's `hipModuleLoadData` function. Once the kernel has been loaded, it can be executed on the GPU using HIP's `hipExtModuleLaunchKernel` function. Arguments are passed to the kernel using a `KernelArguments` object. ### Command Kernel In general, the `ExecutableKernel` object should not be called directly. Instead, a `CommandKernel` object should be created, which will contain an `ExecutableKernel`. A `CommandKernel` object can take a `Command` object and create an `ExecutableKernel` from it. It does this by generating a `KernelGraph` from a `Command`, then generating `ScheduledInstructions` from the `KernelGraph` and finally generating an `ExecutableKernel` from those instructions. Once a `CommandKernel` object has been created, it can be launched on a GPU with unique arguments by using the `launchKernel` method. The arguments are provided with a `RuntimeArguments` object. A `RuntimeArguments` object is a `std::span` that provides a view into a block of memory containing all of the arguments that will be passed to the kernel in the correct order. The actual memory is typically managed by a `KernelArguments` object, which provides methods for constructing and manipulating the argument buffer. ## GPUArchitectureInformation ### GPUArchitectureGenerator The main ideas behind the `GPUArchitectureGenerator` are the following: * minimize the runtime overhead of looking up architecture and instruction information * simplify the specification of architecture and instruction information To keep runtime overhead low we're storing this information in dictionaries (`std::map<>` and `std::unordered_map<>`). However, explicitly specifying these dictionaries in source files is unwieldy and difficult to maintain, so we'd like to be able to specify this info in a more programmatic way, but that would negatively impact runtime performance. To satisfy both of these requirements we have the `GPUArchitectureGenerator`, which allows us to populate architecture and instruction dictionaries in a programmatic method at compile time and serialize it. Those serialized dictionaries are read in and deserialized at runtime. The `GPUArchitectureGenerator::FillArchitectures()` function fills in `GPUArchitectureGenerator::GPUArchitectures` using various predicates, lists, and dictionaries defined in `GPUArchitectureGenerator/include/GPUArchitectureGenerator/GPUArchitectureGenerator_defs.hpp`. There is a custom cmake rule, `GPUArchitecture_def`, which uses the `GPUArchitectureGenerator` to generate `GPUArchitecture_def.yaml` at compile time. ### GPUArchitectureLibrary At runtime, the `GPUArchitectureLibrary` class can be used to query architecture and instruction information. For example, the following determines if `GFX90a` supports `DirectToLDS`: ```C++ GPUArchitectureLibrary::GetArch("GFX90a").HasCapability(GPUCapability::HasDirectToLds) ``` And the following gets the wait count for the `global_store_byte` instruction on `GFX90a`: ```C++ GPUArchitectureLibrary::GetArch("GFX90a").GetInstructionInfo("global_store_byte").getWaitCount() ``` ## Other Topics ### Components The Component system provides a flexible plugin architecture for swapping implementations based on runtime conditions. Components are organized into base classes (defining interfaces) and concrete implementations (providing specific functionality). A `ComponentBase` defines: - An `Argument` type used to select the appropriate implementation - A `Basename` string identifier Concrete `Component` implementations provide: - A `Match` function to determine if this component is appropriate for given arguments - A `Build` factory function to create instances - A `Name` identifier The `ComponentFactory` manages registration and selection of components. When `Component::Get(args)` is called: 1. The factory searches registered components for a matching implementation 2. For non-single-use components, instances are cached and reused 3. For single-use components, new instances are created each time Components are thread-safe, using reader-writer locks for concurrent access. This pattern is used throughout rocRoller for architecture-specific code generation (e.g., different assembler implementations). ### Serialization rocRoller provides a flexible serialization framework for converting objects to and from YAML format. The serialization system supports both LLVM's YAML library and yaml-cpp. Key serialization traits: - `ScalarTraits`: For basic types and enums, providing `output()` and `input()` methods - `MappingTraits`: For structured types with named fields - `SequenceTraits`: For sequential collections - `EnumTraits`: For enumeration types The main API functions are: - `toYAML(obj)`: Converts an object to a YAML string - `fromYAML(yaml)`: Parses a YAML string into an object - `writeYAML(stream, obj)`: Writes YAML to an output stream Many rocRoller types implement serialization traits, including `Command`, `KernelGraph`, `Expression`, and `Operation` types. This enables saving and loading kernel definitions, intermediate representations, and configuration data. Serialization is particularly useful for debugging, caching compiled kernels, and inspecting internal structures. ### Generator The `Generator` class template implements lazy sequences using C++20 coroutines. It provides an efficient way to produce sequences of values on-demand without storing the entire collection in memory. Key features: - Implements `std::ranges::input_range` for compatibility with range-based algorithms - Supports `co_yield` for single values ranges - Provides convenience methods: `map()`, `filter()`, `take()`, `only()`, `empty()` - Can create standard collections with `.to()` Generators are movable but not copyable. The prefix increment operator (`++iter`) is more efficient than postfix (`iter++`). Generators don't begin executing until the first value is requested. Example usage pattern: ```cpp Generator numbers() { for(int i = 0; i < 10; i++) co_yield i; } // Use with range-based for for(int n : numbers().filter([](int x) { return x % 2 == 0; })) std::cout << n << std::endl; // Convert to vector auto vec = numbers().take(5).to(); ``` Generators are used internally in rocRoller for producing sequences of instructions during code generation, allowing lazy evaluation and efficient memory usage.