derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
80 lines
2.5 KiB
ReStructuredText
80 lines
2.5 KiB
ReStructuredText
.. meta::
|
|
:description: Set up a local NVMe drive for use with hipFile.
|
|
:keywords: hipFile, NVMe, install, ROCm, direct storage, GPU I/O
|
|
|
|
.. _hipfile-setup-local-nvme:
|
|
|
|
*******************************************************
|
|
Setting up a local NVMe drive for use with hipFile
|
|
*******************************************************
|
|
|
|
To configure a local NVMe drive for use with hipFile, it must be partitioned with a filesystem hipFile supports. ``amdgpu-dkms`` version 31.40 or newer is required to use an NVMe drive.
|
|
|
|
.. note::
|
|
|
|
hipFile does not support NVMe multipath devices with the fastpath backend.
|
|
|
|
To disable NVMe multipath on Ubuntu 24.04:
|
|
|
|
.. code-block:: bash
|
|
|
|
sudo bash -c 'echo "options nvme_core multipath=N" > /etc/modprobe.d/nvme_core.conf'
|
|
sudo update-initramfs -c -k all
|
|
sudo systemctl reboot
|
|
|
|
Use ``lsblk`` to see the list of drives:
|
|
|
|
.. code:: shell
|
|
|
|
lsblk -d -o NAME,SIZE,MODEL | grep nvme
|
|
|
|
Choose a drive without an OS installation. Partition it using ``sgdisk`` or any other partitioning tool.
|
|
|
|
.. warning::
|
|
|
|
Partitioning the drive will permanently erase all the data stored on it.
|
|
|
|
Create a single partition on the drive. For example, to partition the ``/dev/nvme1n1`` drive:
|
|
|
|
.. code:: shell
|
|
|
|
sudo sgdisk -n 1:0:0 /dev/nvme1n1
|
|
|
|
Format the partition with a file system supported by hipFile:
|
|
|
|
.. note::
|
|
The filesystem must sit directly on the drive's partition, as shown. For
|
|
block layers that break hipFile's direct path, see :doc:`../troubleshooting/limitations`.
|
|
|
|
.. code:: shell
|
|
|
|
sudo mkfs.ext4 /dev/nvme1n1p1
|
|
|
|
Create a mount point and mount the partition with ``data=ordered``:
|
|
|
|
.. code:: shell
|
|
|
|
sudo mkdir /mnt/ext4
|
|
sudo mount /dev/nvme1n1p1 /mnt/ext4 -o data=ordered
|
|
|
|
The mount point can also be added to ``/etc/fstab`` to automatically mount the partition during the boot process.
|
|
|
|
Verify that hipFile has access to the mounted path by copying a file using
|
|
`aiscp.cpp <https://github.com/ROCm/rocm-systems/blob/develop/projects/hipfile/examples/aiscp/aiscp.cpp>`_:
|
|
|
|
.. code:: shell
|
|
|
|
sudo mkdir /mnt/ext4/"${USER}"
|
|
sudo chown "${USER}":"${USER}" /mnt/ext4/"${USER}"
|
|
# Create a file
|
|
dd if=/dev/urandom of=/mnt/ext4/"${USER}"/source bs=4K count=16
|
|
# Copy the file
|
|
HIPFILE_ALLOW_COMPAT_MODE=false ./aiscp /mnt/ext4/"${USER}"/source /mnt/ext4/"${USER}"/dest
|
|
md5sum /mnt/ext4/"${USER}"/source /mnt/ext4/"${USER}"/dest
|
|
|
|
|
|
Use the Linux ``umount`` command to unmount the partition when it's no longer needed:
|
|
|
|
.. code:: shell
|
|
|
|
sudo umount /mnt/ext4 |