derek.holloway
7502bf64a4
CodeQL / Analyze (actions) (push) Canceled after 0s
CodeQL / Analyze (python) (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Multi-Arch CI / setup (push) Canceled after 0s
Publish build_manylinux_x86_64 images / publish_build_manylinux_x86_64 (push) Canceled after 0s
Setup (Multi-Arch) / setup (push) Canceled after 0s
Multi-Arch CI / Linux::${{ fromJSON(needs.setup.outputs.linux_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubi10 image / publish_no_rocm_image_ubi10 (push) Canceled after 0s
Multi-Arch CI / Windows::${{ fromJSON(needs.setup.outputs.windows_build_config || '{}').build_variant_label || 'skip' }} (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Multi-Arch CI / Manifest Diff (push) Canceled after 0s
TheRock Manifest Diff Report / Generate Manifest Diff Report (push) Canceled after 0s
Multi-Arch CI / CI Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_ocl_rt images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04_rocgdb images / publish_no_rocm_image_ubuntu24_04_rocgdb (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_ocl_rt (push) Canceled after 0s
Publish TheRock Docker image / build-and-push-image (push) Canceled after 0s
Unit Tests / Unit Tests :: ubuntu-24.04 (push) Canceled after 0s
Unit Tests / Unit Tests :: windows-2022 (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_media (push) Canceled after 0s
Unit Tests / Unit Tests Summary (push) Canceled after 0s
Publish no_rocm_image_ubuntu24_04 images / publish_no_rocm_image_ubuntu24_04_openmpi (push) Canceled after 0s
114 lines
3.0 KiB
ReStructuredText
114 lines
3.0 KiB
ReStructuredText
.. meta::
|
|
:description: Set up an NFSoRDMA share for use with hipFile.
|
|
:keywords: hipFile, NFSoRDMA, NFS, RDMA, install, ROCm, direct storage, GPU I/O
|
|
|
|
***************************************************
|
|
Setting up an NFSoRDMA share for use with hipFile
|
|
***************************************************
|
|
|
|
The following prerequisites are required to use an RDMA-capable NFS server over RDMA or RoCE with hipFile:
|
|
|
|
* ``amdgpu-dkms`` version 31.40 or later.
|
|
* An RDMA or RoCE fabric connecting the NFS server and client.
|
|
* ``nfs-kernel-server`` installed on the server, with the ``svcrdma`` kernel
|
|
module available.
|
|
- ``nfs-common`` installed on the client, with the ``xprtrdma`` kernel module
|
|
available.
|
|
|
|
Configure the NFS server
|
|
==========================
|
|
|
|
Configure a static IP address on the RDMA or RoCE interface.
|
|
|
|
.. note::
|
|
|
|
The example commands use ``netplan``. Use the appropriate distribution network configuration tools.
|
|
|
|
Create ``/etc/netplan/config.yaml``, replacing ``<rdma_interface>`` and ``<server_ip>/<prefix>`` with values that match the fabric, then run ``netplan apply``:
|
|
|
|
.. code-block:: yaml
|
|
|
|
network:
|
|
version: 2
|
|
renderer: networkd
|
|
ethernets:
|
|
<rdma_interface>:
|
|
dhcp4: false
|
|
dhcp6: false
|
|
accept-ra: false
|
|
addresses:
|
|
- <server_ip>/<prefix>
|
|
|
|
.. code:: shell
|
|
|
|
sudo netplan apply
|
|
|
|
Load the RDMA-capable NFS server module:
|
|
|
|
.. code:: shell
|
|
|
|
sudo modprobe svcrdma
|
|
|
|
Add the directories to export to ``/etc/exports``, where ``EXPORT_PATH`` is the path to each directory to export:
|
|
|
|
.. code-block:: none
|
|
|
|
EXPORT_PATH *(rw,async,insecure,no_root_squash)
|
|
|
|
Reload the NFS server to pick up the new exported files:
|
|
|
|
.. code:: shell
|
|
|
|
sudo systemctl reload nfs-server
|
|
|
|
Add an RDMA port for NFS to listen on:
|
|
|
|
.. code:: shell
|
|
|
|
echo rdma 20049 | sudo tee /proc/fs/nfsd/portlist
|
|
|
|
Confirm that the RDMA port is active:
|
|
|
|
.. code-block:: none
|
|
|
|
$ cat /proc/fs/nfsd/portlist
|
|
rdma 20049
|
|
tcp 2049
|
|
|
|
Configure the client
|
|
======================
|
|
|
|
Load the RDMA-capable NFS client module:
|
|
|
|
.. code:: shell
|
|
|
|
sudo modprobe xprtrdma
|
|
|
|
Create a mount point and mount the share using RDMA:
|
|
|
|
.. code:: shell
|
|
|
|
sudo mkdir /mnt/nfs
|
|
sudo mount -o rdma,port=20049 <server_ip>:<export_path> /mnt/nfs
|
|
|
|
Set ``HIPFILE_UNSUPPORTED_FILE_SYSTEMS=true`` before running hipFile I/O.
|
|
|
|
Verify that hipFile has access to the mounted share by copying a file using
|
|
`aiscp.cpp <https://github.com/ROCm/rocm-systems/blob/develop/projects/hipfile/examples/aiscp/aiscp.cpp>`_:
|
|
|
|
.. code:: shell
|
|
|
|
sudo mkdir /mnt/nfs/"${USER}"
|
|
sudo chown "${USER}":"${USER}" /mnt/nfs/"${USER}"
|
|
# Create a random input file
|
|
dd if=/dev/urandom of=/mnt/nfs/"${USER}"/source bs=4K count=16
|
|
# Copy file
|
|
HIPFILE_ALLOW_COMPAT_MODE=false HIPFILE_UNSUPPORTED_FILE_SYSTEMS=true ./aiscp /mnt/nfs/"${USER}"/source /mnt/nfs/"${USER}"/dest
|
|
md5sum /mnt/nfs/"${USER}"/source /mnt/nfs/"${USER}"/dest
|
|
|
|
Use the Linux ``umount`` command to unmount the share when it's no longer needed:
|
|
|
|
.. code:: shell
|
|
|
|
sudo umount /mnt/nfs
|