Unable to create CPX VFs on MI300X

[marcofaltelli]

Hi,

I am trying to create CPX VFs on a Supermicro AS-8125GS-TNMR2 server equipped with 8 MI300X GPUs.

If I try to insert the gim module with the following command:
sudo modprobe gim vf_num=8
I get many errors in dmesg (see attachment). What is more interesting for me is the following error message:
[Wed Oct 15 16:04:04 2025] gim error libgv: [0:c6:0:0][amdgv_parse_config_opt:1020] VF number 8 to be enabled exceeds the vf limit 1.

Does MI300X support CPX partitions to this day? If yes, could you add to the documentation how to create a VM with a CPX VF?

At this point, if I try to set the CPX partition via amd-smi, I get no reply from the drivers:

ubuntu@amd-monster-gpu:~$ sudo amd-smi partition --accelerator

ubuntu@amd-monster-gpu:~$ sudo amd-smi partition --memory

ubuntu@amd-monster-gpu:~$

If I load the driver without the vf_num=8, it works OK, but I cannot create CPX VFs in that case.

Some info about the environment:
I am using gim module version 8.4.0.K , ROCM v7.0.1 , Ubuntu 22.04 and kernel 5.15.0.157-generic

Any help would be appreciated. This is an important test for a POC about AMD virtualization drivers.

dmesg.txt.rtf

[dumroo]

@marcofaltelli, please try 8.4.1.K and let us know if still facing the issue. Thanks

[marcofaltelli]

@dumroo Hi, unfortunately, I have the same issues as version 8.4.0K.
You can find the dmesg error logs in the attachment.
What is interesting is the following error line:
[Thu Oct 23 08:02:48 2025] gim error libgv: [0:85:0:0][amdgv_parse_config_opt:1020] VF number 8 to be enabled exceeds the vf limit 1.

dmesg-8.4.1.rtf

[marcofaltelli]

Also did a GPU FW update to 1.25.03.12.76 but it didn't help

[xxmtg]

try:

sudo lspci -d 1002: -vvv

and check the Mi300X device's SRIOV capability. The number of VF may be 1 there but we are trying to create 8VFs.

[marcofaltelli]

Hi @xxmtg ,
Right, here is the output.
case vf_num=1
Initial VFs: 1, Total VFs: 1, Number of VFs: 1, Function Dependency Link: 00 VF offset: 16, stride: 1, Device ID: 74b5
case vf_num=8
Initial VFs: 1, Total VFs: 1, Number of VFs: 0, Function Dependency Link: 00 VF offset: 16, stride: 1, Device ID: 74b5
So, does this mean that multiple VFs for MI300X are not supported yet?

[xxmtg]

Hey @marcofaltelli,

The issue we are facing is this:
Total VFs: 1

This is part of the SRIOV capability setting and it is embedded within the GPU BIOS (VBIOS). It is depending on your platform enablement. I am not able to identify the exact VBIOS that you are using from the current logs, and even if I can, it will be a bit challenging to flash it.

I believe @saadrahim can help you move forward!

[marcofaltelli]

Hi @saadrahim ,
Isn't the GPU vBIOS the same thing as the GPU firmware?
I recently did a GPU FW update to version 1.25.03.12.76, but I still get Total VFs: 1

[gcapodagAMD]

hi @marcofaltelli thanks for reporting this. our team is aware of the issue and we are on it. I worked on reproducing your issue on a local MI300X I have access to, I followed the instructions here to install the PF driver with a deb package:

[marcofaltelli]

Hi @gcapodagAMD ,
Thanks for reporting this.
Could you please share your firmware GPU version and the dmesg logs after you execute modprobe gim vf_num=8 ?

[gcapodagAMD]

[gcapodagAMD]

[gcapodagAMD]

is this what you were trying to achive?

[marcofaltelli]

is this what you were trying to achive?

Yup, I see you're having issues as well.
My problem is one step before yours (errors in the dmesg output at module insertion).
I get in particular this error:
VF number 8 to be enabled exceeds the vf limit 1.

[gcapodagAMD]

thanks for the chat @marcofaltelli let me know if doing what we talked about helps

[marcofaltelli]

Hi @gcapodagAMD , thank you for your time!
I think what you showed here is a conjunction of amdgpu and gim modules together. I was able to replicate what you did here using both drivers.
However, I believe that the correct way of doing this is only through the gim module, as the documentation suggests to blacklist the amdgpu driver.
In fact, I think you didn't create any VFs here (e.g., see lspci -d 1002: )