ACPI - Providing Ramdisks to the OS with NVDIMM (NFIT)

[human00731582]

Greetings fellow enthusiasts.
Today I've returned to this community after a 7-year hiatus, with a renewed interest in low-level development, a Bachelor's and a Master's Degree, and plenty of field experience. A far cry from the slice of my life indicated by my post history.

Anyway, I'd like to see if any members of the community have experience with UEFI Driver development, specifically for booting their OS from a ramdisk.
I'm both stuck on a problem, and would like to prime the community for this really neat loader that I think will be valuable for making encrypted installation media without pricey hardware!

I have been working on a generic EFI binary which can decrypt and decompress a FAT filesystem from an ISO image into memory. This in-memory blob is a Virtual Disk.
I am targeting all operating systems which support NVDIMM ACPI details about the loaded ramdisk, so this is intended to be a generic way to encapsulate all compatible boot images (and possibly multiple sets thereof).

This is working, password entry and all, no problem: the loader creates the ramdisk, decrypts et al., attaches Block I/O protocols to the handle, and the firmware does the rest of the black magic (Disk I/O protocols, etc.).
The loader then 'jumps' to the boot entrypoint on the loaded ramdisk at /EFI/BOOT/BOOTX64.EFI when a certain canary file is detected on the target "drive" (i.e., ramdisk) during enumeration. This is done with LoadImage and StartImage, without ever exiting boot services.

The target operating system will boot, but immediately (on Linux) drops into an emergency shell. This is of course because it has no idea where to find the root filesystem it was launched from, which is still hanging around in RAM within the system memory map.
This is further complicated by the fact that the compressed FAT filesystem often will have other fragments the loaders need to reference-- think, for example, RedHat's "kickstart" system.

Everything must be done from UEFI to make this work, so things like editing /etc/fstab are too OS-specific.

Luckily, newer ACPI specs provide an opportunity to install ramdisks as pseudo-device objects. This can make enumerating OSes aware of the ramdisk from the UEFI space.
I have tried to add NFIT entries correctly pointing to the EfiRuntimeServicesData preserved by the second-stage GRUB loader (BOOTX64.EFI within the FAT disk image), but to no avail despite Linux's dmesg output indicating it "sees" an NFIT entry.

...

So my primary question is this: how do I make the host operating system properly aware of the NVDIMM devices (i.e., the ramdisk FAT filesystem) for mounting and executing?
The specification seems to indicate an NVDIMM "root device" is necessary under the _SB ACPI namespace, but I don't really know how to add that and using the full ACPI SDT driver from EDK2 is too much bloat.

Do I need to self-compile an ASL-to-AML file describing the Root Device, a la https://github.com/tianocore/edk2/blob/ ... amDisk.asl??

Any guidance here would be exceedingly valued!
Some sample code to follow...

Adding NFIT to ACPI (using GNU-EFI and inspired by the RamDiskDxe implementation in EDK2):

Code: Select all

  NfitLen = sizeof(EFI_ACPI_6_1_NVDIMM_FIRMWARE_INTERFACE_TABLE) +
            sizeof(EFI_ACPI_6_1_NFIT_SYSTEM_PHYSICAL_ADDRESS_RANGE_STRUCTURE);
  Nfit = AllocateZeroPool(NfitLen);
  if (NULL == Nfit)
    return EFI_OUT_OF_RESOURCES;

  SpaRange = (EFI_ACPI_6_1_NFIT_SYSTEM_PHYSICAL_ADDRESS_RANGE_STRUCTURE *)
              ((UINT8 *)Nfit + sizeof(EFI_ACPI_6_1_NVDIMM_FIRMWARE_INTERFACE_TABLE));

  /* OEM ID shouldn't matter or need to be recognized. */
  UINT8 PcdAcpiDefaultOemId[6] = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06 };

  NfitHeader                  = (EFI_ACPI_DESCRIPTION_HEADER *)Nfit;
  NfitHeader->Signature       = EFI_ACPI_6_1_NVDIMM_FIRMWARE_INTERFACE_TABLE_STRUCTURE_SIGNATURE;
  NfitHeader->Length          = NfitLen;
  NfitHeader->Revision        = EFI_ACPI_6_1_NVDIMM_FIRMWARE_INTERFACE_TABLE_REVISION;
  NfitHeader->OemRevision     = 0x20240501;   /* OEM Revision by date should be fine. */
  NfitHeader->CreatorId       = EFI_SIGNATURE_32('T', 'T', 'O', 'O');   /* Creator: 'TTOO' */
  NfitHeader->CreatorRevision = 0x1;   /* Should just be 1. */
  NfitHeader->OemTableId      = EFI_SIGNATURE_64('T', 'T', 'O', 'O', 'L', 'O', 'A', 'D');
  CopyMem(NfitHeader->OemId, &PcdAcpiDefaultOemId[0], sizeof(NfitHeader->OemId));

  /* Fill in the content of the SPA Range Structure. */
  SpaRange->Type                             = EFI_ACPI_6_1_NFIT_SYSTEM_PHYSICAL_ADDRESS_RANGE_STRUCTURE_TYPE;
  SpaRange->Length                           = sizeof (EFI_ACPI_6_1_NFIT_SYSTEM_PHYSICAL_ADDRESS_RANGE_STRUCTURE);
  SpaRange->SystemPhysicalAddressRangeBase   = PrivateData->StartingAddr;
  SpaRange->SystemPhysicalAddressRangeLength = PrivateData->Size;
  CopyMem(&SpaRange->AddressRangeTypeGUID, &PrivateData->TypeGuid, sizeof(EFI_GUID));

  Checksum             = CalculateCheckSum8((UINT8 *)Nfit, NfitHeader->Length);
  NfitHeader->Checksum = Checksum;

  /*
   * Publish the NFIT to the ACPI table.
   * Note, since the NFIT might be modified by other driver, therefore, we
   * do not track the returning TableKey from the InstallAcpiTable().
   */
  Status = uefi_call_wrapper(
    mAcpiTableProtocol->InstallAcpiTable,
    4,
    mAcpiTableProtocol,
    Nfit,
    NfitHeader->Length,
    &TableKey
  );
  if (EFI_ERROR(Status))
    return Status;

  FreePool(Nfit);
  return EFI_SUCCESS;

...

Also, please do not specify alternative suggestions (like U-Boot, LUKS, GRUB chainloading, etc.).
I'm really set on making this something I've home-brewed, and I'm extremely close to having it functional.

If anything needs clarification, please let me know. Thanks for understanding and for any pointers in the right direction!

--
References:

• 1 - RamDiskDxe EDK2 Driver: https://github.com/tianocore/edk2/blob/ ... ver.c#L116

• 2 - ACPI Device Tree (Linux): https://docs.kernel.org/firmware-guide/ ... space.html

• 3 - NVDIMM Root Information: https://uefi.org/htmlspecs/ACPI_Spec_6_ ... mm-devices

• 4 - UEFI Driver Development: https://tianocore-docs.github.io/edk2-U ... -draft.pdf

[Octocontrabass]

I have tried to add NFIT entries correctly pointing to the EfiRuntimeServicesData preserved by the second-stage GRUB loader (BOOTX64.EFI within the FAT disk image), but to no avail despite Linux's dmesg output indicating it "sees" an NFIT entry.

Are any persistent memory devices created under /dev? If they are, that means Linux was able to find your fake NVDIMM, but couldn't mount the filesystem for whatever reason. Linux might refuse to mount filesystems without appropriate namespace metadata.

The specification seems to indicate an NVDIMM "root device" is necessary under the _SB ACPI namespace, but I don't really know how to add that and using the full ACPI SDT driver from EDK2 is too much bloat.

Do I need to self-compile an ASL-to-AML file describing the Root Device, a la https://github.com/tianocore/edk2/blob/ ... amDisk.asl??

Yes, you can use a SSDT to add the NVDIMM root device to the ACPI namespace. You'll install your SSDT the same way you install your NFIT.

[human00731582]

Are any persistent memory devices created under /dev? If they are, that means Linux was able to find your fake NVDIMM, but couldn't mount the filesystem for whatever reason. Linux might refuse to mount filesystems without appropriate namespace metadata.

I do not see any /dev/pmem* devices. The dmesg does however indicate my allocation is still available and is of type reserved rather than persistent (type 12).

My guess is I'm still missing appropriate NVDIMM registration with ACPI.

Yes, you can use a SSDT to add the NVDIMM root device to the ACPI namespace. You'll install your SSDT the same way you install your NFIT.

I ended up using the iasl compiler tool from the acpica-tools library to create AML bytecode and as a hacky workaround I am inserting this directly in my code as a byte array (because iasl generates a full SSDT entry).

Code: Select all

DefinitionBlock (
  "RamDisk.aml",
  "SSDT",
  2,
  "TTOOLD",
  "RamDisk ",
  0x1000
  )
{
  Scope (\_SB)
  {
    Device (NVDR)
    {
      Name (_HID, "ACPI0012")
      Name (_STR, Unicode("NVDIMM Root Device"))

      Method (_STA, 0)
      {
        Return (0x0f)
      }
    }
  }
}

I think the only thing I'm missing at this point is getting that NVDIMM Root Device to dynamically reference the NVD (NVDIMM Ramdisk) by modifying it on-the-fly and recomputing the checksum.
I'm getting this information from: https://uefi.org/htmlspecs/ACPI_Spec_6_ ... imm-device

EDIT: Also need to find out what it means in that section by referencing the "Device Handle" for the _ADR definition. I should also mention I have no need for _DSM. I just need to provide the SPA information per my first post and be done.

Is that a valid approach? Feels very contrived and not easy to maintain (especially on other systems which might already have an NVDIMM Root Device registered).
But I'm willing to do anything to get it at least working so I can then iterate on it.

[Octocontrabass]

EDIT: Also need to find out what it means in that section by referencing the "Device Handle" for the _ADR definition.

It should match a device handle in your NFIT. Since you're not defining an actual hardware device, the exact value isn't too important (though you should set bit 31).

Is that a valid approach? Feels very contrived and not easy to maintain (especially on other systems which might already have an NVDIMM Root Device registered).

I'm pretty sure it will work, since it only needs to function for reads and not writes.

The only other option I can think of is virtualization. You can write a hypervisor that gives a virtual disk to the host OS, the same way firmware uses SMM to give a virtual PS/2 controller to the host OS.

[human00731582]

It should match a device handle in your NFIT. Since you're not defining an actual hardware device, the exact value isn't too important (though you should set bit 31).

I suppose I am confused here because Section 5 of the ACPI spec (v6.4) doesn't mention a "Device Handle" or such a field, and Section 9 which I linked doesn't actually define the term "Device Handle". I think I'm just missing a key item here-- which field of the structure maps to such a value so I could assign it arbitrarily?

The NFIT Device Handle is a 32-bit value. Bit [31] indicates the format of the NFIT Device Handle. ...
... If Bit [31] is set, then Bits [30:0] are defined as follows: ... Bits [30:0] platform unique value assigned by the platform firmware that is consistent across boots when the NVDIMM is in the same physical location but may change if the NVDIMM is in a different physical location.

Makes absolute sense for me to set bit 31 with this information from the spec.

I'm looking at the 32-bit fields here: https://uefi.org/htmlspecs/ACPI_Spec_6_ ... l#overview
Is it perhaps the OEM Revision I'm trying to match? I apologize it seems like I'm looking to be spoon-fed but I genuinely cannot understand which ID I'm looking to match here.

Really appreciate your help so far.

[Octocontrabass]

I apologize it seems like I'm looking to be spoon-fed but I genuinely cannot understand which ID I'm looking to match here.

You're looking for the "NFIT Device Handle" field, which is part of the NVDIMM Region Mapping Structure, which is one of the structures you can place in the NVDIMM Firmware Interface Table.

[human00731582]

You're looking for the "NFIT Device Handle" field, which is part of the NVDIMM Region Mapping Structure

Understood. I attempted to register a mapping and provide the NVDIMM Root Table knowledge of the arbitrary NVDIMM Device Handle, set as 0xFEFAF0F0. I used AML to point out an "NVDR" sub-tree device with an _ADR of that same value. Nothing.

I hooked up some bogus _DSM items that returned success codes. Nothing.
I then proceeded to try setting the SPA Range Index of the Virtual Disk to something not zero, to see if I could link the objects (SPA & Region Mapping) together. Nothing.

I tried many varying combinations all evening...

From the ACPI 6.4 specification, in Section 5.2.25.2:

System physical address ranges described as Virtual CD or Virtual Disk shall be described as AddressRangeReserved in E820, and EFI Reserved Memory Type in the UEFI GetMemoryMap.

Platform is allowed to implement this structure just to describe system physical address ranges that describe Virtual CD and Virtual Disk. For Virtual CD Region and Virtual Disk Region (both volatile and persistent), the following fields - Proximity Domain, SPA Range Structure Index, Flags, and Address Range Memory Mapping Attribute, are not relevant and shall be set to 0.

I've had my SPA as Index 0 this whole time, except when trying other mixtures to see what sticks, and nowhere does the specification imply I also need the NVDIMM Region Mapping Structure to accompany the SPA, except with references to the Device Handle (as you mentioned).
If I need a region mapping, does the SPA Range Structure Index field there then get set to 0? In that case, it voids other fields within the mapping itself. Not that I need it, but then how would I map and provide multiple VDISK items? The whole thing is either a complete mess or I'm not reading it right.. or both.

Linux does appear to at least be making an effort to discover NFIT items, but fails with something along the lines of:

nfit: probe of "ACPI0012:00" failed with error -12

(I'm not at my workstation at the moment, so the error might be worded slightly differently.) I've discovered this error maps to an rc of -ENOMEM, but that wouldn't make sense given my system resources at boot time.

From my unlearned eyes, I've noticed the libnvdimm code is very fond of that return code: https://github.com/torvalds/linux/blob/ ... re.c#L2689

--

...
Honestly... What a mess of crossover specifications mixed with a reliance upon decade-old implementations.
At this point, I'm just spewing whatever I can hoping to finally make progress.

I'm ready to find someone who's booted from an ISO by PXE and assume it has to use a similar trick.
There must be some way to safely jump into a GRUB instance on a ramdisk, while still allowing that boot ramdisk itself to hang around or be recognized.

Thanks.

[Octocontrabass]

From the ACPI 6.4 specification, in Section 5.2.25.2:

Huh, I must have overlooked the part where NFIT can also describe RAMdisks. For a RAMdisk, you should only need the SPA Range Structure in your NFIT and a stub ACPI0012 device in your SSDT.

How old is the Linux kernel you're testing with? It looks like support for this was added to Linux around version 4.8.

I've had my SPA as Index 0 this whole time, except when trying other mixtures to see what sticks, and nowhere does the specification imply I also need the NVDIMM Region Mapping Structure to accompany the SPA, except with references to the Device Handle (as you mentioned).

That's correct, I was reading the wrong part of the spec. If you're only providing a RAMdisk and not a complete (fake) NVDIMM, the SPA Range Structure Index should be 0, the GUID should be set to one of the RAMdisk values, and you shouldn't have a NVDIMM Region Mapping Structure.

Linux does appear to at least be making an effort to discover NFIT items, but fails with something along the lines of:

I'd like to see the exact messages Linux spits out when it parses your tables. That might help track down what's wrong with them.

[human00731582]

After a small break for sanity's sake, a bit of rubber-duck debugging, and some more trial-and-error, I'm happy to report that Linux installers can now map, find, and use the _pmem_ device by label when booting from the initial ramdisk.

For a RAMdisk, you should only need the SPA Range Structure in your NFIT and a stub ACPI0012 device in your SSDT.

This is absolutely correct: as you also wrote, the NVDIMM Root Device just needs to exist and the SPA structure indeed keeps an index of 0 to indicate it's a virtual media. Ultimately my problem ended up being that I wasn't using specifically the EfiReservedMemoryType (0x00) to reserve the ramdisk buffer.

How old is the Linux kernel you're testing with?

This also was a good tip. While it wasn't my problem, it's good to note that NVDIMM ACPI information and especially virtual ramdisk support is relatively new in the Linux kernel.

So now I have a fully encrypted Linux installation media that installs directly to any system drive from a loaded ramdisk. So cool.

Next on the list of test targets are other Linux distributions and Windows (pray for me lol), then after that it's a matter of letting other OS Developers know how they could use this loader if they wanted to encrypt their boot partitions without LUKS-type support.
It's also not only the NFIT that provides the physical mapping of the ramdisk, but also two EFI variables which describe its location and length (meaning other OSes without NFIT support can just pick up the Runtime Services variables and map them to a block device).

I will absolutely be sharing this project with the OSDev community in the near future, once I can iterate and mature it much more beyond an alpha 'it just werks' state.

Thanks for all your help!

--
To close the loop, I also want to share the prototypical code I'm using to complete this. Much of it of course comes from EDK2's RamDiskDxe driver.
WARNING: This hasn't been improved upon yet and will cause issues or incompatibility if an existing NFIT or NVDIMM Root Device discovery procedure is not performed first.

--
Sending through the SSDT containing the NVDIMM Root Device. This uses pretty much the same ASL code I posted earlier in this thread, compiled into a static byte array.

Code: Select all

static
EFI_STATUS
RamDiskPublishSsdt()
{
  EFI_STATUS Status;

  /* [Debug logging info...] */

  /*
   * Install an SSDT entry for the ACPI table using some compiled AML.
   *   The AML comes from the RamDisk.asl file in the local folder.
   *   This is compiled on Linux using the 'iasl' tool and the resultant
   *   bytecode is placed in a static C variable.
   * 
   * Compilation Command/Macro:
   *    iasl RamDisk.asl && echo && \
   *       T="$(xxd -p RamDisk.aml | tr -d '\n' | sed -r 's/(..)/0x\1, /g')" && echo && \
   *       echo "$T" | sed -r "s/((0x..,\s){8})/\1\r\n/g"
   * 
   * This code assumes that the "RamDisk " SSDT is NOT already installed at boot.
   */
  Status = uefi_call_wrapper(
    mAcpiTableProtocol->InstallAcpiTable,
    4,
    mAcpiTableProtocol,
    gNvdimmAml,
    gNvdimmAmlLength,
    &mAcpiNvdimmTableKey
  );
  return Status;
}

--
Publishing the NFIT table with the single virtual ramdisk entry.

Code: Select all

static
EFI_STATUS
RamDiskPublishNfit(
  IN RAM_DISK_PRIVATE_DATA *PrivateData)
{
  /* ... ... ... */

  /* All data structure definitions use the ACPI specification, except RAM_DISK_PRIVATE_DATA. */
  /* See: https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/05_ACPI_Software_Programming_Model/ACPI_Software_Programming_Model.html#nvdimm-firmware-interface-table-nfit */

  Status = RamDiskPublishSsdt();   /* Publish NVDIMM Root Device first; check Status. */
  /* ... */

  /* In this (very specific) scenario, the NFIT will consist of its own entry, with an SPA stub to represent the virtual ramdisk. */
  UINT32 NfitLen = sizeof(EFI_ACPI_6_4_NVDIMM_FIRMWARE_INTERFACE_TABLE) +
            sizeof(EFI_ACPI_6_4_NFIT_SYSTEM_PHYSICAL_ADDRESS_RANGE_STRUCTURE);
  VOID *Nfit = AllocateZeroPool(NfitLen);
  if (NULL == Nfit)
    return EFI_OUT_OF_RESOURCES;

  EFI_ACPI_6_4_NFIT_SYSTEM_PHYSICAL_ADDRESS_RANGE_STRUCTURE *SpaRange =
              (EFI_ACPI_6_4_NFIT_SYSTEM_PHYSICAL_ADDRESS_RANGE_STRUCTURE *)
              ((UINT8 *)Nfit + sizeof(EFI_ACPI_6_4_NVDIMM_FIRMWARE_INTERFACE_TABLE));

  UINT8 DefaultOemId[6] = /* An array of 6 ASCII characters. */ ;

  EFI_ACPI_DESCRIPTION_HEADER *NfitHeader;
  NfitHeader                  = (EFI_ACPI_DESCRIPTION_HEADER *)Nfit;
  NfitHeader->Signature       = EFI_ACPI_6_4_NVDIMM_FIRMWARE_INTERFACE_TABLE_STRUCTURE_SIGNATURE;
  NfitHeader->Length          = NfitLen;
  NfitHeader->Revision        = EFI_ACPI_6_4_NVDIMM_FIRMWARE_INTERFACE_TABLE_REVISION;
  NfitHeader->OemRevision     = MY_PERSONAL_OEM_REVISION_ID;
  NfitHeader->CreatorId       = EFI_SIGNATURE_32(/* ... */);   /* Creator: '...' */
  NfitHeader->CreatorRevision = MY_CREATOR_REVISION_NUMBER;
  NfitHeader->OemTableId      = EFI_SIGNATURE_64(/* ... */);
  CopyMem(NfitHeader->OemId, &DefaultOemId[0], sizeof(NfitHeader->OemId));

  /* Populate the SPA for the virtual disk. */
  SpaRange->Type                             = EFI_ACPI_6_4_NFIT_SYSTEM_PHYSICAL_ADDRESS_RANGE_STRUCTURE_TYPE;
  SpaRange->Length                           = sizeof(EFI_ACPI_6_4_NFIT_SYSTEM_PHYSICAL_ADDRESS_RANGE_STRUCTURE);
  SpaRange->SystemPhysicalAddressRangeBase   = PrivateData->StartingAddr;
  SpaRange->SystemPhysicalAddressRangeLength = PrivateData->Size;
  CopyMem(&SpaRange->AddressRangeTypeGUID, &PrivateData->TypeGuid, sizeof(EFI_GUID));

  /* Table checksum, now that it's completely filled out. */
  NfitHeader->Checksum = CalculateCheckSum8((UINT8 *)Nfit, NfitHeader->Length);

  /* Publish the NFIT to the ACPI table. */
  Status = uefi_call_wrapper(
    mAcpiTableProtocol->InstallAcpiTable,
    4,
    mAcpiTableProtocol,
    Nfit,
    NfitHeader->Length,
    &TableKey
  );
}

[zirblazer]

Most likely you already found this if you googled about the RAMDisk topic, but if not, then you could be interesed in this:

https://forum.level1techs.com/t/build-d ... fun/189871
https://forum.level1techs.com/t/devemba ... per/191607

His goal was for the RAMDisk to be visible as a drive in Windows, which he never accomplished because you needed to find a way to pass info from the UEFI Ramdisk driver to Windows.