How to boot QEMU from a locally mounted drive on Windows?

[VioletGiraffe]

I'm trying to learn bare metal development with UEFI, I'm developing on Windows in Visual Studio, and I want to test my software in QEMU. The usual QEMU workflow that's described everywhere includes creating a FAT32 drive image file and specifying that file. But I don't know how to do this on Windows and I don't want to have to mess with it, I just want to use a local RAM disk, properly formatted and containing my EFI/BOOT/bootx64.efi file.

How to tell QEMU to use a local drive (e. g. Z:) for booting? What's the right command line incantation?

[iansjack]

Booting from a physical drive is always open to problems. Get something wrong and you may erase your system drive.

Why not use qemu-img (which comes with qemu) to create and manage a disk image? It's very straightforward.

[VioletGiraffe]

Corruption of the partition is not a problem, it's a ramdisk. Thanks for pointing me to qemu-img, but is there really no way to tell qemu to just boot from the damn partition I prepared for it?

[Octocontrabass]

You can give QEMU the UNC path to your disk.

But do you need to create a RAM disk at all? QEMU has built-in support for creating a FAT-formatted disk out of a directory. This is what I use for UEFI development.

[iansjack]

I’m not thinking of corrupting the partition you want to use. Get something wrong and you could end up using the wrong partition.

You can certainly do what you want to - it’s described in the documentation ( https://www.qemu.org/docs/master/system/images.html ) but I would still recommend that you don’t. If you really don’t want to use a disk image use a virtual FAT image from a directory.

[VioletGiraffe]

But do you need to create a RAM disk at all? QEMU has built-in support for creating a FAT-formatted disk out of a directory. This is what I use for UEFI development.

Thank you, that does look like what I need! Could you please show your qemu invocation command line that works? I'm getting a "Block node is read-only" error (of course the boot partition should be read-only), and if I add "rw:" quemu does boot, but fails to see my bootloader, so I'm doing something wrong.

Does "fat" mean FAT32?

[Octocontrabass]

Could you please show your qemu invocation command line that works?

Code: Select all

.\qemu-system-x86_64.exe -drive file=.\edk2-x86_64-code.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=fat:rw:\Users\me\Documents\uefi-test,format=raw

The EFI directory is inside the uefi-test directory.

Does "fat" mean FAT32?

Maybe, but it doesn't matter for testing in QEMU. OVMF works just as well with FAT12 and FAT16.

[VioletGiraffe]

Any idea why it wouldn't work? Tried different options for the path - UNC/regular, absolute/relative, with/without the trailing slash, with/without quotes - it just boots into UEFI shell. Tried on a USB stick and a physical PC - boots fine.

Code: Select all

"C:\Program Files\qemu\qemu-system-x86_64.exe" -cpu qemu64 -net none -drive if=pflash,format=raw,unit=0,file=E:\Development\Tools\UEFI\OVMF\OVMF_CODE.fd,readonly=on -drive if=pflash,format=raw,unit=1,file=E:\Development\Tools\UEFI\OVMF\OVMF_VARS.fd -drive file=fat:rw:E:\Development\Projects\Personal\UEFI-Bootloader\BOOT\,format=raw

The "BOOT" folder contains EFI\BOOT\bootx64.efi inside

[Octocontrabass]

Code: Select all

OVMF_VARS.fd

Your NVRAM might have the boot order set incorrectly. Press ESC during boot to open the OVMF configuration, then use the Boot Maintenance Manager menu to fix the boot order.

[VioletGiraffe]

Your NVRAM might have the boot order set incorrectly. Press ESC during boot to open the OVMF configuration, then use the Boot Maintenance Manager menu to fix the boot order.

Spot on, thank you! Just removing the OVMF_VARS.fd drive from the command line did the trick.

[ChrisChiesa]

I don't see anything here that addresses the real issue: that on Windows versions later than XP, ordinary software (such as the Windows port of the Unix dd utility) can't access the individual raw blocks of a hard drive as it could before, apparently for security reasons. It used to be that specifying the UNC path to the device (if you could figure it out; I can't) was sufficient to do it, but not anymore, as far as I can tell.

There must be ways to do it, of course, otherwise the OS itself couldn't function, but I've been waiting ten years now to find out how one does it in later versions of Windows. I vaguely recall having had a single instant of success with dd on a Windows 10 machine, but have never been able to reproduce whatever the magic incantation was that I hit by accident that once. Maybe I accidentally specified a device that wasn't a hard drive: CD-ROM and DVD-ROM discs seem to be exempt from the general ban on block-level access.

Relevance here is that QEMU seems to be happy to accept either N: or \\.\N: as a -hda device specification on the launch command, but clearly just hangs when it tries to actually access the drive, when N: is physically a SATA hard drive connected via a SATA-to-USB adapter. (It's even worse if the drive is a thumbdrive named O: -- QEMU rejects the attempt to specify either O: or \\.\O: out of hand, seemingly without even trying to access the device (though I can't really be sure, as there are no "drive activity lights" anymore, in this century).

So the question that's actually being asked, and has not yet been adequately answered, here is: How do we get QEMU to bypass the Windows-later-than-XP restriction against accessing raw hard-drive blocks, and use a physical hard drive as a drive for the emulated computer?

Discuss.

[Octocontrabass]

How do we get QEMU to bypass the Windows-later-than-XP restriction against accessing raw hard-drive blocks, and use a physical hard drive as a drive for the emulated computer?

Generally you don't, because it's a terrible idea to do that on Windows. If you must, the syntax to point QEMU to a whole drive is documented, but it might also require administrator privileges and making sure any volumes on that drive are unmounted.

[ChrisChiesa]

What's so terrible about it? There are tons of use cases, at least in my world. Right now, for instance, I need to install 64-bit Windows 7 onto a HDD connected to my 64-bit Windows 10 / Linux Mint 21.1 dual-boot desktop PC via a USB-to-SATA adapter. (Please don't make me explain my reasons; just take it as stipulated, will ya?) I don't want to give the Windows 7 installation procedure access to my real (host) hardware, lest I accidentally pick the wrong HDD as the installation target and wipe out my existing Windows or Linux volumes, so I want to run the installation procedure in QEMU, to limit it to being able to see only the source CD-ROM and the destination external USB target HDD, so no mistakes can occur. Voila -- a use case!

I have qemu-system-x86_64 on both Windows and Linux, but they're probably different versions, because on Windows I can easily boot either the installation CD-ROM or a Windows 7 HDD image I made a few years ago, in that tool -- but on Linux, any attempt to boot either of those media bluescreens the host machine before any GUI ever comes up. I seem to recall reading, years ago, that there was a problem booting the Windows 7 installation CD-ROM in some version of QEMU, but that it had been fixed; so maybe I just have TOO OLD a version of QEMU on the Linux side -- but if so, I don't know how to update it, or even uninstall it to make room for a fresh installation of a newer version... (I've also never seen KVM or the other associated tools, front ends, etc., that I keep reading about; until just a few minutes ago I thought they just plain didn't exist for Windows, but a Google search just now brought up, for the first time ever, references to a Windows version. In any case, none of those things have ever been in evidence anyplace I've ever found QEMU for download, so I've never had them, don't use them, and don't know anything about them. I assume this is a side issue, unless it is somehow crucial -- you tell me!)

Anyway. I can boot the Windows 7 installation CD-ROM just fine on the Windows version of qemu-system-x86_64 -- but I on Windows I can't get QEMU to see the external, USB-connected, HDD, because I can't find any name for a drive, or a partition, that QEMU will accept (the only thing it will accept is a drive letter; it gives no error but also does not access a device). I've tried all sorts of things: the lengthy UNC names reported by the Windows port of dd given the --list option, the Windows drive letter (e.g. "H:"), and the advice at the link you shared -- "\\.\PhysicalDriveN" -- but none of them work. (The error in the case of your link, is "Could not open '\\PhysicalDrive2': The specified path is invalid." Note that it's not "could not access" or "no such device" -- the name itself is being rejected as invalid. I don't know if that's a QEMU thing, a Windows 10 thing, or what.

(In Windows, as everyone knows, just figuring out which drive letters are attached to which partitions on which physical devices is a challenge -- but it's even worse than I thought. dd --list prints a list of UNC device names, noting that some of them are "mounted on <drive letter>" -- and DISKPART will give you a list of "volumes," some of which have drive letters associated with them (which I assume is the Windows way of mounting a drive; removing the letter, dismounts the drive -- right?!?). But a given partition doesn't get the same volume name/number in the two tools: the partition that dd identifies as "\\?\Device\HarddiskVolume10" is known in DISKPART as "Volume 8," and so on.

The only tool that conclusively correlates drive numbers, partitions, and drive letters, is the Windows GUI tool known in Windows 10 as "Create and format hard disk partitions," and in previous versions of Windows as "Disk Management" -- but then that one doesn't tell the drives' or partitions' UNC names, as far as I can tell! It's maddening -- every single tool has something just a little incomplete about it, and the whole business gives me a bad taste in my mouth regarding Windows as a whole.

[Octocontrabass]

What's so terrible about it?

Mostly that Windows has its own ideas about what you can do with your drives, and when that causes problems, you're likely to lose data.

I don't want to give the Windows 7 installation procedure access to my real (host) hardware, lest I accidentally pick the wrong HDD as the installation target and wipe out my existing Windows or Linux volumes, so I want to run the installation procedure in QEMU, to limit it to being able to see only the source CD-ROM and the destination external USB target HDD, so no mistakes can occur. Voila -- a use case!

So you're only ever going to boot that Windows 7 installation in QEMU, right? If not, um, good luck.

but on Linux, any attempt to boot either of those media bluescreens the host machine before any GUI ever comes up.

Host or guest? Anyway, aside from version differences, QEMU on Linux often enables hardware acceleration by default. That could change the emulated hardware enough to make it incompatible with Windows 7.

maybe I just have TOO OLD a version of QEMU on the Linux side -- but if so, I don't know how to update it, or even uninstall it to make room for a fresh installation of a newer version...

Usually updating it through your distro's package manager is good enough. If you need something newer, you'll probably have to compile it yourself.

The error in the case of your link, is "Could not open '\\PhysicalDrive2': The specified path is invalid." Note that it's not "could not access" or "no such device" -- the name itself is being rejected as invalid. I don't know if that's a QEMU thing, a Windows 10 thing, or what.

Looks like a typo. You probably wanted "\\.\PhysicalDrive2" instead of "\\PhysicalDrive2".

[ChrisChiesa]

I've finally gotten around to getting back here to update you on the situation. It got real real complicated.

I apologize, but I don't know how to include your remarks here, the way you did mine.

1) "Windows has its own ideas about what you can do with your drives" -- that is a succinct summary of what, alas, I found out the hard way after much more blood-sweat-and-tears.

The first thing that happened after I posted her, was that somebody I trust pooh-poohed QEMU and recommended VirtualBox. After some blood-sweat-and-tears I did indeed get VB to access that USB-connected-external-HDD (actually a SSD, which introduced additional complications), and to boot the Windows 7 installation CD-ROM -- but when I selected the former for installation of the latter, the operation immediately failed with a vague error that took weeks -- and AI assistance I had sworn never to accept -- to figure out that the problem was that VB was being told the HDD was unreadable or unwritable or both. After more hours/days/weeks it was determined that this was not actually true but was WIndows 10 nannying, intercepting VB's requests for access to the drive's boot blocks in particular. I ultimately had to install Windows 7 to a virtual disk image (.VDI file), then convert that to a .VHD (virtual hard drive) file, then reboot into Linux and use good old dd to clone the .VHD to the physical drive.

I should then have just unplugged the drive and tried taking it back to the physical laptop for which it was intended: I would immediately have discovered that that laptop was gone from where I thought I'd left it, and there was no computer sitting waiting for this drive, though there surely was when I started.

As is, though, I thought, and decades of experience with earlier versions of windows had taught me to expect, it to be easy to just test-boot the drive in an emulator, so that if there were something wrong with it it would still be in the test harness and easy to correct. Hoo boy. I didn't reckon with how much nannying and interference MIcrosoft had added in Windows 10 as compared to, say, Windows 7 or XP. After a great many failed attempts, another swap to Linux and back, and another LONG week of conversations with AI, "we" ultimately gave up the whole business as a lost cause. And that was when I went to "just put the drive back in the physical computer it was headed for in the end," and found it missing. Oh brother.

I'm glossing over a ton of stuff, of course. I learned a lot, became even more disillusioned, discouraged, and disgusted with MIcrosoft and Windows than I already was, and renewed my vow to switch fully to Linux Real Soon Now -- certainly in preference to ever having to deal with Windows 11, another thing I accidentally found myself doing for a friend, in betwixt and between all this. Windows 11 is even worse than Windows 10, if you can believe it. If somebody gives you a WIndows 11 computer for free, I tell you in all seriousness your best move is to wipe the hard drive and put some other OS on there. I'm talking brand-new laptop from Walmart, right out of the box. NOT WORTH IT. Microsoft and Windows have finally fully "jumped the shark," as we used to say.


2) "You're only ever going to boot that Windows 7 installation in QEMU, right?" As noted above, no -- I was trying to build a boot drive for a laptop whose CD-ROM drive had failed and on which, therefore, it was impossible to install Windows 7 "on board." I might be able to make and boot a Windows 7 Installation CD-ROM ISO and boot that from either a dedicated thumbdrive, or via Ventoy. It all depends on whether the target laptop -- old enough to want Windows 7! -- is new enough to boot from USB. There are unwanted Catch-22s all over the damn place in this hobby nowadays. UGH.


3) "Usually updating it through your distro's package manager is good enough..." I suppose it might be, if I originally installed it "through my distro's package manager" in the first place -- but I'm not sure I did! See, I myself pre-date package managers, so I don't think in terms of package managers, definitely don't go to them as my first choice when I want a program, and therefore may very well not have used one. I may simply have gone out onto the Internet in a bare-bones Firefox Google search, and downloaded a bunch of executable files. I certainly don't have all the QEMU tools I've been reading about in the (highly Linux-oriented) documentation for the last several years; I have yet to see 'kvm' for example.


4) "You probably wanted '\\.\PhysicalDrive2' instead of '\\PhysicalDrive2'." That's possible. It's also possible that I got it right, on the actual QEMU launch commandline, and it still didn't work, and only typoed it when writing about it here. TBD.