I purchased the book but there are a coumple of things that aren't clear enough. What I really want to know is what steps do I have to follow from downloading the zip file at http://www.fysnet.net/book_series/usb_vol8.zip to have the code working at my computer.
First, among other things, it seems like the author is assuming the reader has a computer with a floppy disk, but most computers nowadays don't have it because it is very obsolete, and I would like to ask how to install the FreeDOS operating system from the img file BenLunt provides in that zip file I linked above.
I tried installing in VirtualBox (Although he said somewhere that you are supposed to test the source code in a real commputer and not in a virtual one) but I still don't know what I am supposed to do with the extra freedos_hd.img file that is included in the root directory of the zip file next to freedos.img and I'd like to install in VirtualBox to at least acknowledge how I am supposed to install the operating system in a real computer before proceeding.
In the book he says this: "The examples do use a DOS Protected Mode Interface (DPMI) driver for DOS."
Is the DPMI driver included with the img files? Do you have to install it separately? How are you supposed to do so?
These are the steps that I am following to install FreeDOS in VirtualBox (please bear in mind that this is only to make sure how I am supposed to install FreeDOS in a real computer. As I said I know you said the reader is supposed to test the code in a real machine):
1. I create a new virtual machine called FreeDOS with a virtual hard disk
https://ibb.co/FbDpYNx
https://ibb.co/8rMS6bN
2. I enter the freedos.img floppy disk
https://ibb.co/FYqRqpx
https://ibb.co/ngJS4C5
https://ibb.co/4gZ9tJW
3. Finally, I start the machine without selecting any .iso file, since there is no .iso in the zip file, and then I select the default option
https://ibb.co/wKQQnYz
https://ibb.co/dpGfrHX
https://ibb.co/sPNkvYD
Finally, how you compile the source code? is it enough with "gcc gd_xhci.c -o gd_xhci.exe"?
Please note that I am posting here instead of sending a private message to the author because I think other people struggling could find this post useful, and also because this way I can get help from other people and not only the author. Also sorry if this question seems too stupid, but this is the best place I can think of to ask this kind of question.
How to test the code of BenLunt's USB book
Re: How to test the code of BenLunt's USB book
For your reference, and for others as well, the source code is now freely available at https://github.com/fysnet/FYSOS/tree/master/main.slammar wrote:I purchased the book but there are a coumple of things that aren't clear enough. What I really want to know is what steps do I have to follow from downloading the zip file at http://www.fysnet.net/book_series/usb_vol8.zip to have the code working at my computer.
I use the floppy disk as an example of one of the items you might use to boot a computer. You can use a USB thumb drive, a floppy disk, or the actual hard drive within the machine. You just need some source of media to boot from. Yes, the floppy disk is now obsolete, however, it gives a very convenient way to transfer files to and from computers. Also, it is highly recommended that you find an older computer to do your testing with, one that you can find for a couple of dollars at a yard sale or something. This way you don't damage a new one when using it as a test machine. These older computers will have floppy drives anyway. Another thing about floppy drives, is Volume 8 is about USB. None of the example utilities will work if you boot from a USB thumb drive. The System Management firmware won't let go of the USB so that you can run the utilities as intended. So you will need to put the USB examples on another media type and boot from it instead.slammar wrote:First, among other things, it seems like the author is assuming the reader has a computer with a floppy disk, but most computers nowadays don't have it because it is very obsolete, and I would like to ask how to install the FreeDOS operating system from the img file BenLunt provides in that zip file I linked above.
The img files are all ready to boot image files. You can write them to some sort of media. The 1.44meg image can be written to a floppy or a USB Thumb drive. The other image file can be written to a Thumb drive or a hard drive. If you want to use VirtualBox, you will need to add a VHD footer.slammar wrote:I tried installing in VirtualBox (Although he said somewhere that you are supposed to test the source code in a real commputer and not in a virtual one) but I still don't know what I am supposed to do with the extra freedos_hd.img file that is included in the root directory of the zip file next to freedos.img and I'd like to install in VirtualBox to at least acknowledge how I am supposed to install the operating system in a real computer before proceeding.
Please be advised that I assume that you have some knowledge of computers and how to use them. Writing a file to a floppy disk or a Thumb Drive should be an easy task with a little research.
This is also assumed that you know what a DPMI is. Since my utilities need to access memory regions about the 1 meg mark, you need some sort of extension to DOS to allow this. This is where the DPMI is. My books tell you where to place the DPMI and this DPMI is already in the correct directory of the included image files.slammar wrote:In the book he says this: "The examples do use a DOS Protected Mode Interface (DPMI) driver for DOS."
Is the DPMI driver included with the img files? Do you have to install it separately? How are you supposed to do so?
Each file shows within the header of the file how to compile it. Plus, did you try it?slammar wrote:Finally, how you compile the source code? is it enough with "gcc gd_xhci.c -o gd_xhci.exe"?
Not at all. You did the right thing. I think the same thing.slammar wrote:Please note that I am posting here instead of sending a private message to the author because I think other people struggling could find this post useful, and also because this way I can get help from other people and not only the author. Also sorry if this question seems too stupid, but this is the best place I can think of to ask this kind of question.
Please feel free to continue to post questions. I (we) will do our best to help out.
Ben
Re: How to test the code of BenLunt's USB book
This is what I've done to get the VDI file:Now when I create the FreeDOS operating system in virtualbox and it asks me to select a virtual hard disk file, I select kernel.vdi
Also in the Storage section in settings I don't select any floppy disk this time, I leave it empty because with the hard disk apparently is enough.
If you want to check XHCI functionality, do as follows:
1. Make sure in VirtualBox that the xhci controller is enabled in Settings-->USB
2. In Storage, make sure that you have the freedos.img file selected in the Floppy controller and the vdi file in the IDE controller
3. Run the virtual machine
4. Select "FreeDOS Safe Mode (don't load any drivers)"
5. Execute XHCI.EXE > C:\XHCI.TXT
6. Turn off the virtual machine
7. Go to this website: https://vmxray.com/, select your vdi file and download the XHCI.TXT file.
This is the XHCI.TXT file that you get in VirtualBox: By the way, How do you load the protected mode drivers? Is it enough to execute the CWSDPMI.EXE file?
Code: Select all
dd if=/dev/zero of=kernel.raw bs=1024 count=2048
dd if=freedos_hd.img of=kernel.raw conv=notrunc
VBoxManage.exe convertfromraw kernel.raw kernel.vdi --format VDIAlso in the Storage section in settings I don't select any floppy disk this time, I leave it empty because with the hard disk apparently is enough.
If you want to check XHCI functionality, do as follows:
1. Make sure in VirtualBox that the xhci controller is enabled in Settings-->USB
2. In Storage, make sure that you have the freedos.img file selected in the Floppy controller and the vdi file in the IDE controller
3. Run the virtual machine
4. Select "FreeDOS Safe Mode (don't load any drivers)"
5. Execute XHCI.EXE > C:\XHCI.TXT
6. Turn off the virtual machine
7. Go to this website: https://vmxray.com/, select your vdi file and download the XHCI.TXT file.
This is the XHCI.TXT file that you get in VirtualBox: By the way, How do you load the protected mode drivers? Is it enough to execute the CWSDPMI.EXE file?
Re: How to test the code of BenLunt's USB book
I you don't mind, I thought I would elaborate on my reply a little bit.
First, I suggest to use both an emulator and real hardware. I even include the instructions to use Bochs with a compiled version of Bochs within the ISO you pointed to.
Using an emulator is much more convenient that real hardware. You can use the same machine--in my case, my flagship--to code, debug, build, have a few .PDFs open, as well as load the emulator and run the current build. This makes it very convenient where you don't have to copy anything to a physical media device, and/or transfer it to another machine, and/or reboot any machine. This is the preferred way of doing this processes.
However, you have found out that on the emulator, your code seems to work just fine, yet when you move to the hardware, it all of a sudden, doesn't. This, unfortunately, will happen most every time when it comes to hardware programming. The emulator, no matter how well it is written, can't simulate the hardware exactly enough to allow your code to run on both without extensive debugging.
Now, since you do need to run it on real hardware, you have to have access to that hardware. For example, you can't reset the xHCI controller and send/receive packets without the current operating system--Windows 10 or a *nix varient--getting in the way. On top of that, if you did reset the controller, how would the current OS then continue to use that controller? Therefore, you must run the test code on bare hardware using no existing (modern) operating systems, or at least an OS that will let you have full control of the hardware.
DOS-which FreeDOS is a good variant--will allow you to have full control of the hardware. Now, how do you run DOS on your machine? You boot a floppy or a USB thumb drive with an already formatted partition, with DOS loaded on it. Hence, the freedos.img on all my ISOs that come with my books. The freedos.img file is an already built image that will boot to FreeDOS and allow you to run all of the compiled test suites I include with each book.
Now, why a floppy? Two reasons. First, as I suggested earlier, I highly recommend that you find a machine that someone is throwing out, a machine at a yard sell, a machine off ebay, any cheap somewhat modern machine. You can find many 32-bit machines with Legacy BIOS at yard sells, garage sells, on the curb for the trash guy, etc. For a 64-bit machine, you will need to do a little more looking, but still can find a decent older 64-bit machine for less than 20 USD. I picked up one the other day just because someone thought it wasn't fast enough. A complete wipe of the hard drive and I had a new 64-bit, 3.0g machine for testing.
Why find an older machine? If you pick one up for 20 USD and use it for a test machine, what happens if you write bad data to the graphics card hardware and you now have a line down the center of your screen? So what, 20 USD for another machine and your on your way. What happens if you do the same to your flagship machine? You don't sleep at night any more...
Second reason for the older machine, it might have a floppy drive in it. I will come back to this in a minute.
You will need some sort of media device to transfer your operating system to this test machine. Currently, a USB Thumb drive is the best way to do this. There are a few utilities out, bzt has one, that will write images to USB thumb drives. Most machines, even the older 20 USD machine I wrote of above, will boot from that USB thumb drive. This thumb drive will be emulated as a 1.44meg floppy or more likely, a hard drive, by the BIOS/Firmware. You can test your code just fine.
However, what happens when you tell the UEFI to stop emulating the USB thumb drive as a hard drive (EFI::ExitBootServices) or when you move to protected mode on the Legacy BIOS machine? That Thumb Drive is now gone. You have no access to it until you load a USB driver--hopefully you already have before you get this far. This means you have to have a working UHCI, OHCI, EHCI, and/or xHCI driver(s) just to access this Thumb Drive.
There is also a huge caveat here. If your BIOS/Firmware emulated the USB Thumb Drive as a hard drive, it most likely will try to keep control of that drive. The System Management firmware/hardware will not give up control of that drive for you to run your tests untouched until you tell the SM to release it. Now you have to write code, most likely quite specific to the current hardware, to release the USB before you can run your tests.
This is where the floppy drive comes back in to play. If I run my tests from the floppy drive, the USB is completely mine since I told the BIOS settings to turn off all USB emulation.
In all of my tests, when the USB was concerned, all of my tests and utilities ran just fine from the floppy. However, when I tried to run them from the emulated USB Thumb Drive, none of them worked as expected.
Yes, if you don't have a floppy drive, there is another route you can go. You can write your code to the Thumb Drive as simply a transfer media device, boot the thumb drive to DOS, Windows, *nix, whatever, and copy these files to the test machine's hard drive. Then, after removing the thumb drive, reboot the machine so that it boots from the hard drive. This will also get around the emulation part of the USB. However, this is two reboots per test. The floppy way was only a single reboot per test.
I guess my point is, it doesn't really matter the type of media device you boot from, as long as it allows you to have full control of the hardware during your tests. You can choose the type that suits your needs the best. In my tests, and throughout the R & D of my books, I used a floppy drive on five or more 32-bit machines. It was quite simple to insert the floppy disk, boot, test, and move to the next machine with the same floppy disk.
Now for the reason I used FreeDOS. Again, as mentioned before, you need a way to have full access to the hardware. FreeDOS allows this. The advantage to FreeDOS, or any other DOS for that matter, is it also has File functions and memory allocation.
The File functions was a good advantage so that I could 'pipe' my results out to a file and return that file to my flagship.
The Memory Allocation was a good advantage so that I could allocate physical memory. However, DOS--including FreeDOS--won't allow you to allocate physical memory, let alone physical memory above the 1 Meg mark, and more than 64k at a time. There needs to be an extension to allow this. This is where the DPMI (DOS Protected Mode Interface) comes in. Using the compiler I chose, DJGPP, I can tell it to use these extensions when it compiles my code.
These extensions allowed me to allocate physical memory, and write to memory above the 1 Meg mark, more specifically, hardware Mem-mapped areas. Using DJGPP, a variant of the popular *nix compiler but for DOS, allowed me to easily write code to access this physical memory.
However, to do so, you have to have the extension driver loaded at run time. This driver could be any compatible driver, I chose the CWSDPMI driver since it is well known and used. To answer your question, the compiled code the DJGPP prepends to your code will check to see if this driver is already loaded, and if not, will load it for you. If you don't load the DPMI at AUTOEXEC.BAT time, you will have to load the DPMI before every load of the .EXE file. It slows down the load time, but removes itself from memory afterward. If you do load the DPMI at AUTOEXEC.BAT time, it remains resident (TSR) and speeds up the .EXE file execution time.
I clearly state in all my books that I use this type of memory allocation, that it is merely a way to allocate and/or access memory. When you write your own drivers, you will need to have another way to allocate and access memory. If you are writing a driver, whether it be USB or otherwise, you should have already a working memory allocation mechanism and physical memory access mechanism before you even get to hardware drivers.
So to summarize, I used a floppy disk for all media transfers and boot-ups for convenience. I could have used a USB thumb drive, but this got in the way of my USB testing. I could have used a Linux or other variation, or any other type of operating system that would allow me to have full control of the machine, but I chose a freely available, still maintained, operating system that was easily available and easy to use.
In your research and development, you might find a different platform of working items. If your machines, your test machine(s) and your flagship, neither have floppies, you can't use that form of media transfers. If you are working on your USB code, which you clearly are at the moment, I strongly suggest that you do not use the booted Thumb Drive method unless you copy to an internal hard drive and reboot to that hard drive.
I hope this helps. Please do not feel that I was in any way belittling you since you are clearly at the beginning stages of your project. Please continue to post questions and we will do our best to answer them.
Thank you to everyone here for spending your time on this hobby of ours, giving help to those in need.
Ben
First, I suggest to use both an emulator and real hardware. I even include the instructions to use Bochs with a compiled version of Bochs within the ISO you pointed to.
Using an emulator is much more convenient that real hardware. You can use the same machine--in my case, my flagship--to code, debug, build, have a few .PDFs open, as well as load the emulator and run the current build. This makes it very convenient where you don't have to copy anything to a physical media device, and/or transfer it to another machine, and/or reboot any machine. This is the preferred way of doing this processes.
However, you have found out that on the emulator, your code seems to work just fine, yet when you move to the hardware, it all of a sudden, doesn't. This, unfortunately, will happen most every time when it comes to hardware programming. The emulator, no matter how well it is written, can't simulate the hardware exactly enough to allow your code to run on both without extensive debugging.
Now, since you do need to run it on real hardware, you have to have access to that hardware. For example, you can't reset the xHCI controller and send/receive packets without the current operating system--Windows 10 or a *nix varient--getting in the way. On top of that, if you did reset the controller, how would the current OS then continue to use that controller? Therefore, you must run the test code on bare hardware using no existing (modern) operating systems, or at least an OS that will let you have full control of the hardware.
DOS-which FreeDOS is a good variant--will allow you to have full control of the hardware. Now, how do you run DOS on your machine? You boot a floppy or a USB thumb drive with an already formatted partition, with DOS loaded on it. Hence, the freedos.img on all my ISOs that come with my books. The freedos.img file is an already built image that will boot to FreeDOS and allow you to run all of the compiled test suites I include with each book.
Now, why a floppy? Two reasons. First, as I suggested earlier, I highly recommend that you find a machine that someone is throwing out, a machine at a yard sell, a machine off ebay, any cheap somewhat modern machine. You can find many 32-bit machines with Legacy BIOS at yard sells, garage sells, on the curb for the trash guy, etc. For a 64-bit machine, you will need to do a little more looking, but still can find a decent older 64-bit machine for less than 20 USD. I picked up one the other day just because someone thought it wasn't fast enough. A complete wipe of the hard drive and I had a new 64-bit, 3.0g machine for testing.
Why find an older machine? If you pick one up for 20 USD and use it for a test machine, what happens if you write bad data to the graphics card hardware and you now have a line down the center of your screen? So what, 20 USD for another machine and your on your way. What happens if you do the same to your flagship machine? You don't sleep at night any more...
Second reason for the older machine, it might have a floppy drive in it. I will come back to this in a minute.
You will need some sort of media device to transfer your operating system to this test machine. Currently, a USB Thumb drive is the best way to do this. There are a few utilities out, bzt has one, that will write images to USB thumb drives. Most machines, even the older 20 USD machine I wrote of above, will boot from that USB thumb drive. This thumb drive will be emulated as a 1.44meg floppy or more likely, a hard drive, by the BIOS/Firmware. You can test your code just fine.
However, what happens when you tell the UEFI to stop emulating the USB thumb drive as a hard drive (EFI::ExitBootServices) or when you move to protected mode on the Legacy BIOS machine? That Thumb Drive is now gone. You have no access to it until you load a USB driver--hopefully you already have before you get this far. This means you have to have a working UHCI, OHCI, EHCI, and/or xHCI driver(s) just to access this Thumb Drive.
There is also a huge caveat here. If your BIOS/Firmware emulated the USB Thumb Drive as a hard drive, it most likely will try to keep control of that drive. The System Management firmware/hardware will not give up control of that drive for you to run your tests untouched until you tell the SM to release it. Now you have to write code, most likely quite specific to the current hardware, to release the USB before you can run your tests.
This is where the floppy drive comes back in to play. If I run my tests from the floppy drive, the USB is completely mine since I told the BIOS settings to turn off all USB emulation.
In all of my tests, when the USB was concerned, all of my tests and utilities ran just fine from the floppy. However, when I tried to run them from the emulated USB Thumb Drive, none of them worked as expected.
Yes, if you don't have a floppy drive, there is another route you can go. You can write your code to the Thumb Drive as simply a transfer media device, boot the thumb drive to DOS, Windows, *nix, whatever, and copy these files to the test machine's hard drive. Then, after removing the thumb drive, reboot the machine so that it boots from the hard drive. This will also get around the emulation part of the USB. However, this is two reboots per test. The floppy way was only a single reboot per test.
I guess my point is, it doesn't really matter the type of media device you boot from, as long as it allows you to have full control of the hardware during your tests. You can choose the type that suits your needs the best. In my tests, and throughout the R & D of my books, I used a floppy drive on five or more 32-bit machines. It was quite simple to insert the floppy disk, boot, test, and move to the next machine with the same floppy disk.
Now for the reason I used FreeDOS. Again, as mentioned before, you need a way to have full access to the hardware. FreeDOS allows this. The advantage to FreeDOS, or any other DOS for that matter, is it also has File functions and memory allocation.
The File functions was a good advantage so that I could 'pipe' my results out to a file and return that file to my flagship.
The Memory Allocation was a good advantage so that I could allocate physical memory. However, DOS--including FreeDOS--won't allow you to allocate physical memory, let alone physical memory above the 1 Meg mark, and more than 64k at a time. There needs to be an extension to allow this. This is where the DPMI (DOS Protected Mode Interface) comes in. Using the compiler I chose, DJGPP, I can tell it to use these extensions when it compiles my code.
These extensions allowed me to allocate physical memory, and write to memory above the 1 Meg mark, more specifically, hardware Mem-mapped areas. Using DJGPP, a variant of the popular *nix compiler but for DOS, allowed me to easily write code to access this physical memory.
However, to do so, you have to have the extension driver loaded at run time. This driver could be any compatible driver, I chose the CWSDPMI driver since it is well known and used. To answer your question, the compiled code the DJGPP prepends to your code will check to see if this driver is already loaded, and if not, will load it for you. If you don't load the DPMI at AUTOEXEC.BAT time, you will have to load the DPMI before every load of the .EXE file. It slows down the load time, but removes itself from memory afterward. If you do load the DPMI at AUTOEXEC.BAT time, it remains resident (TSR) and speeds up the .EXE file execution time.
I clearly state in all my books that I use this type of memory allocation, that it is merely a way to allocate and/or access memory. When you write your own drivers, you will need to have another way to allocate and access memory. If you are writing a driver, whether it be USB or otherwise, you should have already a working memory allocation mechanism and physical memory access mechanism before you even get to hardware drivers.
So to summarize, I used a floppy disk for all media transfers and boot-ups for convenience. I could have used a USB thumb drive, but this got in the way of my USB testing. I could have used a Linux or other variation, or any other type of operating system that would allow me to have full control of the machine, but I chose a freely available, still maintained, operating system that was easily available and easy to use.
In your research and development, you might find a different platform of working items. If your machines, your test machine(s) and your flagship, neither have floppies, you can't use that form of media transfers. If you are working on your USB code, which you clearly are at the moment, I strongly suggest that you do not use the booted Thumb Drive method unless you copy to an internal hard drive and reboot to that hard drive.
I hope this helps. Please do not feel that I was in any way belittling you since you are clearly at the beginning stages of your project. Please continue to post questions and we will do our best to answer them.
Thank you to everyone here for spending your time on this hobby of ours, giving help to those in need.
Ben