Is possible to make Bochs work with HDD faster?

[fano1]

Hi! I'm having a problem with Bochs... that its R/W on HDD are really slow, I've implemented FAT32 quick format in Cosmos OS and in VmWare it lasts 22 seconds on Bochs it is running for 20 minutes already

This is our .bochsrc file:

Code: Select all

# configuration file generated by Bochs
plugin_ctrl: unmapped=1, biosdev=1, speaker=1, extfpuirq=1, parallel=1, serial=1, gameport=1
config_interface: win32config
display_library: win32
debug_symbols: file="C:\Users\fano\Documents\GitHub\Cosmos\Cosmos\Tests\Cosmos.TestRunner\bin\Debug\netcoreapp2.0\win7-x86\WorkingDirectory\BoxingTests\Kernel.sym"
memory: host=256, guest=256
romimage: file="C:\Program Files (x86)\Bochs-2.6.9/BIOS-bochs-latest"
vgaromimage: file="C:\Program Files (x86)\Bochs-2.6.9/VGABIOS-lgpl-latest"
boot: cdrom
floppy_bootsig_check: disabled=0
# no floppya
# no floppyb
ata0-master: type=cdrom, path="C:\Users\fano\Documents\GitHub\Cosmos\Cosmos\Tests\Cosmos.TestRunner\bin\Debug\netcoreapp2.0\win7-x86\WorkingDirectory\BoxingTests\Kernel.iso", status=inserted, model="Generic 1234", biosdetect=auto
ata0: enabled=1, ioaddr1=0x1f0, ioaddr2=0x3f0, irq=14
ata0-slave: type=none
ata1: enabled=1, ioaddr1=0x170, ioaddr2=0x370, irq=15
ata1-master: type=disk, path="C:\Users\fano\Documents\GitHub\Cosmos\Cosmos\Tests\Cosmos.TestRunner\bin\Debug\netcoreapp2.0\win7-x86\WorkingDirectory\BoxingTests\Harddisk.vmdk", mode=vmware4, cylinders=0, heads=0, spt=0, model="Generic 1234", biosdetect=auto, translation=auto
ata2: enabled=0
ata3: enabled=0
pci: enabled=1, chipset=i440fx
vga: extension=vbe, update_freq=5, realtime=1
cpu: count=1, ips=4000000, model=p4_prescott_celeron_336, reset_on_triple_fault=1, cpuid_limit_winnt=0, ignore_bad_msrs=1, mwait_is_nop=0
print_timestamps: enabled=0
port_e9_hack: enabled=0
private_colormap: enabled=0
clock: sync=none, time0=local, rtc_sync=0
# no cmosimage
# no loader
log: -
logprefix: %t%e%d
debug: action=ignore
info: action=report
error: action=report
panic: action=ask
keyboard: type=mf, serial_delay=250, paste_delay=100000, user_shortcut=none
mouse: type=ps2, enabled=0, toggle=ctrl+mbutton
sound: waveoutdrv=win, waveout=none, waveindrv=win, wavein=none, midioutdrv=win, midiout=none
speaker: enabled=1, mode=sound
parport1: enabled=1, file=none
parport2: enabled=0
com1: enabled=1, mode=pipe-client, dev="\\.\pipe\cosmos\serial"
com2: enabled=0
com3: enabled=0
com4: enabled=0

I know that Bochs emulates the entire PC and so is a different thing from VmWare but yet is 100x slower than VmWare so much that I fear my timeout of unit test (25 minutes) it will be reached before the disk will be "quickly" formatted.

Any idea of how to make it faster?

[stlw]

I see you have configured Bochs for IPS=4000000. It means that after each 4M emulated instructions emulated second is passed.
Modern OSes may issue timer interrupts more than 1000 times per second and take thousands of instructions to handle an interrupt.
With such IPC setting you probably emulating timer interrupt routine ~50% of the time

Where did you take the 4M setting ?
I believe your hardware is capable for much more, unless you really running it on Prescott Celeron 336Mhz or so.
My Android phone is able to do 10x faster.

[stlw]

Example .bochsrc shows typical speed to expect for:

# 2.4.6 3.4Ghz Intel Core i7 2600 with Win7x64/g++ 4.5.2 85 to 95 Mips
# 2.3.7 3.2Ghz Intel Core 2 Q9770 with WinXP/g++ 3.4 50 to 55 Mips
# 2.3.7 2.6Ghz Intel Core 2 Duo with WinXP/g++ 3.4 38 to 43 Mips
# 2.2.6 2.6Ghz Intel Core 2 Duo with WinXP/g++ 3.4 21 to 25 Mips
# 2.2.6 2.1Ghz Athlon XP with Linux 2.6/g++ 3.4 12 to 15 Mips

latest Bochs sources on Intel Core i7 4770 with Win10x64/g++ 7.3.0 running between 125Mips and 150Mips

[fano1]

I don't know where the 4 MIPS comes, probably has been so from the beginning... what could be a good number in your opinion?
If I simply does not set it will use the value of an emulated "p4_prescott_celeron_336" (this one https://ark.intel.com/products/27119/In ... 33-MHz-FSB) that should have a frequency of 2.6 GHz or I'm forced to set IPS always?

I remember to have changed the CPU model years ago because I needed SSE2 support and that is one of the first CPU to do that.

Changing to 40 MIPS (150 MIPS seems to do the same) I've got 12 minutes that is half of before but it is always a lot respect to VmWare where all is done in 22 seconds!

[BenLunt]

Having worked with Bochs for quite some time I have to say that, though you do need to up your MIPs a bit, that is not the issue.

Bochs is much slower than other emulators for the fact that it checks for things in a much more detailed manner. For example, some emulators allow the host system to process a memory move. Bochs processes the move completely independent from the host checking for everything from paging errors to being physically available. This was the intent with Bochs. Bochs was designed to test and debug guests more so than to simple emulate the guest.

Since the slowdown is considerable, 12 minutes compared to 22 seconds on other emulators, I am guessing that you are doing a lot of memory manipulation for your "quick" format. For example, are you configuring offsets, FAT locations, etc., while formatting the drive? This will take a considerable amount of time. If you calculate each part of the format before hand, then write to the disk as the last process, this will considerably speed up the process.

Please look over your process of "quick" formatting the drive. I believe that this might be the speed issue, not the physical part of writing to the disk. As far as Bochs writing to the disk, it simply has a handle to a file on the host machine, allowing reading and writing via the host. This is done at the speed of the host. Since it will be the same for any emulator on that particular host, this implies that the disk IO is not the issue.

Ben
- http://www.fysnet.net/osdesign_book_series.htm

[Brendan]

Hi,

Hi! I'm having a problem with Bochs... that its R/W on HDD are really slow, I've implemented FAT32 quick format in Cosmos OS and in VmWare it lasts 22 seconds on Bochs it is running for 20 minutes already

How are you transferring data to/from disk?

If you use PIO, then to transfer 20 MiB of data Bochs probably has to emulate ~5 million instructions (and because Bochs is an almost pure interpreter without JIT or hardware virtualisation, and because a device is involved and they aren't normal instructions, that's going to be extremely slow); and if you use some type of DMA then Bochs can probably do the transfer at native speeds (e.g. maybe transfer the data using "read()" or "write()" in C and then pretend the transfer happened instantly by generating an "transfer completed" IRQ within the guest immediately after the instruction that started the transfer).


Cheers,

Brendan

[fano1]

My FAT quick format implementation algorithm is this:

1. Get any file / directory that are in the RootDirectory and delete all of them, "delete" as usual mean not delete the data part but simply to set the entry as not used. I'm in C# and any directory entry is actually a class so obviously there will be memory allocations but notice that we are talking of 5 / 6 directory or files maximum. There is no recursion too.
2. I put 0s in all the FAT Table, avoiding the first 2 clusters and any bad cluster. I've done this reading any element of the FAT in a 4KB C# array doing a for on it to put the 0s on it and then I write all the 4KB on the disk again. If I could have used Array.Clear() (sort of memset()) it would have been more faster but I've reserved and bad clusters to preserve so...

@Brendan yes my colleagues confirm Cosmos is using PIO because it was more simpler to implement... probably we need to implement DMA as soon as possible.

[BenLunt]

Maybe I don't understand the type of "quick format" you want to do. In my idea, a "quick format" of a volume (not a media device) is simply to clear the root directory and file allocation tables/nodes/etc. Completely forget about the files and filenames.

On a FAT volume, this would be to simply write zeros to the root directory and change all used FAT entries to unused, preserving any entries marked bad. The file data needs absolutely no modification, nor do you need to recurse through all files. Only the root directory and the FAT gets over-written. Depending on the type of FAT, you may need to update the BPB and/or FAT32_Info blocks.

This is an extremely quick operation, even in Bochs this should be 2 or 3 seconds at most.

Ben

[fano1]

Yes I've only changing the RootDirectory and the FatTable it is this second operation that is really slow in Bochs sadly...

[Brendan]

Hi,

My FAT quick format implementation algorithm is this:

1. Get any file / directory that are in the RootDirectory and delete all of them, "delete" as usual mean not delete the data part but simply to set the entry as not used. I'm in C# and any directory entry is actually a class so obviously there will be memory allocations but notice that we are talking of 5 / 6 directory or files maximum. There is no recursion too.
2. I put 0s in all the FAT Table, avoiding the first 2 clusters and any bad cluster. I've done this reading any element of the FAT in a 4KB C# array doing a for on it to put the 0s on it and then I write all the 4KB on the disk again. If I could have used Array.Clear() (sort of memset()) it would have been more faster but I've reserved and bad clusters to preserve so...

What you've described is "fast bulk delete". What happens if I have a partition that was formatted as ext2 (or something else), and I use quick format to change it to FAT? What happens if it's a newly partitioned drive that didn't have anything before? What happens if the partition was formatted with FAT but something went wrong and the cluster allocation table was corrupted (filled with trash)?

For quick format; you'd generate new data (BPB, cluster allocation table, root directory) and write them to disk, without reading anything that was previously on the disk (because all of the reasons for formatting a volume imply that the old data can't or shouldn't be relied on).

For slow format; it's mostly the same except that you'd test the sectors while you're formatting.

Note that modern hard drives have spare sectors and will auto-replace faulty sectors with these spare sectors; so if you actually are replacing FAT with FAT then all of the sectors that were previously marked as "bad" were probably replaced with sectors that are good and should probably be marked as "not bad anymore". It's only when the hard drive needs to be thrown in the trash (because there's been so many bad sectors that there are no more spare sectors left) that a previously bad sector will remain bad.

Of course for SSD it's all completely different - wear levelling means that all physical sectors are shuffled around regularly (and that when formatting the device you should probably just "trim" everything so that the device knows that all sectors can be used for wear levelling again).


Cheers,

Brendan

[fano1]

I've changed my algorithm a bit not checking anymore for bad cluster... that is de-facto I'm not reading the FatTable anymore before writing on it... I get an improvement Bochs lasted circa 3 minutes instead of 21.
VmWare lasts the same time (22 seconds) so Bochs seems to suffer all those branches.

For now I'm happy with this time reduction in the end I think Bochs should not be used by normal Cosmos users but by us developers the possibility to have GDB and to configure what model of CPU to emulate is great for us but for the user is not interesting while he will notice the slowness.