bewing's complete bochs rewrite: test request

[bewing]

After a piece of code is created, it ends up getting patched unmercifully. This happens to almost all code. After too many patches, it is mandatory to rewrite the code from scratch: to properly incorporate all the kludgy patches ... to organize the code the way you realize it should have been organized in the first place ... and to add new features that were impossible with the previous design.

Bochs reached that point a long time ago. It is unreadable. I can't bear it. I decided to sacrifice my sanity and take on the job of rewriting the whole damned thing from scratch. So, ReBochs is my new complete rewrite of bochs that I've been working on for the last year. I am hoping that as a community we can collaborate on making this a significantly better emulator than either bochs or qemu. I am confident that the new basic design of this code can get us there.

A Subversion repository lives at QuokForge: svn co http://svn.quokforge.org/svn/rebochs/
Sourcecode files can also be downloaded at QuokForge: http://quokforge.org/projects/rebochs/repository
A tarball and a zipfile (with a pre-compiled Win32 build) can be downloaded from: http://quokforge.org/projects/rebochs/files

Please email any bug reports or code suggestions to me -- the best address is in the attached documentation file alpha_test.txt

Soon-to-be Advantages of ReBochs:
ReBochs is "ISC" (public domain) licensed, and has no bochs code in it
Benign and Nasty modes! I think this will be everybody's favorite feature (see the attached documentation for a description)
The code should be at least a million times easier to read than bochs
-- I know that's still not saying much, but it should make it easier to maintain and collaborate on it
ReBochs does not need special disk image files -- any old file will work (including raw disks soon)
The hardware models should be more accurate than the ones in bochs
There is hardly any conditional code in ReBochs
-- this makes the config/compilation process much simpler (and makes the code easier to read)
-- it also means that all hardware models can be turned off and on without recompiling
The code is designed to model up to 65 thousand CPUs, in a NUMA configuration
The code is designed to be easy to port to any hobby OS
The code is designed to be easy to port to any newfangled GUI API
The code is designed to run in a multithreaded multiCPU environment
The number of simulated CPUs per thread is user-configurable
Multiple styles of x86-type CPUs (with varying capabilities) can be modeled
The debugger interface is completely standardized, so you can freely write your own
The BIOS has been rewritten by Brendan and me, and has some advantages over the bochs BIOS
-- specifically in simplicity, readability, maintainability, and being "traceable" so far
The Make procedure has been tremendously simplified -- all the bochs "Make" problems with Windows are gone
There are long-term plans to add important new features and models
I am confident the code will be significantly faster than qemu (and certainly bochs)
Full support for OSX
Fast to configure and build


If you compile this code, see an executable size of 300 to 360K, and think to yourself "That can't be right!" -- it is. I write very tight code.


Current drawbacks & problems:
Most of the above features are only partially implemented so far
There is still lots and lots of stubbed code -- ie. floating point, mmx, sse, longmode (I'm coding as fast as I can, but oh well)

Current emulator version status:
Real mode should be running
Pmode is 99% running
-- a few rarely-used pmode-specific opcodes are still stubbed
Long mode is completely stubbed

Current debugger status:
GUI: 95% working, but some features have been stubbed out -- especially the parameter tree
Textmode: 10% working

If you are interested in submitting code/doing alpha testing, please read the attached document for more details.

[Thomas]

Hi ,
First of all, congrats for the great work done . But then there are certain things i want to see improvemet,there is not even single unit test written.I cannot stress the importance of that,infact all the good developers I know write more unit tests than the actual lines of code .Apoloiges in advance if you feel I am acting a little negative. I hope that your project comes out in flying colours

--Thomas

[technik3k]

To continue along the testing lines:

1. In the Nasty mode do you plan to implement checks for undefined flags and report if any conditional statement uses them? That would be very useful and, for example, is the reason for me to always test any assembly code on both qemu and vmware since they handle this differently.

2. Do you anticipate any external scripting interface that could drive automated testing? For example, keyboard and mouse inputs, IRQs, hardware events, suspend/resume, and simulated clock.

I am looking forward to read your code. Sounds very interesting!

[Firestryke31]

Probably not a high priority, but VC2008Express has trouble compiling the Win32 GUI file (2010E wouldn't even open the project, stupid VC6). I couldn't figure out how to get it to work properly, so I have no solutions. You might consider #defining the _CRT_SECURE_NO_WARNINGS or whatever it is to reduce the retarded superfluous warnings and make getting to the real problems easier (you don't even have to set it to anything, just define it).

Also, why write 80 different GUI frontends when you could use a library like wxWidgets that abstracts the native GUI system for you? I'm sure you could write a C wrapper if you feel you need to.

Edit: Got 2010E to open the project by importing the converted 2008E solution. Whee.

[bewing]

@Thomas: Intel has some kind of proprietary software/hardware to unit test CPUs. So does AMD. It is so complicated to do that NOBODY ELSE has one, AFAIK. I looked. I wish I could unit test this. You may want to notice that no other emulator has the ability to unit test itself, either -- it is that hard to do. Unit testing software would be twice the size of the emulator, and would need to be written in pure ASM, and would only be useful if it tested all the corner cases. So: feel free to write it yourself as a contribution to the project, but I don't have that kind of time. Or if you can just unit test one opcode, that'd be great -- especially if a whole bunch of alpha testers could all do one, in some coordinated way.

@technik3k: Yes, I very specifically set all undefined flags that result from all calculations to 0. Even in benign mode. If your code depends on the state of an undefined flag, it may not crash -- but that's the most efficient solution. Attempting to tristate undefined flags might be theoretically possible, but then I couldn't use bitflags anymore, and that'd make my life too hard. I tried to figure out a way to psuedorandomize undefined flags, that was also deterministic (so your code would always work repeatably, and you could find the bug) -- but for the moment, that was too hard. Edit: Hmmm. Actually, I just thought of a way I could do it, and still be quite efficient in Nasty mode. Thanks for the jog to my creativity! I'll add it to the list of medium-priority features.

Q2: there will probably be some sort of a scripting thing built into the textmode debugger that works similarly to what bochs does. The textmode debugger doesn't even exist yet, so its functionality is still quite flexible, you might say.

@Firestryke31: Look, I went through all this "platform independent" BS when I wrote the initial GUI debugger for Bochs. The platform independent wrappers for GUI systems barely work. Did you notice that GTK *IS* platform independent (basically), and it is the main piece of code that I am having the most trouble with? Current platform independent GUI code sux sooo hard that I abandoned the concept in disgust after looking at it thoroughly.
The debugger is the center of this emulator, and I want waaaay more out of a debugger (visually) than some piece-of-carp like WxWidgets can provide. Have you seen the "debugger" for bochs that they tried to build with WxWidgets? It went beyond totally sucking. It was so bad Stanislav deleted it. It might be possible to write a debugger that was decent in Javascript, as mcheung63 did -- and have it be truly independent. Do you know that WxWidgets is written as a lobotomized front-end on GTK?

But that's great that you managed to get the workspace to load -- I didn't realize it was so non-backward compatible. Sorry about that. And if you want fewer warnings, just reduce the /Wx warning level to 2 or 1. I'm surprised you saw any, truly. At some point, I'll ask for lists of trivial warnings from various compilers.

[Brynet-Inc]

I've managed to compile and run this on OpenBSD, but is hasn't been able to boot anything I've thrown at it yet.. but I suppose it's still at an early stage.

[bewing]

Yes, you are probably going to need to go through the boot process a few steps at a time, to see where it fails. That's the interesting thing I want to know -- precisely where does it fail? Success is not interesting.

@Firestryke again -- If you could make and send me some .dsw, .dsp, etc files from your 2010 -- I'd love to include them in the package. (Note: in a month or two, I'll be adding more sourcefiles to the project, so this is something that would have to be done again later -- I do not have access to 2010E.) -- Oh! And did you download the zipfile? Did you try the executable?

[Artlav]

Intel has some kind of proprietary software/hardware to unit test CPUs. So does AMD. It is so complicated to do that NOBODY ELSE has one, AFAIK. I looked. I wish I could unit test this. You may want to notice that no other emulator has the ability to unit test itself, either -- it is that hard to do. Unit testing software would be twice the size of the emulator, and would need to be written in pure ASM, and would only be useful if it tested all the corner cases. So: feel free to write it yourself as a contribution to the project, but I don't have that kind of time. Or if you can just unit test one opcode, that'd be great -- especially if a whole bunch of alpha testers could all do one, in some coordinated way.

I think i can help you with that.
I developed and adapted various tests in a full-coverage test suit of 80186 commands while developing and cleaning up my x186 emulator, and i guess it can be of use.

I've tested some of them on yours, and there are plenty of errors in flags, and some of the results for BCD commands, but without a way to dump memory to file (that i can find) it's a lot of drudgery to verify the results (in otherwise gorgeous debugger, nice work there).

You can get them here:
http://orbides.1gb.ru/80186_tests.zip
res_*.bin contains the results expected at the offset 0x0 when you run *.bin test placed as a BIOS.
Note, that they are designed to match the 80186 behaviour, so flags might differ.

[bewing]

Hey! Great work, Artlav! I'll look at that closely!
Dumping things to disk is one more future feature that is somewhere on my todo list. Maybe I'll move it up.
For the moment, I think I'll try adding an Int13h command to your code to dump data back out to the disk image file.
And yes, the BCD opcodes are all completely stubbed. So I'm not surprised they didn't quite work properly.

Edit: there is no corresponding res_*.bin file for jmpmov.bin -- is there not one? Or did it just not happen to get into your zipfile?

[Firestryke31]

@Firestryke again -- If you could make and send me some .dsw, .dsp, etc files from your 2010 -- I'd love to include them in the package.

I'll do you one better: I'll give both the 08E and 10E solutions. I'm pretty sure they both work, but I'd appreciate verification.

Oh! And did you download the zipfile? Did you try the executable?

I did download the zip, but have not tried the exe yet.


Once I can figure out the source, I'll see if I can write a wxWidgets GUI for this thing. Maybe it's my little bit of experience in Java, but I understand how it works and can make reasonable looking things using it. I can tell you, it's better than the PoS GTK+ port for Windows simply because it uses the native dialogs instead of reinventing everything. I'm sure under Mac it's also far superior to the PoS GTK+ port there too.

Edit: Note to self: Try something before bashing the libraries it uses. And don't base all conceptions of GTK+ around stInkscape's version.

[bewing]

I'll post your workspaces soon. Thanks for those.

I agree that GTK is a total PoS, but after lots of struggles I don't think I have any better choice available on posix systems to get the complex graphics behavior that I need -- even to just barely match what I can do moderately easily on Win32. And even on Win32 I think the GDI design is unnecessarily difficult.
What I find amazing is that some crazy people managed to write complete Ubuntu desktops and apps using GTK. I'm not sure how many people died in the process, but I'm sure it was a few.

[Brynet-Inc]

ReBochs is crashing (...SIGABRT) when I close it here, turns out it was running into ProPolice/SSP stack protection (..my /var/log/messages was being spammed).

The fault appears to be originating in toolkits/gui_gtk_tkit.c:void GUI_TkitInit(), perhaps you should consider using safe string handling functions.. strlcpy/strlcat from OpenBSD and snprintf instead of sprintf.

Fixing this at least allows me to return to the numbered text interface, although pressing "2" to resume emulation causes a SIGSEGV in some of the GLIB/GTK libs.. which I have no desire to hunt down.

As for reporting success vs. failure, so far, no success at all in the sense of a working boot sector.. the i86 version of the Minix 2.0.3 loader seems to cause screen distortions (..when you press keys).

All in all, I like the idea of an ISC licenced emulator.. but some of the coding practices could maybe be improved a little, as mentioned.

Good luck.

[Firestryke31]

What complex behavior do you think you'll need? The basics of what I've seen so far are pretty simple to do in wxWidgets. One big window showing the 'display' (an image object from the raw bitmap data), a window with a few list controls (the colored text IIRC isn't even all that special to it), and some buttons. The hard part would be integrating the C-based backend with the C++-based GUI, and I'm sure if I spent more than 30 minutes glancing through the code I could think of something with a little help. I think the two most annoying design aspects based on what I've seen so far are you calling the GUI system instead of the other way around (and that might just be me not looking close enough) and those odd typedefs.

[bewing]

ReBochs is crashing (...SIGABRT) when I close it here, turns out it was running into ProPolice/SSP stack protection (..my /var/log/messages was being spammed).

The fault appears to be originating in toolkits/gui_gtk_tkit.c:void GUI_TkitInit(), perhaps you should consider using safe string handling functions.. strlcpy/strlcat from OpenBSD and snprintf instead of sprintf.

This code must be multi-platform -- it also has to run on Win32. So I can't go using any old function I feel like. Either I write it myself, or it should probably be a 1976 C-standard function. So, no BSD-redefined crap allowed.

But, if this helps me track down GTK-bug #2, as mentioned in my original post, then you get 4 gold stars.

[Brynet-Inc]

Plenty of projects include a copy of the strl* functions, they're quite small... portable.. bundling them can't hurt, same permissive licence.

Hardly crap.

http://www.gratisoft.us/todd/papers/strlcpy.html