Another Alternate GUI for Bochs

[bewing]

This app has been fully incorporated into the most recent version of bochs (anything 2.4 or higher). -- This thread is now obsolete. Please download either bochs at sourceforge, or bebochs to get an emulator with a very nice GUI frontend.

[stlw]

I have taken the Bochs GUI frontend for windows that was created by WindowsNT, and posted on this thread -- and modified it quite a bit to suit my tastes.
It has different commands, more options, different menus, a slightly different look, extra features, and should be faster.

So, this is a formal test request for my beta version (0.3). The download on this post contains (very easy) instructions for hand-patching the code into bochs -- it should be compatible with any bochs ver 2.3.X -- I hope.

If you patch it into a different version than 2.3.7, I would be esepcially interested to hear how it goes.

In a few hours (as soon as I can take a nap and then figure out how to do it ... ) I will post another copy that does the patch automatically into bochs 2.3.7 (especially for Stanislav to play with).

This version still has the ability to interact with the "internal bochs textmode debugger" -- to issue the commands that you all are familiar with. Within a week or so, I will be posting a version that completely replaces the bochs internal debugger.

Hi, I am not sure the version which compleetely replaces the Bochs debugger will be so accepatble. At least I see amny problems with integrating it into Bochs CVS.
I think you should keep this ability.
Could you explain what is the problem with it for you ?

Stanislav

[bewing]

Could you explain what is the problem with it for you?

1) If I get rid of the internal debugger, I don't need the space at the bottom of the screen for the Input and Output windows. This leaves more room for important things, and simplifies the design of my program.

2) The current bochs internal debugger is inefficient. For example, the watchpoint and breakpoint lists are not sorted. If they were sorted (and the algorithm made use of the fact that they were sorted), it would speed up the simulation a lot.

Also, the handling of the memory callbacks is not as efficient as it should be -- and the "if" is backwards, so the watchpoints don't work correctly anyway.

3) I don't need the internal debugger anymore. It is just a barrier between me and the sim. You have been pointing out how a debugger can control the sim directly. I am intending to do exactly what you have said.

4) The internal debugger has hardcoded limitations. For example, the 16 entry limit on the watchpoints. Or the fact that internal debugger watchpoints are limited to physical memory only, and there is no support for multi-cpu linear memory watchpoints.

5) There is an enormous amount of code duplication between my debugger and the internal debugger.

6) The internal debugger does not provide a good interface between me and the sim for running a multiple CPU simulation, where all the CPUs are running simultaneously.

For example, if I was controlling the sim directly, then I could have a multi-threaded debugger, controlling the simulation of each CPU with a different thread, simultaneously. I cannot do that through the internal debugger.

7) The current internal debugger is not stateless, as far as I can tell. So I have to always keep its state consistent, even though I am not really using it anymore, except to singlestep the sim.

8) The way that the internal debugger informs me of breaks in the simulation is kludgy and stinks. The way that it informs me that it is ready to process another command is also kludgy.

9) You said yourself: "The problem that the Bochs internal debugger is currently part of Bochs and it is not the best debugger ever so I would like to be able to separate it from Bochs and have an option to attach another debugger, GUI one or new command line one."

I agree with what you said. If the internal debugger is taken out of bochs, and made optional -- then I shouldn't make my debugger dependent on it, should I? Especially since I don't need to. So I am making "another GUI debugger that you can attach to bochs" instead of the current internal debugger.

###########

Attached is the "patch" version of the code archive. Instructions are at the front of the patch file. Nov 2, Ver 1.1

[stlw]

1) If I get rid of the internal debugger, I don't need the space at the bottom of the screen for the Input and Output windows. This leaves more room for important things, and simplifies the design of my program.

Keep it optional

2) The current bochs internal debugger is inefficient. For example, the watchpoint and breakpoint lists are not sorted. If they were sorted (and the algorithm made use of the fact that they were sorted), it would speed up the simulation a lot.

BTW, I till would likee to keep the "legacy" debugger and as I step by step make it more and more usable since 2.1 release - I want to keep doing that.
So whatever inetrface we will choose - the new debugger you suggest or current internal debugger interface *which WindowsNT used) - I want to make currrent debugger compatible with this inetrface and working. Also if you hav some suggestions for making it more efficient - I am open for ideas

Also, the handling of the memory callbacks is not as efficient as it should be -- and the "if" is backwards, so the watchpoints don't work correctly anyway.

This I can't understand. As far as I know the watchpoints work correctly everywhere.

3) I don't need the internal debugger anymore. It is just a barrier between me and the sim. You have been pointing out how a debugger can control the sim directly. I am intending to do exactly what you have said.

I see, so you want to go to your new debugger. Actually I intent to leave everything in your hands. You write the debugger trying to follow Bochs coding guidelines and if you think you need ANY modification in Bochs CPU - you just ask and get them. This probably will require for you to move into CVS version ans stop working on 2.3.7 as baseline.

6) The internal debugger does not provide a good interface between me and the sim for running a multiple CPU simulation, where all the CPUs are running simultaneously.

For example, if I was controlling the sim directly, then I could have a multi-threaded debugger, controlling the simulation of each CPU with a different thread, simultaneously. I cannot do that through the internal debugger.

But until your debuger has ALL that internal debugger has (including save/restore support and work with param treee) + will suit for all Bochs' usages (x86-64 support, smp support) - it will not become default replace for internal debugger. Also I think testual internal debugger always will be, just for hosts with very strange GUI you can't predict.

7) The current internal debugger is not stateless, as far as I can tell. So I have to always keep its state consistent, even though I am not really using it anymore, except to singlestep the sim.

So make it stateless
My best suggestion is to go and develop internal debugger in parallel with your s.t. intenal debugger will become a true subset of yours.
Like if you want - you disable all GUI functionality and stay with text only.

The way that the internal debugger informs me of breaks in the simulation is kludgy and stinks. The way that it informs me that it is ready to process another command is also kludgy.

Your suggestions ?

Stanislav

[bewing]

But until your debuger has ALL that internal debugger has (including save/restore support and work with param treee) + will suit for all Bochs' usages (x86-64 support, smp support) - it will not become default replace for internal debugger.

I already have full 64 bit compatibility. By the end of this week, I will have better SMP and param_tree support than the current internal debugger has.
So the question is whether I can figure out how the internal debugger is preventing me from having really good SMP support. Because right now, if I bypass the internal debugger and call cpu_loop(0) from "the wrong thread", then cpu_loop crashes.

As far as save/restore goes: there is no documentation on this, so it will be hard to do. How does it work? What do I need to do to support it? What needs to be saved and restored?

You write the debugger trying to follow Bochs coding guidelines

I think I have. The only thing I didn't do was convert tabs to spaces. When I have a more fully-tested version, that I really intend to submit as a bochs patch, I can write a quickie routine to do the tab conversion on the files.

I want to make currrent debugger compatible with this inetrface and working.
Your suggestions?

The problem is: the current internal debugger interface is single-threaded, and is not thread-safe.
The sim also assumes that it is single-threaded.
There are places where it tests SIM->is_sim_thread() -- this will need to be eliminated. The CPU code is probably already thread-safe, except that it currently expects to ONLY be called from the single debugger/sim thread.

Probably the easiest and best way to "patch" this (so it continues to work with everything), is to use a registered callback inside the bx_dbg_user_input_loop() function. That way, I can change it from accessing the current sleep loop, and have it call MY input function synchronously, when it wants input.
This would also serve as a flag that the simulation was at a break.
I will write an example, and see if I can get you and WindowsNT to agree to it.


Errors I have seen so far in the bochs code:

...and the "if" is backwards, so the watchpoints don't work correctly anyway.

This I can't understand. As far as I know the watchpoints work correctly everywhere.

The watchpoint "if" currently says:

Code: Select all

      if (write_watchpoint[i] <= phy && write_watchpoint[i] + len > phy) {

It should say:

Code: Select all

      if (write_watchpoint[i] >= phy && write_watchpoint[i] < phy + len ) {

To see that the watchpoints don't work correcty, try setting one to a byte address in the middle of a dword, and then accessing the entire dword. Your watchpoint won't fire.

ALSO -- I mentioned this to you once before (but I said the problem wrong):

Currently, the A20ADDR macro says:

Code: Select all

#if BX_SUPPORT_A20
#  define A20ADDR(x)                (bx_phy_address(x) & bx_pc_system.a20_mask)
#else
#  define A20ADDR(x)                (bx_phy_address(x))
#endif

But the bx_phy_address is supposed to be a cast -- so it should have more parentheses and say:

Code: Select all

#if BX_SUPPORT_A20
#  define A20ADDR(x)                ((bx_phy_address)(x) & bx_pc_system.a20_mask)
#else
#  define A20ADDR(x)                ((bx_phy_address)(x))
#endif

My compiler is picky enough to get a fatal error over this A20ADDR macro.

Also if you hav some suggestions for making it more efficient - I am open for ideas.

This applies to the watchpoint list, and also to the way breakpoints are handled in CPU.

If you have a sorted breakpoint list, you can make a loop that looks like this:

Code: Select all

	bx_address h = CurrentAddress;
	int j= BreakCount;
	while (--j >= 0)			// loop over all breakpoints
	{
		// brk list is sorted -- if the value goes too low, end the loop
		if (BrkLAddr[j] < h)
			j = 0;		// force the loop to end if it goes too far -- no breakpoint match
		else if (BrkLAddr[j] == h){
			j = 0;
			break!	// hit a breakpoint
		}
	}

Notes:
1) Counting the loop down, so it tests on zero, is always more efficient -- if possible.
2) Storing the breakpoint list in a separate array, rather than inside a structure, is always more efficient.
3) The point about this loop is that on average it only has to test HALF of the breakpoints before it knows that there is no match.
4) It is even possible to start j at a better value than BreakCount. You can start the loop by testing if BrkLAddr[BreakCount/2] >= h -- if it is, then you start j there. Then, on average, you only have to test a quarter of the breakpoints before you know there is no match.
5) Most current compilers are smart enough to understand that you are doing the same comparison twice, so GOOD compiled assembly code would say something like:

Code: Select all

	mov ecx, [BreakCount]
.brklp:
	dec ecx
	jl short .done
	mov eax, [h]
	cmp [BrkLAddr + ecx*4], eax
	jge short .tste
	xor ecx, ecx		; kill the loop
	jmp short .brklp
.tste:	jne short .brklp
	break!			; hit a breakpoint
.done:

Unfortunately, MY compiler is so stupid that I have to force it, with something like:

Code: Select all

	int j= BreakCount;
	while (--j >= 0)			// loop over all breakpoints
	{
		// brk list is sorted -- if the value goes too low, end the loop
		register breakpt_address i = BrkLAddr[j] - h;		// precalculate the comparison
		if (i < 0)
			j = 0;		// force the loop to end if it goes too far -- no breakpoint match
		else if (i == 0){
			j = 0;
			break!	// hit a breakpoint
		}
	}

And if I had lots of time, I'd completely rewrite your param tree interface for you. If you put everything into the param tree, the way it is built right now, bochs is going to be one of the slowest emulators ever seen.

[Troy Martin]

bochs is going to be the slowest emulator ever seen.

No offense to the developers, but it already is. That's why (among with the configuration file editing crap) I don't use it for anything other debugging.

Now all we need is for MS to release Virtual PC as open source, we'd have the BEST virtualization platform ever with enough modding...

[bewing]

Dream on.

[Brynet-Inc]

Bochs is an emulator, not a "Virtualizer", it's also very portable.. runs on a wide range of architectures and operating systems.

If you want something open source, decent.. and fast.. try QEMU.

[bewing]

Bochs is good enough for me ... now, if it just had a nice GUI frontend ....

[stlw]

Bochs is good enough for me ... now, if it just had a nice GUI frontend ....

Now I have some suggestion about how we could converge. I am 100% sure we can't make major rewrite of interfaces + debugger GUI in one shot.
The same we had with CPU emulation, it is just impossible to integrate all the chnages at the same time and hope it will work anytime soon. Especially when you base your changes on some specific release (even latest 2.3.7) and not CVS tree. Your recently posted patch is already a bit outdate after fixes in watchpoints handling in Bochs CVS AFTER 2.3.7.

But this could be done step-by-step. This doesn't mean the steps can't run in parallel and it should take a lot of time between them. But it means that the some things would go first and some things will have to wait until previous patch was merged and accepted, probably even released.
I see you plan to go too far with your 'alternative debugger interface'. I don't mean your alternative interface is bad, but it cernainly too big for first step.
My suggestion for now is to integrate into CVS your 'fully compatible with text debugger' interface and start living with it.
You could add a separate patch/suggestions for improving param tree interface as well. I would like to release it as soon as possible. It means integrate the code, make it suit Bochs guidelines, widely test in various scenarios, fix issues found and release 2.3.8 even this month if you like. And when to go into next step and next step. TBD

I hope you will agree with that. Anyway, at this time I think you already should move into CVS version. I am sure somebody could open place where the most recent CVS version compiled wit new debugger interface is published everyday/week. This also will help to make a new release sooner.

Stanislav

[stlw]

I already have full 64 bit compatibility. By the end of this week, I will have better SMP and param_tree support than the current internal debugger has.
So the question is whether I can figure out how the internal debugger is preventing me from having really good SMP support. Because right now, if I bypass the internal debugger and call cpu_loop(0) from "the wrong thread", then cpu_loop crashes.

You should understand how the SMP CPU_LOOP is emulated and, most important, how timers and outside CPU stuff is handled in SMP.
Bochs is NOT running every CPU in other thread. It would be nice to have and some Chinese students started a project to support this - but their work is not finished yet.
Bochs running every CPU one after another, quantum of instructions on every logical CPU and when advance a timers in one shot after all CPUs are done with the quantum.
The CPUs are NOT trread-safe and you can't parallel them without crashing the emulation. Also if you mess with the timers you won't be able to boot anything serious as well.
So you must maintain the things as they are at least until we do some big work and make as is should be

As far as save/restore goes: there is no documentation on this, so it will be hard to do. How does it work? What do I need to do to support it? What needs to be saved and restored?

It is simple - Bochs debugger could save and restore any part of param tree by demand. Saving simulation state is done through usual save/restore dialog and debugger could restore any part of the tree using 'restore' command. Try 'help restore'. If you look into save/restore files - they are in text format, human readable and changeable - it is very powerful feature.

The problem is: the current internal debugger interface is single-threaded, and is not thread-safe.
The sim also assumes that it is single-threaded.
There are places where it tests SIM->is_sim_thread() -- this will need to be eliminated. The CPU code is probably already thread-safe, except that it currently expects to ONLY be called from the single debugger/sim thread.

No, the problem is bigger. Bochs CPU and devices are single-threaded and not thread-safe !
You can't run two cpu_loops in parallel at ANY time.

Probably the easiest and best way to "patch" this (so it continues to work with everything), is to use a registered callback inside the bx_dbg_user_input_loop() function. That way, I can change it from accessing the current sleep loop, and have it call MY input function synchronously, when it wants input.
This would also serve as a flag that the simulation was at a break.
I will write an example, and see if I can get you and WindowsNT to agree to it.

Didn't catch this ...

Errors I have seen so far in the bochs code: [skipped]

BTW, your fixes for watchpoints and compilation are already committed to Bochs CVS

And if I had lots of time, I'd completely rewrite your param tree interface for you. If you put everything into the param tree, the way it is built right now, bochs is going to be one of the slowest emulators ever seen.

The param tree currently is NOT used where the performance is anyhow important. It is used for configuration interface, for saving/restoring of the simulation state and for debugger. All these usages are considered NOT performance critical. The emulation is never accessing param tree, it just don't need to do it !
So it was designed for flexibility and not for speed. But if you going to make some intensive usages of param tree in debugger interface - it could be very good idea to improve the param tree as well.

Stanislav

[stlw]

Bochs is an emulator, not a "Virtualizer", it's also very portable.. runs on a wide range of architectures and operating systems.

If you want something open source, decent.. and fast.. try QEMU.

But be aware that QEMU CPU emulation is really far far away from how real Intel or AMD CPUs behave.

Stanislav

[bewing]

Yes, I mostly agree with that. The CVS thing is mostly a problem because
1) I don't have it installed, and have never used it.
2) Most of the users here do not use the latest CVS snapshot of bochs -- they use the public releases. So if I create a patch for the CVS version, nobody here can use it.
3) A patch for a CVS snapshot is always obsolete in a week, so there is almost no point in it.

So, if I create patches for the CVS version, they will only be for you, and I should send them to you directly.

Bochs running every CPU one after another, quantum of instructions on every logical CPU and when advance a timers in one shot after all CPUs are done with the quantum.

Yes, and this is exactly what I would like to fix next. I understand that you will not want to integrate it immediately into bochs. If I make the fix now, and you integrate it into bochs 6 months from now, that is OK with me.

I would certainly like to get some testing done on the current frontend that I've built, on many platforms -- but my test request here on osdev isn't getting much feedback. I hope that is a good sign that nobody is having problems, but who knows!

But that does not mean that I can't be writing new things, for future release, while people are testing what I've done so far.

No, the problem is bigger. Bochs CPU and devices are single-threaded and not thread-safe !

I understand that they are not, currently. However -- I don't see any reason why CPU is not, since all the simulated cpus are already independent, and instruction simulations all seem to be stateless. And the devices all need locks. The problem is the way that time is simulated, as I see it.

[stlw]

Yes, I mostly agree with that. The CVS thing is mostly a problem because
1) I don't have it installed, and have never used it.
2) Most of the users here do not use the latest CVS snapshot of bochs -- they use the public releases. So if I create a patch for the CVS version, nobody here can use it.

So, if I create patches for the CVS version, they will only be for you, and I should send them to you directly.

No problem. I will get them directly
You just should understand that Bochs CVS is moving and if your patch grows above some specific threshold it will be just impossible to integrate it.
We already had such a situation with a few patches before.
For example WinCE port patch.

Bochs running every CPU one after another, quantum of instructions on every logical CPU and when advance a timers in one shot after all CPUs are done with the quantum.

Yes, and this is exactly what I would like to fix next. I understand that you will not want to integrate it immediately into bochs. If I make the fix now, and you integrate it into bochs 6 months from now, that is OK with me.

Would be nice to know how. Do you have any ideas you could share ?

I would certainly like to get some testing done on the current frontend that I've built, on many platforms -- but my test request here on osdev isn't getting much feedback. I hope that is a good sign that nobody is having problems, but who knows!

But that does not mean that I can't be writing new things, for future release, while people are testing what I've done so far.

This is still have some consequences. You could maintain few separate threads of development at same time but if you can't hold them down and converge - none of them will be integrated into Bochs eventualy.

No, the problem is bigger. Bochs CPU and devices are single-threaded and not thread-safe !

I understand that they are not, currently. However -- I don't see any reason why CPU is not, since all the simulated cpus are already independent, and instruction simulations all seem to be stateless. And the devices all need locks. The problem is the way that time is simulated, as I see it

You forgot something very big. The shared memory. All CPUs and devices run with the same memory object. In the real hardware locked RMW memory acesses are fully atomic, in Bochs they NOT. You can't assume SMP will work with non-atomic locks. I am already not talking about APIC and devices.
You actually could take a look on to para-bochs project to see the challenge:
http://grid.hust.edu.cn/cluster/Virtual ... /main.html

Stanislav

[bewing]

Would be nice to know how. Do you have any ideas you could share ?

Any function/module/method that needs to be atomic, and cannot be made stateless, needs a lock. It seems quite simple to me. Since we are working with a maximum of 8 concurrent threads/CPUs, the lock will probably be quite simple. As far as I know, the current memory routines are also basically stateless. Yes, two cpus can read or write the memory at the same time, in multi-threaded cpu_loops. That is exactly how a multi-CPU real-world machine works, too.