New BCOS release

[smiddy]

Holy Tomatoes (replace Tomatoes with your favorite explitive) Batman!

Well, apparently I haven't read much from your site...I shall. That is an exciting set of requirements for sure. The virtual file system will be a nice touch. A simple thought comes to mind, backups...

I have to run...and you need some sleep!

[Brendan]

Hi,

Well, apparently I haven't read much from your site...I shall. That is an exciting set of requirements for sure. The virtual file system will be a nice touch. A simple thought comes to mind, backups...

It's a little more ambitious than typical distributed OSs as it will use peer-to-peer methods rather than the client/server approach, such that at any time any computer can be turned off. This requires that all native file systems be super-imposed upon each other and that any files may disappear at any time.

Typically files are "cached" on local disks when used and the VFS tries to keep as many redundant copies as storage space allows. The file system also uses versioning such that if the latest version of a file is not available an older version may be.

One side effect of this is that it can take a relatively long time to open a file (as each computer may need to be asked which versions of it, if any, it is holding). The VFS layer caches as much as it can to minimize this problem. Memory mapped files may also need to be stored locally (or in the VFS cache) before the mapping can work, nullifying the advantages in some cases.

Then there's things like IPC and shared data, where it cannot be guaranteed that different processes are running on the same computer. This makes things like aliased pages (where the same physical pages of RAM are shared by multiple processes) prohibitively impractical. It also explains my instance on non-blocking IPC for all message based interfaces (which makes something simple, like the standard C "fopen()" function very difficult to implement).

As if that's not ambitious enough, once it all does work there are plans to extend it to other architectures so that a cluster can consist of many different computers. The main problem here is the format of messages, which is easy to fix - on little-endian machines messages are stored from lower addresses to higher addresses within a message buffer, while on big-endian machines the message is "upside-down" with the messages stored from higher addresses to lower addresses. This means different fields within a message are at offsets from the start of the message, where these offsets may be positive or negative depending on endian-ness. On the network messages are sent from start to end, byte by byte, and this "upside-down" thing reverses the endian-ness when a computer receives a message.

The web site itself is currently understated, concentrating on what it does do rather than what it will do, with only subtle clues to what is planned. Considering that I am unable to conceive of a more ambitious project, I hope that by understating things on the web site I can minimize the amount of embarrassment when none of it ever works ;D...


Cheers,

Brendan

[smiddy]

Greets Brendan,

Very nice indeed!

System Specs
smiddy1
AMD Athlon XP 3000+ (2.17 GHz)
MSI KT4V Motherboard
640 MB RAM (333 MHz)
80 GB ATA HDD ST380013A
CD-ROM RW LITE-ON LTR-40125S
NVIDIA GeForce4 Ti 4200 with AGP8X
1.44 FDD
PS/2 Mouse
MS Natural PS/2 Keyboard
RealTek 8139A NIC
CanoScan FB620U #2 (on USB)
VIA Compatable Fast Ethernet Adapter
2 x Serial Ports
1 x Parallel Port
RealTek AC'97 Audio for VIA
Texas Instruments SilverLink (on USB)
Canon S9000 Photoprinter (on USB)
APC UPS (on USB)
4 x USB hubs

smiddy2
AMD Athlon 1900+
AMD Athlon XP 3000+ (2.17 GHz)
ASUS A7A266 Motherboard
1024 MB RAM (266 MHz)
3 x 80 GB ATA HDD ST380013A
CD-ROM RW BTC BCE16101M
NVIDIA GeForce2
1.44 FDD
PS/2 Mouse
MS Natural PS/2 Keyboard
RealTek 8139A NIC
2 x Serial Ports
1 x Parallel Port
RealTek AC'97 Audio for VIA
APC UPS (on USB)
4 x USB hubs

Mrs_smiddy1
HP zd7000 Media Center Laptop
3.0 GHz Intel Extreme with HT
1024 MB RAM (800 MHz)
80 GB HDD
DVD/CD RW
4 USB Hubs
NO FLOPPY
...plus...

Test Results
smiddy1
CD-ISO: booted fine and ran 132 x 60 lines, picked up all devices and determined correct bus and cpu speeds. Stops after last device, no exiting...<I assume this is correct>
Floppy Image: booted similarly, but didn't make it to full PCI device check, hung on ISA Plug and Play Device Scan
smiddy2
CD-ISO: wouldn't boot...reran on smiddy1 to determine if copy was ok, and it did boot.
Floppy Image: Hangs after ISA Plug and Play Device Scan <no way to escape>
Mrs_smiddy
CD-ISO: wouldn't boot either.
Floppy Image: , no floppy drive, no worky.

I really like the text modes. I initially was a little confused on the startup screen, but after playing with it figured it out. Oh, on smiddy1, I played with the time and I think that it changed the OS time to 12 hours ahead of current time, even though I chose my location, however the difference between UTC and local (the offset) was correct. Uhm, initially when I tested the 132 x 60 (or any of the text modes) I was afraid the system died because of the display, which was unxpected. However, in retrospec it is perfect for testing the text mode as it allows the tester the ability to see everything fairly well.

I have no idea why the CD-ISO works on smiddy1 and not the others. A side note, loading the CD-ISO into the drive under Window XP, the explorer sees what is there but I can not read the readme file or explore the folders. BTW, if I write my smiddyOS floppy image to CD, it does boot on the other two machines.

I didn't try the DOS version, which I may do this coming weekend if I get the opportunity.

I think that is it, I'm tired so I am heading to bed. Good job! I am eager to hear/see how you'll do USB scanning. I haven't as yet read anything on USB. That will come after I've gotten my device manager in place for loading and unloading driver for the motherboard and PCI devices.

[Brendan]

Hi,

Thanks for the tests!

I've found a Compaq Proliant that also locks up during *something*. It was locking up during the ISA Plug & Play detection code, so I disabled that and it locked up during the serial port detection, so I disabled that and it locked up during the PS/2 device detection. I've been trying to find the cause for 13 hours now, and so far all I've found out is that IRQ8 (the RTC timer) stops working once the kernel's idle thread starts (on this computer only), but IRQ8 shouldn't really effect anything and the scheduler (PIT or local APIC timer) still works fine.

I also have no idea why the CD-ISO works on smiddy1 and not the others (it uses "floppy emulation", which I thought would either work or not). It also confuses me that a boot floppy behaves differently to a CD emulating a boot floppy - in all cases the "boot image" is loaded into RAM and started, and everything *should* be the same after that :-[.

Windows XP not reading the installation CD is less of a concern for me at the moment. I haven't tried to read from the installation CD (other than booting) for quite a while, so it could be broken. I've never tried to read the CD from Windows XP either (Win98, Win95, Redhat and Gentoo only) so I might also be doing something WinXP doesn't like. I've put it on my "todo" list (to be investigated once that Compaq doesn't lock up).

During the 13 hour "bugfest" I've found another wierd bug that I introduced after the last release, and think I've found a problem with the APICs in Bochs ("send to lowest priority CPU" sends to all CPUs). I also tried your OS (SmiddyOS) on 4 computers here, and was intending to do more testing and write a full report (and still will). Briefly, SmiddyOS also locks up on the same Compaq I'm having trouble with but works perfectly on the other 3. I've also got questions about how you're detecting those PCI devices in Bochs without "i440fxsupport" enabled - as far as I can tell none of these devices exist in the Bochs source code, and the Bochs PCI BIOS code checks the vendor ID and device ID of the host controller and returns "unsupported function" if it doesn't match the I440FX Intel PCI host controller (for all PCI BIOS functions).

For the RTC clock, my OS sets it to UTC (not local time) and also puts it into binary mode rather than BCD mode. The boot menu uses 24 hour format only (what we call "military time"), so for 10:43 pm you'd need to enter 22:43, and for 12:00 am you'd need to enter 0:00.

Anyway, more debugging (and re-bugging) tomorrow .


Thanks,

Brendan

[smiddy]

I'll see what the specifics are on the differences on Bochs this evening if I get the opportunity (this is a big IF).

DennisCGc's system also locks up with my code when I go into unreal mode (for loading files into upper memory). It dies just after changing the CR0. This is something I need to look at more closely myself. Where on boot up does mine die on the Compaq?

I am currently not using the RTC, though I plan to shortly. I am however using the PIT, set at 1kHz.

When I am able I'll run you through my detection scheme and see if we can hash out the differences and perhaps colaborate in making them better. But I may need until next weekend before I can devote more time to it.

[smiddy]

The following is the post I tried on your site:


Nice board. I wanted to post the two different screen shots of with i440FX support and without, here instead of on Mega-Tokyo. With:

Code: Select all

smiddy's OS (C) 2004, 2005. All Rights Reserved.
Version 1.00.00.000000A7
CPU: CPUID Instruction: GenuineIntel Pentium III (Coppermine)
A20 Gate: A20 gate enabled...
E820: 1,073,741,824 bytes.
E801: 1,073,741,824 bytes.
88xx: 16,777,216 bytes.
CMOS: 67,108,864 bytes.
Entering UnReal Mode...
GDT Loaded...
Interrupts are back on...
Probe: 1,073,741,824 bytes.
Mapping Memory 1: 1,073,741,824 installed: E820:
Mapping Memory 2: Bitmap Initialized at 14MB: 1,056,964,608 bytes free for use.
Looking for VENDORS.TXT in root...Found! Loading VENDORS.TXT 198,196 bytes.
BIOS32 Entry Found at: 000F9D90h Revision: 0
PCI32 BIOS Found at: 000F9DD0h
PCIIRQ Table found: 000FA030h  Slots: 6 Bus: 0 Device: 1 Function: 0
PCI IRQ Slot: 0 Bus: 0 Device: 1 INT A#: 60h
PCI IRQ Slot: 1 Bus: 0 Device: 2 INT A#: 61h
PCI IRQ Slot: 2 Bus: 0 Device: 3 INT A#: 62h
PCI IRQ Slot: 3 Bus: 0 Device: 4 INT A#: 63h
PCI IRQ Slot: 4 Bus: 0 Device: 5 INT A#: 60h
PCI IRQ Slot: 5 Bus: 0 Device: 6 INT A#: 61h
About to cook VENDORS.TXT file for use in enumeration.
The first pass completed:
This file was created on: Tue Jun 14 19:07:13 PDT 2005
Creating database of 891,224 bytes: VendorID, DeviceID, VendorString, DeviceString.
Reducing required information size to items only currently installed.
8086 1237 Intel Corporation - 82440LX/EX PCI & Memory
8086 7000 Intel Corporation - 82371SB PIIX3 PCI-to-ISA Bridge (Triton II)
8086 7010 Intel Corporation - 82371SB PIIX3 IDE Interface (Triton II)
Start     End       Size (bytes)  Type Owner/Description
--------- --------- ------------- ---  -----------------
00000000h 000003FFh         1,024 02h  Interrupt Vector Table (IVT)
00000400h 000004FFh           256 02h  BIOS Data Area (BDA)
00000500h 000005FFh           256 02h  smiddyOS Data Area (OSDA)
00000600h 0009FBFFh       652,800 02h  smiddyOS code
0009FC00h 0009FFFFh         1,024 02h  Extended BIOS Data Area (EBDA)
000A0000h 000BFFFFh       131,072 02h  VGA Buffer Area
000C0000h 000FFFFFh       262,144 02h  ROM/BIOS Area
00100000h 00FFFFFFh    15,728,640 02h  smiddyOS code
01000000h 010000BFh           192 02h  PCI Vendors and Devices Information (Reduced)
010000C0h 010307FFh       198,464 01h  <Free Useable Memory>
01030800h 0110A157h       891,224 02h  PCI Vendors and Devices Information (Cooking)
0110A158h 3FFFFFFFh 1,055,874,728 01h  <Free Useable Memory>
40000000h FFFFFFFFh 3,221,225,472 02h  Unuseable addressable memory area
smiddyOS Clock Initializing...Done!
Boot up time> 17:57:59 System Time> 17:57:05 Current Time> 17:57:25

See next post..

[smiddy]

Part II

Without:

Code: Select all

smiddy's OS (C) 2004, 2005. All Rights Reserved.
Version 1.00.00.000000A7
CPU: CPUID Instruction: GenuineIntel Pentium III (Coppermine)
A20 Gate: A20 gate enabled...
E820: 1,073,741,824 bytes.
E801: 1,073,741,824 bytes.
88xx: 16,777,216 bytes.
CMOS: 67,108,864 bytes.
Entering UnReal Mode...
GDT Loaded...
Interrupts are back on...
Probe: 1,073,741,824 bytes.
Mapping Memory 1: 1,073,741,824 installed: E820:
Mapping Memory 2: Bitmap Initialized at 14MB: 1,056,964,608 bytes free for use.
Looking for VENDORS.TXT in root...Found! Loading VENDORS.TXT 198,196 bytes.
BIOS32 Entry Found at: 000F9D90h Revision: 0
PCI32 BIOS Service is not present.
PCIIRQ Table found: 000FA030h  Slots: 6 Bus: 0 Device: 1 Function: 0
PCI IRQ Slot: 0 Bus: 0 Device: 1 INT A#: 60h
PCI IRQ Slot: 1 Bus: 0 Device: 2 INT A#: 61h
PCI IRQ Slot: 2 Bus: 0 Device: 3 INT A#: 62h
PCI IRQ Slot: 3 Bus: 0 Device: 4 INT A#: 63h
PCI IRQ Slot: 4 Bus: 0 Device: 5 INT A#: 60h
PCI IRQ Slot: 5 Bus: 0 Device: 6 INT A#: 61h
About to cook VENDORS.TXT file for use in enumeration.
The first pass completed:
This file was created on: Tue Jun 14 19:07:13 PDT 2005
Creating database of 891,224 bytes: VendorID, DeviceID, VendorString, DeviceString.
Reducing required information size to items only currently installed.
003D 00D1 Lockheed Martin Corp - i740 PCI
0E11 0001 Compaq Computer Corp. - PCI to EISA Bridge
0E11 0002 Compaq Computer Corp. - ISA Bridge
0E11 000F Compaq Computer Corp. - CPQB1A9 StorageWorks Library Adapter (HVD)
0E11 0012 Compaq Computer Corp. - 686P7 686P7
0E11 0049 Compaq Computer Corp. - NC7132 Gigabit Upgrade Module
0E11 004A Compaq Computer Corp. - NC6136 Gigabit Server Adapter
0E11 00C0 Compaq Computer Corp. - Adaptec AIC-7899G 64Bit,66MHz,Dual Channel WideUltra3 SC
SI
0E11 0508 Compaq Computer Corp. - Neteligent 4/16 TR PCI UTP/STP Controller
Start     End       Size (bytes)  Type Owner/Description
--------- --------- ------------- ---  -----------------
00000000h 000003FFh         1,024 02h  Interrupt Vector Table (IVT)
00000400h 000004FFh           256 02h  BIOS Data Area (BDA)
00000500h 000005FFh           256 02h  smiddyOS Data Area (OSDA)
00000600h 0009FBFFh       652,800 02h  smiddyOS code
0009FC00h 0009FFFFh         1,024 02h  Extended BIOS Data Area (EBDA)
000A0000h 000BFFFFh       131,072 02h  VGA Buffer Area
000C0000h 000FFFFFh       262,144 02h  ROM/BIOS Area
00100000h 00FFFFFFh    15,728,640 02h  smiddyOS code
01000000h 010002EAh           747 02h  PCI Vendors and Devices Information (Reduced)
010002EBh 010307FFh       197,909 01h  <Free Useable Memory>
01030800h 0110A157h       891,224 02h  PCI Vendors and Devices Information (Cooking)
0110A158h 3FFFFFFFh 1,055,874,728 01h  <Free Useable Memory>
40000000h FFFFFFFFh 3,221,225,472 02h  Unuseable addressable memory area
smiddyOS Clock Initializing...Done!
Boot up time> 18:01:49 System Time> 18:01:53 Current Time> 18:02:00

One thing the PCI32 doesn't do is allow you to do a class/sub-class check like on a normal PC. However the devices do present themselves when you scan by Device ID and Vendor ID. As for your question about not seeing it within the code, well, I may not be using the same verion you are, possibly. I'm using 2.2 release.

BTW, where would you prefer I post my info, here, or Mega-Tokyo?

[bubach]

"here, or Mega-Tokyo?"

lol, look at the site banner, at the page top...

[smiddy]

"here, or Mega-Tokyo?"

lol, look at the site banner, at the page top...

Yeah, I'ma dork! Actually I wrote that on Brendan's site, then when it failed due to being to large, I cut and pasted it here, which subsequently was too large for here too, so I made a second post. In the process, I forgot to re-read it for errors.

[distantvoices]

@brendan:
have tested BCOS in QEMU emulation:

boots up fine (cool boot menu).

It detects all qemu devices quite fine, althou taking its time for doing the job. That's most prolly due to the calculations you have mentioned in this thread.

What is this "user interface module" supposed to do? It might be a cool gag to have it spill out a table of threads upon pressing some key (say "t" or so ...).

Except of the slowness, there 's nothing special happened. Is some final "init done" message expected to show up?

[Brendan]

Hi,

I think I've made some progress!

The first bug was caused by a mis-understanding of how the local APIC "task priority register" (or TPR) behaves - it actually blocks IRQs from being received, rather than just determining the processor's priority during arbitration for "send to lowest priority" IRQs.

Some of my re-entrancy locks cause received IRQs to be placed into a FILO queue and handled when the lock is released. My OS uses the local APIC TPR so that IRQs are sent to a CPU that isn't within one of these locks if possible, but instead disabled the IRQs in this case. This would have caused IRQs to be lost on any multi-CPU computer if all CPUs are within these locks.

I still need to play with the TPR a bit - I want to add code so that on multi-CPU computers TPR depends on the priority of the currently running thread and any re-entrancy locks the code has acquired.

The second problem is that IRQ8 may have been set too fast - for my Dual 400 MHZ Pentium II the calculation resulted in a timer frequency of 4096 HZ, but testing reveals that it can't handle anything above 512 HZ. This took me ages to find because it's intermittent (it'd work fine 75% of the time). I'm still looking for places where interrupts may be disabled for too long, but will also rewrite this calculation so that it works on the speed of the slowest CPU rather than the combined total speed of all CPUs.

The third problem (re-entrancy lock errors within the memory manager for multi-CPU computers) I haven't looked at yet. This shouldn't be too hard to find & fix though .

I think half the reason these errors went unnoticed before the release is that my code doesn't really get a thorough work out - if it's lucky enough to get past the device detection then it's worked. For this reason I'll refrain from posting an update until everything mentioned above is done, until I've got the PS/2 keyboard and mouse IRQs are connected to something (to further test the IRQ handling) and until there's some "busy loop" threads to give the scheduler much more work.


@beyond infinity

@brendan: does your kernel/os work with qemu? (for giving it a test - virtualPC refuses to take your fd-image). as for heavy and endless bug searching: The more it evolves, the more hidden bugs you will find. *gg* don't despair. They usually show up when you don't expect them at all, and as sudden you find solutions for such problems. *gg* Just go for a hike 'n a chat.

I gave up on VirtualPC - last time I tried it I got some generic error message that meant nothing, and searching for the error message on the internet brought up a long list of other OSs that caused the same generic message. I have a feeling it might be caused by code that probes for a local APIC when one hasn't already been found (I assume VirtualPC thinks I'm trying to access memory that doesn't exist and pukes - hard to tell without a useful error message). To make it worse I don't have VirtualPC here, and need to accidentally use the computers at Uni when no-one is looking.

Oh - and Virtual PC uses a strange format for floppy disk images, which isn't the same as anything else. To get it to boot I told VirtualPC to use the real floppy drive. I think there's a way to get VirtualPC to create an image from a real floppy, but can't find out at the moment. I'll take a look tomorrow morning .

What is this "user interface module" supposed to do? It might be a cool gag to have it spill out a table of threads upon pressing some key (say "t" or so ...).

Except of the slowness, there 's nothing special happened. Is some final "init done" message expected to show up?

Thanks for testing .

The "User Interface" module is an interface between other software and user-related devices (keyboard, mouse, video, sound, etc). Unfortunately there's no devices connected to it yet - no keyboard.

There also isn't any final "init done" message - the boot code starts several threads, and these threads do their work independantly (and when everything is done the scheduler runs the idle thread/s). The last thing you should see is the last device at the end of the detected device tree.


Cheers,

Brendan

[distantvoices]

re VirtualPC: I should have spoken more precisely: VirtualPC tells that the image isn't 1.44 mb big, so it canna be a floppy image - and as a result it refuses to load anything. Pretty stupid, me thinks, for qemu is able to deal with your image without problem. So, it's not a problem with your OS or floppy image but with the VirtualPC thing.

stay safe & happy debugging

(one ring to rule them, one ring to find them
and in the darkness bind them ... muuuaaahahahahahaaaa)*rofl*

[smiddy]

Brendan,

I was wondering if you found an answer to the ISA problems yet? I am about to look into PnP, but from a BIOS perspective in 32-bits, if possible and I as wondering if you had any words of wisdom before I head down that path.

FYI, I'm about to release a device driver interface document detailing how I'm inplmenting it on my own OS...I'm in concert, for the most part, with Dex4u and BOS, which is a variation on DOS' device drivers. Knowing PnP will play a significant role in my device driver manager. Any information you can provide would be greatly appreciated.

[Brendan]

Hi,

I was wondering if you found an answer to the ISA problems yet?

Yes (or at least I hope so). The code I used to calculate the frequency for IRQ8 was too aggressive - it set the frequency for IRQ8 too high, which resulted in lost IRQs. The first IRQ8 that wasn't handled left RTC register C unread, so the timer stopped generating further IRQs.

As soon as IRQ8 stopped being generated all the timing code that relied on IRQ8 stopped working. On faster computers IRQ8's timer is used for timing the little delays used by the ISA Plug & Play detection code, the PS/2 device detection code, and the serial port detection code.

The current verdict is that the code to calculate IRQ8 frequency wasn't implemented according to my OS timing spreadsheet and my timing spreadsheet may have been optimistic to begin with. In addition, it's possible that my kernel may be disabling IRQs for too long. Further analysis (starting with "grep cli" on a disassembly) revealed that the only place I disable IRQs for more than 8 instructions is within the scheduler's thread switch code. For a thread switch it disables IRQs, searches for the next thread to run then does a software thread switch.

The search itself isn't too good - it can be O(2*n) for 3 of the 4 scheduling policies due to the "variable frequency" scheduling method chosen. I've been considering changing 2 of the scheduling policies to "variable qTime" which can be done in O(1) but isn't as responsive. The code to do a thread switch is also a little heavy - I do a pile of time accountancy to track the number of mS used by the previous thread, pre-calculate some values used by the rest of the kernel, and set the timer (PIT or local APIC) count according to the new thread (I use "one-shot" mode rather than "periodic").

With this in mind I've been considering the possibility of doing the search for the next thread with interrupts enabled, so that interrupts are only disabled for the final thread switch itself (where CPU state is inconsistant). I haven't yet identified all of the re-entrancy locking issues that this would involve though.

Suffice to say, as far as I can tell the code that does the ISA Plug & Play detection was never faulty, but stopped due to the IRQ8 bug - it'd ask for a 2 mS delay that actually went forever.

I am about to look into PnP, but from a BIOS perspective in 32-bits, if possible and I as wondering if you had any words of wisdom before I head down that path.

For ISA Plug & Play I can provide the "direct IO" algorithm I use (derived from the Plug & Play standard), but I've never attempted to use the BIOS functions for this. To be honest I should look into the BIOSs functions (I haven't even looked at them) - for e.g. if the BIOS can reliably tell me if there is any ISA Plug & Play or not I can reduce my boot time by about 2.5 seconds on lots of computers by skipping the detection code. ISA Plug & Play devices are relatively rare - the amount of time between non-PNP ISA and PCI wasn't a lot.

FYI, I'm about to release a device driver interface document detailing how I'm inplmenting it on my own OS...I'm in concert, for the most part, with Dex4u and BOS, which is a variation on DOS' device drivers. Knowing PnP will play a significant role in my device driver manager. Any information you can provide would be greatly appreciated.

I'd like the chance to read through your device driver interface document - it'd be nice to see how others intend to do it .

For the most part, my method involves starting the detection sequence at the highest point (AGP/PCI) and working down towards non-plug and play devices while disabling everything I can on the way. Once this is done, the fixed resources for non-plug and play devices are allocated and then I'll work out resource conflicts from bottom up, leaving device disabled. Once this is all done I load device drivers, tell the device drivers which resources they need to use and enable the device (if it was disabled).

The idea behind disabling devices is that certain devices can be detected twice. For example, a PCI ATA controller would be detected as a PCI device and also detected as a non-plug & play ISA device later. Both PCI and ISA Plug & Play devices can be disabled, which prevents them from accessing their IO ports, etc (so the code used to detect a non-plug and play ISA devices can't find anything it shouldn't).


Cheers,

Brendan