How long does it take to compile your OS?

[thepowersgang]

CmpXchg: LD can be compiled for windows, gcc does need a lot of arguments to be told to do a complete strip down. (maybe it should have a argument for a plain binary?)

Candy: That is a lot of storage. Is your OS a 64bit one? and 32768EB is a 65 bit address space, isn't that inaccessible on current hardware?

[Brendan]

Hi,

Candy: That is a lot of storage. Is your OS a 64bit one? and 32768EB is a 65 bit address space, isn't that inaccessible on current hardware?

File systems aren't limited to current hardware, or at least they *shouldn't* be (not forgetting that a file system might be stored on a RAID array consisting of a large number of separate hard drives)...

Note: with 512 byte sectors, 32768 EB works out to 56-bit sector numbers.


Cheers,

Brendan

[Osbios]

My OS need ~1 Sec at the first time and then ~0.1 Sec. Thats because all source files are cached in memory after the first time.

Asm Rulls!

[Candy]

Candy: That is a lot of storage. Is your OS a 64bit one? and 32768EB is a 65 bit address space, isn't that inaccessible on current hardware?

File systems aren't limited to current hardware, or at least they *shouldn't* be (not forgetting that a file system might be stored on a RAID array consisting of a large number of separate hard drives)...

Note: with 512 byte sectors, 32768 EB works out to 56-bit sector numbers.

I'm actually using 4k cluster indexes that are 64-bits each. An exabyte is 2^60 bytes though, so this is 2^75 (and yes, that's off by a factor of 2, it should be 65536EB instead. That's like, 64ZB or so?).

That's exactly the point. Your file system should not be limited to hardware that's likely or even possible to come out before you die yourself (imo). Given Moore's law, 64 bit indexes hold until I collect a pension. I was thinking about preemptively making them 128 bits instead for that reason.

[bewing]

I think it's trickier than Candy & Brendan say, about the filesystems. To be able to support a big filesystem generally requires a lot of overhead. So you really don't want to go too overboard -- or half of a 1GB disk will be used simply for filesystem structures. On my filesystem, 32 bit "cluster numbers" just happened to naturally map to 8 petabytes, with 512b sectors. By having a simple adjustment factor for the sector size, my typical "natural" max partition size ends up being around 500PB. But if I tried to push it any farther than that, my structures would all double in size. When the boundary starts getting pushed, I'll just created a "version 2" of my filesystem, that handles things differently. With a VFS system, I don't think that you need to be looking that far into the future. You can hook in a newer bigger FS later.

[jnc100]

Going back to the original question (I can only comment now because I've only just got the whole build system working), I get 15.3 seconds on a C2D T7100 in cygwin, although it must be one of the most complicated builds:

1. Compile the CIL to native code assembler
2. Compile the kernel to CIL
3. Assemble the kernel to native code
4. Link the kernel
5. Compress the kernel
6. Output a header file describing the layout of the managed classes
7. Compile bzip2
8. Compile the bootloader, incorporating bzip2 and the header file from (5)
9. Link the bootloader
10. Package all in a disk image

I think that's about it, compile log is attached if anyone's interested

Regards,
John.

[CmpXchg]

I think that's about it, compile log is attached if anyone's interested Wink

Ok, you seem to be using Cygwin and Microsoft .NET Framework.

So does your OS compile to native code? No interpreter required at runtime? I thought it was impossible with managed code.

And one more question. You seem to use 'mcopy' to make the disk image. Is it a built-in linux (cygwin) command? What is the purpose of the 'menu.cfg' file?

Wow, you have your own decompression routines!

[Korona]

My os takes about 4 minutes to compile as the compiler is written in an interpreted language (with an interpreter written in java). I could reduce this compilation time if I would add ELF output to my linker and then use my virtual machine (running under Linux) to compile my os. In the near future my os will be able to run the compiler on it's jit compiler and that will greatly reduce my build times. (Even now it could run my compiler but I have to improve my ext2 support and fix some memory leaks before the system is really suitable for this task.)
The build process consists of:
-> Build some drivers, init scrips and applications
-> Compile the just-in-time compiler to x86 code
-> Build a initrd that contains all drivers and the jit compiler
-> Assemble some runtime functions
-> Compile the kernel to x86 code
-> Assemble the kernel runtime
-> Link everything together
-> Build an image file that contains the kernel and the initrd

Here is my make log (The JScriptClient just sends a request to a daemon that executes the command (the daemon is running with java -server for better performance)):

Code: Select all

# build the initrd
java -cp ../JManaCc:../JManaVaLib:../JManaCcLib Korona -o Build/Initrd/VgaTextMode.code ../Drivers/x86/VgaTextMode/Driver.kor
java -cp ../JManaCc:../JManaVaLib:../JManaCcLib Korona -o Build/Initrd/Keyboard.code ../Drivers/x86/Keyboard/Driver.kor
java -cp ../JManaCc:../JManaVaLib:../JManaCcLib Korona -o Build/Initrd/Ata.code ../Drivers/x86/Ata/Driver.kor
java -cp ../JManaCc:../JManaVaLib:../JManaCcLib Korona -o Build/Initrd/Mash.code ../Mash/Main.kor
java -cp ../JManaCc:../JManaVaLib:../JManaCcLib Korona -o Build/Initrd/InitScript.code ../Drivers/x86/Init/InitScript.kor
java -cp ../JManaCc:../JManaVaLib:../JManaCcLib Korona -def VM_WITHOUT_COLLECTOR -o Build/InitMain.code ../Drivers/Init/Main.kor
java -cp ../JScriptClient Main localhost 57391 "../ManaBinAssembler/Main.kor -o Build/Initrd/Runtime.o -i ../ManaVm/Runtime/x86/Collection.x86.mbasm"
Request executed.
java -cp ../JScriptClient Main localhost 57391 "../ManaVm/AotBinMachine.kors -o Build/Initrd/Init.o -s Build/InitMain.code -c Managarm.Drivers.Init"
Request executed.
java -cp ../JInitrdTool InitrdTool -o Binary/Initrd.bin \
		-i Build/Initrd/Init.o \
		-i Build/Initrd/Runtime.o \
		-i Build/Initrd/InitScript.code \
		-i Build/Initrd/VgaTextMode.code \
		-i Build/Initrd/Keyboard.code \
		-i Build/Initrd/Ata.code \
		-i Build/Initrd/Mash.code
# compile the kernel to manacode
java -cp ../JManaCc:../JManaVaLib:../JManaCcLib Korona -def VM_WITHOUT_COLLECTOR -o Build/KrnMain.code Source/Kernel/Main.kor
# compile the kernel to x86 asm
java -cp ../JScriptClient Main localhost 57391 "../ManaVm/AotBinMachine.kors -o Build/KrnMain.o -s Build/KrnMain.code -c ManaCore.Kernel -c Korona.VmCollection"
Request executed.
# assemble the core
java -cp ../JScriptClient Main localhost 57391 "../ManaBinAssembler/Main.kor -o Build/RtStartup.o -i Source/Runtime/Startup.x86.mbasm"
Request executed.
java -cp ../JScriptClient Main localhost 57391 "../ManaBinAssembler/Main.kor -o Build/RtPaging.o -i Source/Runtime/Paging.x86.mbasm"
Request executed.
java -cp ../JScriptClient Main localhost 57391 "../ManaBinAssembler/Main.kor -o Build/RtGdt.o -i Source/Runtime/Gdt.x86.mbasm"
Request executed.
java -cp ../JScriptClient Main localhost 57391 "../ManaBinAssembler/Main.kor -o Build/RtIdt.o -i Source/Runtime/Idt.x86.mbasm"
Request executed.
java -cp ../JScriptClient Main localhost 57391 "../ManaBinAssembler/Main.kor -o Build/RtIo.o -i Source/Runtime/Io.x86.mbasm"
Request executed.
java -cp ../JScriptClient Main localhost 57391 "../ManaBinAssembler/Main.kor -o Build/RtThreading.o -i Source/Runtime/Threading.x86.mbasm"
Request executed.
java -cp ../JScriptClient Main localhost 57391 "../ManaBinAssembler/Main.kor -o Build/RtLoader.o -i Source/Runtime/Loader.x86.mbasm"
Request executed.
java -cp ../JScriptClient Main localhost 57391 "../ManaBinAssembler/Main.kor -o Build/RtUtils.o -i Source/Runtime/Utils.x86.mbasm"
Request executed.
java -cp ../JScriptClient Main localhost 57391 "../ManaBinAssembler/Main.kor -o Build/RtDebug.o -i Source/Runtime/Debug.x86.mbasm"
Request executed.
# assemble the runtime
java -cp ../JScriptClient Main localhost 57391 "../ManaBinAssembler/Main.kor -o Build/VmReflection.o -i ../ManaVm/Runtime/x86/Reflection.x86.mbasm"
Request executed.
# link them together
java -cp ../JScriptClient Main localhost 57391 "../ManaBinLinker/Main.kor -o Binary/Core.bin -object Build/RtStartup.o -object Build/RtPaging.o -object Build/RtGdt.o -object Build/RtIdt.o -object Build/RtIo.o -object Build/RtThreading.o -object Build/RtLoader.o -object Build/RtUtils.o -object Build/RtDebug.o -object Build/VmReflection.o -object Build/KrnMain.o"
Request executed.
rm -f Binary/Core.bin.gz
gzip Binary/Core.bin
sudo mount -o loop -t ext2 Binary/Floppy.img Mount
sudo cp Binary/Core.bin.gz Mount/managarm/Core.bin.gz
sudo umount Mount

[jnc100]

So does your OS compile to native code? No interpreter required at runtime?

I use the Microsoft tools (and hopefully soon mono) to compile my kernel to CIL packaged within a PE exe file (i.e. an assembly). Then, I use my own program to convert that PE-CIL file to an native x86_64 ELF file. This program is again written in C#, so I need to compile it first (to PE-CIL) as part of the build.

And one more question. You seem to use 'mcopy' to make the disk image. Is it a built-in linux (cygwin) command?

Its part of Mtools which is part of most linux distributions. It wasn't in the cygwin distribution last time I checked, I had to compile it myself under cygwin. I have binaries here.

What is the purpose of the 'menu.cfg' file?

Its for Grub. Grub loads my second-stage bootloader with the kernel as a module. The second stage bootloader sets up long mode.

Wow, you have your own decompression routines!

Not really, its just the core bzip2 library compiled with -DBZ_NO_STDIO linked into my bootloader, its quite simple to incorporate actually.

@Korona: Nice. Do you spend as much time swearing at Sun as I do at Microsoft?

Regards,
John.