Building 64-bit native subsystem PE executable with clang

[x64dev]

My OS second stage boot loader, kernel and drivers (which I haven't done anything for the last 3 years) is currently built using the windows driver kit and are 64-bit PE images. I did this for several reasons, at the time I started it, I was working on the Windows Kernel team at Microsoft and just wanted to try something in spare time. The compiler & assembler made it easy for me to write the necessary code and get it to compile and produce a bootable operating system (although still very limited in functionality).

I left about 4 years ago to do a PhD which I'm hoping to complete this year.

While I was permitted to use the ms tool chain for my hobby as an employee, I'm no longer allowed to use some of the tools I had access to and need replacements. One of these is the compiler and so I've been looking around to figure out how I can build the second stage boot loader and the kernel, drivers etc.

I have considered moving to ELF or another format, however, I've also written a kernel debug interface and a kernel debugger that connected across serial and was starting to read PDB files for debug info. I would prefer not to have to throw all this away. As I'm currently traveling I don't have access to my usual development systems to check if I can build a native subsystem no default lib 64-bit PE image either on Linux or perhaps on Windows - although I recall the Linux tool chain for windows seemed to want to link in a helper library to bridge the gap. I don't have a default lib because I also wrote my own Runtime library for the boot loader, kernel and boot/other drivers.

So does anyone know if this is possible with clang or I guess GCC (although I don't like GCC much)?

Thanks.

[zaval]

So you say you cannot use WDK, SDK, VS compiler for your purpose? Why? I genuinely am interested, because I thought it's possible. I need PE/COFF support too, and tbh I just think MS compilers are of better quality. but since my project first deals with arm/mips, I just hope to use what is available (Linaro toolchain for arm) and then translate ELFs into PE/COFFs. Just like edk guys do. They use both GCC and Clang and Arm compiler I guess. For arm, they don't support mips. But if you are talking about x64, why can't you use available tools from MS? Both free and commercial? Has it something to do with legal concerns (MS doesn't allow compiling other OSs and its components by their compiler?)
Or are you just kidding?

[x64dev]

Not kidding, if you read the license terms it says that the compiler etc. can only be used to build programs, drivers etc. for Windows.

[zaval]

Not kidding, if you read the license terms it says that the compiler etc. can only be used to build programs, drivers etc. for Windows.

But how Tianocore uses it for Uefi builds then?

[x64dev]

But how Tianocore uses it for Uefi builds then?

They likely have permission since this will also be used in booting Windows etc.

[zaval]

But how Tianocore uses it for Uefi builds then?

They likely have permission since this will also be used in booting Windows etc.

emmm, something is telling me, you worry in vain, but i'm not a lawyer so.

anyway, clang/llvm, it seems, doesn't support PDB yet. gcc supports PE/COFF. theoretically.

[dchapiesky]

The only reason to worry about legal stuff is if you plan to sell your OS...

If this is the case, then I would worry more about the legals related to using the Windows Driver Kit more.... pretty sure those would be restricted as well....

[x64dev]

The only reason to worry about legal stuff is if you plan to sell your OS...

If this is the case, then I would worry more about the legals related to using the Windows Driver Kit more.... pretty sure those would be restricted as well....

That's why I'm looking at clang or another compiler - since the WDK also has restrictions.

If I have to I'll change the first stage boot loader to make it load ELF and change the second stage boot loader, kernel, boot and other drivers to ELF and then write DWARF version 4/5 debug support into my boot/kernel debugger.

But was hoping to avoid that if possible

I don't have plans to sell the OS (currently - and may never do) but was considering building it for a university level course at some point. For that it would be best to have it be utilizing fully legal software.

You may wonder why I'd would do another OS, well, I spent 15 years working on Windows and there are a lot of things I like about it, but there are also things that I feel could be improved upon and which would be easier to improve in a new system.

[iansjack]

I think a more likely concern is that, having worked on the Windows kernel, you are in danger of using information gleaned there in your own kernel!.

[alexfru]

I think a more likely concern is that, having worked on the Windows kernel, you are in danger of using information gleaned there in your own kernel!.

Then one shouldn't probably read books like Inside Windows NT and Windows Internals either. Not to mention viewing videos like the one on fighting global locks in the kernel. Wait, did Linux steal the idea from there?

[x64dev]

I think a more likely concern is that, having worked on the Windows kernel, you are in danger of using information gleaned there in your own kernel!.

Then one shouldn't probably read books like Inside Windows NT and Windows Internals either. Not to mention viewing videos like the one on fighting global locks in the kernel. Wait, did Linux steal the idea from there?

Everything I would use has been openly documented anyway.

Heh, I remember that, I checked in a piece that removed another kernel Global kernel lock prior to Arun removing the dispatcher lock.

Been a while

[iansjack]

I suspect that those who have actually worked on the Windows kernel, and have had access to the source code for years, may know more about its workings than is revealed in those documents.

I doubt that either of these factors is a real concern for a personal project, but the possibility of inadvertently using Microsoft copyright material is probably greater than any danger inherent in using their compiler.

[MollenOS]

If you ever figure out how to use Clang to output PE files for bare targets, _please_ let me know. I don't know how much time I've spent trying to figure that out.

[x64dev]

If you ever figure out how to use Clang to output PE files for bare targets, _please_ let me know. I don't know how much time I've spent trying to figure that out.

Sure

I may end up implementing a backend to produce the PE if I need to. In theory any linker that produces PE images could do the final stage linking of the clang compiled object code and produce a PE native subsystem image.

[alexfru]

I may end up implementing a backend to produce the PE if I need to. In theory any linker that produces PE images could do the final stage linking of the clang compiled object code and produce a PE native subsystem image.

In theory, it's in there:

LLD is a linker from the LLVM project. That is a drop-in replacement for system linkers and runs much faster than them. It also provides features that are useful for toolchain developers.

The linker supports ELF (Unix), PE/COFF (Windows) and Mach-O (macOS) in descending order of completeness. Internally, LLD consists of three different linkers. The ELF port is the one that will be described in this document. The PE/COFF port is almost complete except the lack of the Windows debug info (PDB) support.