CompilerDev?

All off topic discussions go here. Everything from the funny thing your cat did to your favorite tv shows. Non-programming computer questions are ok too.
User avatar
~
Member
Member
Posts: 1226
Joined: Tue Mar 06, 2007 11:17 am
Libera.chat IRC: ArcheFire

Re: CompilerDev?

Post by ~ »

bwat wrote:I think it's the other way around - assuming I've understood you correctly. The generic stuff, w.r.t., compilers and OS stuff, has been done to death, but not on internet fora, instead it's in books. OS design, compiler design, and abstract machine design have been very well described by people who know how to teach. If you want book suggestions just ask.
Yes, suggesting books, tutorials and other resources would be great. And this is one of the first things that would liven up a compiler subforum here or anywhere else. I use to skim through books and there are always details that catch my attention, and then I decide what would be best to start with.
bwat wrote:What's missing, what's hard about it all, is the nitty gritty issues like:
  • How do I write a high level debugger for this language?
  • How do I write a control-C handler that throws an exception in this language so the user can break out of an infinite loop in his/her code and jump to their own handler?
  • How do I implement, test, and document a logarithm function?
  • How do I test my garbage collector?
  • How do I best let the user define their own syntax extensions to the language?
These things are still part of the black art of programming language implementation. The books that describe this are often old, obscure, and difficult for the hobbyist to get their hands on. A good forum will help with this.

W.r.t. comp.compilers, I've only had one conversation about that group with another professional compiler writer and we both had bad things to say about the moderator. It was quite funny actually.
To this list I would add, for each language, like C, talking about all of the conventions, both in general and for x86 (which is the most common by now). Not just calling convention, but things like where things will end up when compiled, their exact structure, and independent of a specific toolchain, relying only in the specifications, in a resumed way, and ways to make the outcome of the compiled programs for languages like C, as analogously transparent and simple as their Assembly counterparts (matching chunks in the binary with elements in the code -structures, variables, padding, etc.-). Besides, it would make much easier for anyone to learn that language, since everything would be much clearer, from start to end, including obvious bugs in the structure of the data and packing, etc., just as with Assembly programs, their code, and a disassembler.


Combuster wrote:
dozniak wrote:It's not easy to set up a community, but if chase and mods are not against it, we could have a corner of this forum dedicated to that.
$.02 (and to get it back on track):

What I see with many people starting off on OS development, is that they can use a compiler from the instruction template they have always used - and for most desktop things you wouldn't need more than that. The result is that nobody knows how a toolchain really works until they get slapped hard by that mistake.
What happens next is that people will start to build runtime linkers - and at that point you will really need to know how the compiler outputs its stuff, and you will have to perform the necessary logic. Besides the task-specific linking, many system parts will end up having configuration files that will need to be read and parsed.
In other words, any sufficiently advanced OS has all the characteristic components of a (often simple) compiler, and depends quite a bit on the remaining parts of the compiler which the developer didn't need to write. In reverse, an actual compiler writer will need to be aware of how the OS deals with binaries without actually having to write that code.

So to include compiler development in this place would not be a far stretch and would probably have a symbiotic relation and a number of people migrating topics for fun and profit as a result.

----

Just as important is the follow-up question: would it take off if we tried? That one concerns me more actually.
Yes, it would in time. But, like in any other issue, it would require people who know about the subject to make it take off by adding material in a way that is simple to understand, before somebody asks (as seems to be the case with the Wiki too), and increasingly complete, avoiding questions that are far too specific to a toolchain/software piece/problem (these could be general-programming-related or whatever, unless they are truly relevant) as these questions would erode the overall interest on the really important topics.

That requirement is only known to the people who can start by adding the relevant starting points, so they are the ones who can figure that out. I for one have questions to ask and some little details to teach from time to time (I can do that in any of the sub-forums that already exist or in new ones), and that goes for many other areas (and they all fit in one way or another, at least for the users of any OS, to make it more interesting and serve as real-world tests for the capabilities of a kernel, to make something useful/educative with them).
YouTube:
http://youtube.com/@AltComp126

My x86 emulator/kernel project and software tools/documentation:
http://master.dl.sourceforge.net/projec ... 7z?viasf=1
Zondartul
Posts: 7
Joined: Sat Aug 29, 2009 3:33 pm

Re: CompilerDev?

Post by Zondartul »

I'm no expert but I could make a few short wiki entries on what I've learned so far about compiler-making and then we could expand them as we go. Is that okay?

Edit: How's this then? http://wiki.osdev.org/Making_a_Compiler

I think I got most of it right but please tell if I made any gross factual errors.
User avatar
JackScott
Member
Member
Posts: 1031
Joined: Thu Dec 21, 2006 3:03 am
Location: Hobart, Australia
Contact:

Re: CompilerDev?

Post by JackScott »

I only had a brief look through that page, and the gist of it looks ok... but with one glaring problem: I don't think Microsoft would be amazingly happy that we're using their icon files.

As far as having a compiler subforum on this site, I fully support that move, for much the same reasons as Combuster put forward.
User avatar
bwat
Member
Member
Posts: 359
Joined: Fri Jul 03, 2009 6:21 am

Re: CompilerDev?

Post by bwat »

Zondartul wrote:I think I got most of it right but please tell if I made any gross factual errors.
You did a good job there. I was pleasantly surprised. I would only make the following quite trivial suggestions:

1) What about a caveat at the beginning stating this is a description assuming the reader is an OS developer interested in the usual OS development languages. Lots of compilers have abstract machines or even other languages as their target, and some languages have peculiar parsing specifications. After stating this assumption in the beginning you wont have to point out that some compilers or some languages are different to what you're currently explaining.

2) At an even higher level, you could describe the compiler in two phases: analysis and synthesis. Then break down these two phases into your three: front, middle, and back. I've always found the analysis/synthesis model pleasing.

3) If you're going to describe the phases in detail, then how about writing a compiler for a very restricted subset of C as a running example? Yes I know this is quite alot to ask! But, if you think its a good idea then I can help you out and there's no doubt others who would like to help as well. What about taking the original small-c http://www.cpm.z80.de/small_c/smc88dos.zip rewriting it for a modern C compiler and in line with your wiki text.

Again, well done.
Every universe of discourse has its logical structure --- S. K. Langer.
kutkloon7
Member
Member
Posts: 98
Joined: Fri Jan 04, 2013 6:56 pm

Re: CompilerDev?

Post by kutkloon7 »

I've skimmed through the pages, and it looks like something nice is going on here :)
I'm very interested in compiler design but I haven't got much hands-on experience.

I read 'Let's Build a Compiler', which is great to get started with the actual coding. It is kind of an extremily watered-down version of the dragon book, but it has the advantage that you can start with actually coding something. I recently bought the dragon book (and I enrolled in a compiler design course at my university) and I like it very much.

It would be great to get a small community started, I'm definitely interested!
User avatar
skeen
Member
Member
Posts: 59
Joined: Tue Sep 27, 2011 6:45 am
Location: Denmark

Re: CompilerDev?

Post by skeen »

I'd just like to add, that I support the idea of having a compiler related subsession of the for a few reasons;
I like the idea of having seperation between the OS concepts and the required tools. Personally I'd prefer to see the OS dev forum be filled with issues about paging rather than padding or being unable to compile GCC, as this is, at least to me, not OS development, but rather about knowing ones tools.
To use myself as an example, I've had plenty of compiler related issues in my OS development (mainly due to fact that the compiler implementations of C++11/14 are still quite buggy), these issues are however not actually quite fit for the OS development boards, at least that's my opinion, because they are compiler issues, however QA sites like StackOverflow will mark these issues as targeting a too small audience, due to the lack of runtime support and the standard library.
Also as a hobbyist compiler writer, I'd love a casual and osdev like forum for compilers and programming language theory. And I think if this could be produced as a side-effect of having a tools/compiler subsection of this forum, then that'd be simply amazing.
However I don't believe most people on here, are going to be developing compilers, but rather linkers and loaders, so I guess these would be more interesting for the OS development domain.
Also maybe we're not supposed to be considering compilerdev, but rather tooldev, as the compiler is just one of the tools of OS development.

These are my thoughts at least.
// Skeen
// Developing a yet unnamed microkernel in C++14.
User avatar
bwat
Member
Member
Posts: 359
Joined: Fri Jul 03, 2009 6:21 am

Re: CompilerDev?

Post by bwat »

Maybe we should make 2014 the year of the C compiler.

We could collect as a group of individuals who want to rewrite the old(*) Unix PCC (Portable C Compiler) and create their own compiler. This compiler is quite simple http://citeseerx.ist.psu.edu/viewdoc/su ... .1.48.3512 and the source is out there http://ftp.math.utah.edu/pub///mirrors/ ... rtable_CC/ If a group of us were to undertake this at the same time, then there is more chance that those who haven't worked on compilers before would find any support they might need from those more experienced.

I've read that people have retargeted this compiler in 4 weeks or so working full-time. This means those who don't have much time would be able to do a retargeting in one year, and those with more time to spend might be able to do a complete rewrite, or develop a preprocessor, assembler, linker, etc. Surely, after attempting this, even an experienced C programmer would be able to say they know how a C compiler works that little bit better, and that they have a deeper understanding of the C language and its history.

I spent this morning getting the PCC targeted for the 8086 to compile and run on a 64-bit PC running linux. The part that took the longest time was rewriting parts of the YACC file which implements the parser.

Code: Select all

bwat@acer:~/BarrysPCC/8086/c86$ make
yacc -dy cgram.y
conflicts: 7 shift/reduce
cc -ggdb   -c -o y.tab.o y.tab.c
cc -ggdb   -c -o trees.o trees.c
cc -ggdb   -c -o optim.o optim.c
cc -ggdb   -c -o pftn.o pftn.c
cc -ggdb   -c -o code.o code.c
cc -ggdb   -c -o local.o local.c
cc -ggdb   -c -o scan.o scan.c
cc -ggdb   -c -o xdefs.o xdefs.c
cc -ggdb   -c -o reader.o reader.c
cc -ggdb   -c -o order.o order.c
cc -ggdb   -c -o match.o match.c
cc -ggdb   -c -o allo.o allo.c
cc -ggdb   -c -o comm1.o comm1.c
cc -c -O -DX8087 local2.c
cc -c -O -DX8087 table.c
cc -o c86 y.tab.o trees.o optim.o pftn.o code.o local.o scan.o xdefs.o reader.o order.o match.o allo.o comm1.o local2.o table.o -ly
bwat@acer:~/BarrysPCC/8086/c86$ cat /home/bwat/slask/test.c
/* test.c */

int factorial (n)
     int n;
{
  if(n <= 1)
    {
      return 1;
    }
  else
    {
      return n*factorial(n-1);
    }
}
bwat@acer:~/BarrysPCC/8086/c86$ cat /home/bwat/slask/test.c | ./c86 
	.data
	.text
	.globl	_factori
_factori:
	push	bp
	mov	bp,sp
	push	si
	push	di
	sub	sp,#LF1
	cmp	*4(bp),*1
	bgt	L13
	mov	ax,*1
	br	L12
L13:
	mov	ax,*4(bp)
	dec	ax
	push	ax
	call	_factori
	add	sp,*2
	imul	*4(bp)
	br	L12
L12:
	lea	sp,*-4(bp)
	pop	di
	pop	si
	pop	bp
	ret
	LF1 = 0
	.data
bwat@acer:~/BarrysPCC/8086/c86$ 
(*) Yeah, there's a modern C99 port out there but messing around with that wouldn't be as much fun or educational.
Every universe of discourse has its logical structure --- S. K. Langer.
User avatar
Combuster
Member
Member
Posts: 9301
Joined: Wed Oct 18, 2006 3:45 am
Libera.chat IRC: [com]buster
Location: On the balcony, where I can actually keep 1½m distance
Contact:

Re: CompilerDev?

Post by Combuster »

Well, one thing that bothers me here is that writing a compiler has to be about full-fledged programming languages. Unless you want to write a compiler for the sake of it, languages like C have been beaten to death, and hobbyist implementations of C++ will probably never meet the standard. And yes, I have once compiled a working version of my OS using PCC as a drop-in substitute for GCC. I don't even remember having to tweak the source code. As for the four week figure, you can write a simple java compiler in less than a week full-time if you don't force yourself to use a braindead language (read: C) to do it. Homework assignment on that one even.

That said, the formal definition of a compiler is about taking a language, and emitting a different computer language. It doesn't need to be a programming language, let alone a turing-complete one. Instead the practical focus can be on less elaborate things like configuration compilers and template stuff. You could even generate the source for an UI based on a high-level description. For practical purposes this adds much more convenience to your OS development than just rewriting a C compiler just because.

Also, you don't need a full parser-emitter pair to make some good use of parsing theory and use the result inside of some app. There's little to gain in staring yourself blind on needing to have exactly one thing.
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]
User avatar
bwat
Member
Member
Posts: 359
Joined: Fri Jul 03, 2009 6:21 am

Re: CompilerDev?

Post by bwat »

Combuster wrote:And yes, I have once compiled a working version of my OS using PCC as a drop-in substitute for GCC. I don't even remember having to tweak the source code.
That's the new C99 version of the compiler, not the one I linked to. There's not much challenge in doing anything with the modern one, the heavy lifting has already been done by others.
Combuster wrote: As for the four week figure, you can write a simple java compiler in less than a week full-time if you don't force yourself to use a braindead language (read: C) to do it. Homework assignment on that one even.
A C compiler written in C is probably more relevant for the readership of this board than any other option.
Every universe of discourse has its logical structure --- S. K. Langer.
User avatar
Combuster
Member
Member
Posts: 9301
Joined: Wed Oct 18, 2006 3:45 am
Libera.chat IRC: [com]buster
Location: On the balcony, where I can actually keep 1½m distance
Contact:

Re: CompilerDev?

Post by Combuster »

Well, if "no it isn't" is the best response you can give...
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]
User avatar
bwat
Member
Member
Posts: 359
Joined: Fri Jul 03, 2009 6:21 am

Re: CompilerDev?

Post by bwat »

Combuster wrote:Well, if "no it isn't" is the best response you can give...
Are we discussing PCC here?

If so, I referred to an old K&R version of the compiler (which requires code changes for successful compilation) and made a note that it wasn't the new C99 version (which no doubt doesn't need code changes). You started talking about the new version. I pointed out the differences for the second time.

What's the problem?
Every universe of discourse has its logical structure --- S. K. Langer.
kutkloon7
Member
Member
Posts: 98
Joined: Fri Jan 04, 2013 6:56 pm

Re: CompilerDev?

Post by kutkloon7 »

Just wondering, are there people still enthousiastic for a compilerDev section?

I'd love to help with making it happen, not in the last place because I'm playing with lexers/compilers too, and I'd love to discuss things.

I am totally for a subsection of the site. A whole wiki and forum would be even better IMO, but I don't know if the community would be big enough.
User avatar
nerdguy
Member
Member
Posts: 70
Joined: Wed Oct 30, 2013 8:11 am

Re: CompilerDev?

Post by nerdguy »

Not sure if this is already posted but do any of you guys know about this book :
Compilers: Principles, Techniques, and Tools
Also known as the dragon book. Its pretty famous between Compiler Developers.
When you say, "I wrote a program that crashed Windows," people just stare at you blankly and say, "Hey, I got those with the system, for free." - Linus Torvalds
64 bit Kernel in early development
http://github.com/nerdguy12/core64
User avatar
iansjack
Member
Member
Posts: 4685
Joined: Sat Mar 31, 2012 3:07 am
Location: Chichester, UK

Re: CompilerDev?

Post by iansjack »

Yes. It's listed in the "Books" section of the Wiki.
User avatar
bwat
Member
Member
Posts: 359
Joined: Fri Jul 03, 2009 6:21 am

Re: CompilerDev?

Post by bwat »

nerdguy wrote:Not sure if this is already posted but do any of you guys know about this book :
Compilers: Principles, Techniques, and Tools
Also known as the dragon book. Its pretty famous between Compiler Developers.
I've got the 1986 version. I like the theory but it's not my favourite compiler book. I've gotten more out of this these books:

Lisp In Small Pieces, Christian Queinnec. It is LISP specific but so many languages these days have their roots in LISP and this is a great book to learn how to implement members of that language family. The author shows how to build interpreters, compilers, and an object system. This book is one big bootstrap. At the end you just want to go back to the beginning and reimplement everything in the system you've built. You need to know LISP or SCHEME to get anything from this book.

Programming Languages, An Interpreter-Based Approach, Samuel N. Kamin. This book shows interpreters for LISP, APL, SCHEME, SASL, CLU, SMALLTALK, and PROLOG. It then looks at compilation and garbage collection. The languages are more flavours than faithful implementations. The big idea is that all of the interpreters share a common core at the heart of the initial interpreter design and when we alter small parts of the design we get the different languages. All interpreters are written in PASCAL - yes it's an old book.

Abstract Computing Machines, A Lambda Calculus Perspective, Werner Kluge. This is a brilliant book if you're into functional programming. The author shows loads of different machine designs and compilers for translating from the lambda calculus to the instructions of the machines. The thesis of the book is that modern computing machines are degenerate versions of the lambda calculus reducers described. If you're the practical type, then you'll actually learn a lot of theory by building the machines and compilers described here.


Each of the above books is written at a level which the average programmer would be comfortable with. They're all written by academics and they were intended for use in university courses at the undergraduate level.
Every universe of discourse has its logical structure --- S. K. Langer.
Post Reply