Does it exist?(portable code generation library)

[earlz]

Hi, I've recently been starting to write a little project to generate code on the fly(at runtime) and it's actually quite difficult(skimming this opcode reference searching for 'ret') but I've thought such a thing would be very handy for JIT compiler creation, or skipping the assembly portion of a compiler(not really powerful enough though), or whatever..

Well, does such a thing exist. I tried searching and the only thing I've seen is a thing to generate java bytecode.

And my project is also designed so it can be targetted to multiple archs at runtime and it's completely standalone, so it could be used easily in an OS to create something like Synthesis....

Really, it seems like too familiar of a project to not already exist, and if it does, then I'm not going to waste my time...

[Combuster]

A library that essentially does the equivalent of compiling something...

What is wrong with just incorporating a compiler? (or calling the thing by its name?)

[earlz]

Well my aim is not a compiler... I'm not implementing

Code: Select all

eval("x=x+10;");

I'm implementing something much more light weight(but harder to use)
Like here is a bit of working code:

Code: Select all

	printf("pcg_init\n");
	pcg_init();
	printf("pcg_set_arch\n");
	pcg_set_arch(ARCH_I386);
	printf("setup\n");
	int a;int sz=512;
	a=pcg_setup_function(mem,sz); //function prelogue
	if(a<0){printf("a<0 1\n");return -1;}
	mem+=a;
	sz-=a;
	printf("assign\n");
	a=pcg_assign_mem32_const32((uint32_t)&x,0xBADBABE,mem,sz);
	if(a<0){printf("a<0 2\n");return -1;}
	mem+=a;
	sz-=a;
	a=pcg_end_function(mem,sz);
	if(a<0){printf("a<0 3\n");return -1;}
	mem=mem_old;
	printf("mem=%x\n",mem);
	callme=(void*)mem;
	printf("callme\n");
	callme();

and the assign function(is ugly):

Code: Select all

int pcg_assign_mem32_const32(uint32_t ptr,uint32_t c,void *mem,int sz){
	char *m=mem; //makes things hella easier
	if(sz==0){return -2;}
	switch(pcg_arch_num){
		case ARCH_I386:
			if(sz<5){return -2;}
			m[0]=0xB8; //op for mov eax,imm32
			*(uint32_t*)&m[1]=c; //the imm32
			m[5]=0xA3; //op for mov offset16/32,eAX
			*(uint32_t*)&m[6]=ptr; //the offset
			return 10;
		break;
		default:
			return -1;
		break;
	}
	return -1;
}

So, what I intend is, well say qemu used this library(minus kqemu) to generate it's temporary code. Whenever it wanted for another architecture to be supported as a host, it would only have to retarget this library, rather than trying to track down inline assembly and such(and add a #ifdef and all that crap)

Also, in case your wondering, I will be handling the size differences in different archs(like how the Z80 is 16bit, well there will still be a working assign_mem32_const32 though some of the mem32 bits will be ignored)

[MasterLee]

Will that make any sense? For speed optimzation looking LLVM maybe an better choice.

[Owen]

LLVM is pretty much a portable code generation library. I've seen at least one family of embedded processors use it as a base for porting GCC.

[manonthemoon]

Look into GNU Lightning. I think that's what you're looking for.

[earlz]

Look into GNU Lightning. I think that's what you're looking for.

the GPL is not an acceptable license for a library IMO, as I don't want any project that uses Lightning to also be forced to be GPL...

Though, that does seem to mostly be what I want.. in fact, I think I'm going to rip off a few of their ideas but have the license as BSD. Also, it will be more multithreading friendly(as in, not rely on the preprocessor hacking to support reentrancy)

[rootnode]

Seems like it's License Discussion Week.

[Solar]

While I'm right there in the front row to throw stones at the GPL, GNU Lightning is licensed under the Lesser GNU Public License, i.e. software linked to GNU lightning can be under whatever license you like, as long as you provide an unlinked version of your application as well so that users could re-link a modified version of GNU lightning with your app.

[earlz]

While I'm right there in the front row to throw stones at the GPL, GNU Lightning is licensed under the Lesser GNU Public License, i.e. software linked to GNU lightning can be under whatever license you like, as long as you provide an unlinked version of your application as well so that users could re-link a modified version of GNU lightning with your app.

um, did you look in the archive? I saw COPYING and COPYING.LESSER with COPYING containing the GPL and COPYING.LESSER containing the LGPL. Then I looked in some of the source, and found:

Code: Select all

/* Disassemble from a buffer, for GNU.
   Copyright (C) 1993, 1994 Free Software Foundation, Inc.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.  */

#include "sysdep.h"
#include "dis-asm.h"

So, even if portions of the code are LGPL. The library itself must be GPL because it uses GPL code.

[Solar]

Well, the file I looked at read LGPL... so, "licensing situation unclear", the worst of all licenses...

[Owen]

As I said, LLVM. It's interface is probably not as nice, but it generates some very good code (In fact, I think LLVM-GCC now beats GCC at optimizing)

[earlz]

As I said, LLVM. It's interface is probably not as nice, but it generates some very good code (In fact, I think LLVM-GCC now beats GCC at optimizing)

I didn't think LLVM really had a programmable interface though, isn't it more of just a compiler?

[Owen]

LLVM is a library for

* Generating code which runs on the LLVM virtual machine
* Executing said code
* JIT compiling said code
* Ahead-of-time compiling said code

[earlz]

Try libjit maybe?

Of the actually working jit libraries i know only libjit, GNU Lightning, LLVM and VCODE (really old jit prototype). You might just write your own if you have problems with licensing, the idea is about the same everywhere.

libjit seems pretty close to what I want.. however, I still have the excuse of making my own for licensing reason, and that the generated code isn't very easy to work with(can't use a regular function pointer), and it sems geared for 32bit only.

also, I wish for the code produced to be potentially position independent(if possible, per architecture) and to be standalone, yet possible to use external functions say from libc or wherever.