Loading External Code And How It Ties In?

[PearOs]

Hey guys! This is something I've thought about for a long time and I was curious as to how this is usually done. So I write in C# as some of you may know with a compiler I wrote. I have a map file for the Kernel+OS and I know that when I start to write programs my compiler will use this map for so that if I did "iPear.Core.List.Add" it would use the memory address in the Map file. Which makes sense, kind of like a plug which I use for replacing Microsoft calls in IL. But how what about when the software wants to change a static variable defined like this "Variable: dd 0" if the program was loaded lets say into the 4mb mark in Memory or wherever the memory manager put it, how would it know the true address of where the variable is located? Is it a Data pointer or something? Sorry for my lack of understanding I think it's due to not knowing all the quirks to x86 assembly 100%.


Thanks, a ton

Matt

[bluemoon]

I do not know much on C# but for traditional languages and ABI this is usually done with link-time, load-time or run-time relocation techniques.

[jnc100]

It should in general be no different than for any other language.

The only slight difference I have found in writing a C# OS is that with the C# version the kernel itself often needs the standard library (mscorlib). For other kernels (e.g. C), typically the C standard library is not used in kernel space. This means that in the C# case the standard library is already loaded when you come to start any user-space program. You therefore need some way of linking to it at run-time.

Personally, for AOT compiled user-space binaries, I use the dynamic linking mechanism of ELF libraries. For JIT compiled ones, I hard-code the appropriate address at JIT time. These are both accomplished by storing a hash table of symbols and their addresses that is added to at run time as more libraries are loaded.

The referencing of static variables should be no different from the referencing of functions.

Regards,
John.

[bwat]

Personally, for AOT compiled user-space binaries, I use the dynamic linking mechanism of ELF libraries. For JIT compiled ones, I hard-code the appropriate address at JIT time. These are both accomplished by storing a hash table of symbols and their addresses that is added to at run time as more libraries are loaded.

And for dynamic languages you can use an atom table. The indirection means you can update code/variables whilst the system is up and running.

[PearOs]

Ok I see. My C# Kernel doesn't use mscorlib at all. It's 100% off the grid so to speak, but I think I may have a way to make everything work. Is there a register I can use as an address offset that wont affect how the code runs? But can be used to point variables to where the program was loaded to in memory? Thanks, Matt

[h0bby1]

Ok I see. My C# Kernel doesn't use mscorlib at all. It's 100% off the grid so to speak, but I think I may have a way to make everything work. Is there a register I can use as an address offset that wont affect how the code runs? But can be used to point variables to where the program was loaded to in memory? Thanks, Matt

normally with C modules you have to modify the binary machine code loaded in memory to add the offset to the relocation entries, the informations in the relocation table store all the location of direct memory address in the code section that need to be changed (added the ofset to the memory location where the code section is loaded) to match the location where the module is loaded

with x64 there is a register that can be used as a memory ofset, but i don't know how it works exactly

[jnc100]

Is there a register I can use as an address offset that wont affect how the code runs? But can be used to point variables to where the program was loaded to in memory? Thanks, Matt

As its your own compiler, and you can define your own ABI, there is nothing stopping you from using a dedicated register for this purpose. You will, however, lose a register that could otherwise be used by your code. Possibly more than one if more than one external library is referenced?

However, I think you are overcomplicating things. What exactly does your compiler produce as output? If it produces an object file with appropriate relocation entries (like a C compiler), there is nothing stopping you from resolving these at load time (as in dynamic objects) rather than at link time.

Regards,
John.

[Love4Boobies]

And for dynamic languages you can use an atom table. The indirection means you can update code/variables whilst the system is up and running.

"Atom" is not a standard term in PLT. I presume you're using Prolog vocabulary here? Perhaps you meant something different but I feel like you're conflating dynamic linking (which is linking done at run-time), dynamic type systems (languages which use such type systems are sometimes called dynamic languages---we're really talking about a potentially dangerous and restrictive special case of static typing), and/or JIT (which is dynamic translation).

[bwat]

And for dynamic languages you can use an atom table. The indirection means you can update code/variables whilst the system is up and running.

"Atom" is not a standard term in PLT. I presume you're using Prolog vocabulary here? Perhaps you meant something different but I feel like you're conflating dynamic linking (which is linking done at run-time), dynamic type systems (languages which use such type systems are sometimes called dynamic languages---we're really talking about a potentially dangerous and restrictive special case of static typing), and/or JIT (which is dynamic translation).

No, I mean atom table as in dynamic linking of symbols and their associated values, e.g., variable value, function value, print name, etc. It has nothing to do with dynamic types or run-time compilation.

Atom is a term that has been in use long before the term PLT and is still in use. The first reference I have to it in my personal library is from 1962: LISP 1.5 Programmer's Manual, McCarthy et al., and the most recent references I have are from 1996: Prolog: The Standard, Deransart, and Lisp in Small Pieces, Queinnec. You'll no doubt find later references than those.

we're really talking about a potentially dangerous and restrictive special case of static typing

No, static typing is restrictive as it is, by definition, "context-sensitive constraints on the well-formedness of programs" (Language Semantics and Implementation, Course Notes, Rod Burstall). With dynamic typing you can choose to avoid the constraints.
I'm also surprised to see dynamic typing being described as a special case of static typing. The following is an example where dynamic typing works fine and static typing cannot cope. The first example is in my dialect of Scheme, and the other is in SML.

Code: Select all

(set! twice (lambda (f v) (f (f v))))
#<closure:0x7f2cdbc0fba0>
OK
(set! head car)
#<primitive:0x7f2cdba92050>
OK
(twice head '((1 2 3) (4 5 6)))
1
OK

Code: Select all

Standard ML of New Jersey v110.69 [built: Mon Jun  8 23:24:21 2009]
- fun twice f v = f (f v);
val twice = fn : ('a -> 'a) -> 'a -> 'a
- exception NonList;
exception NonList
- fun head [] = raise NonList
= |   head (h::_) = h;
val head = fn : 'a list -> 'a
- twice head [[1,2,3], [4,5,6]];
stdIn:5.1-5.30 Error: operator and operand don't agree [circularity]
  operator domain: 'Z list -> 'Z list
  operand:         'Z list -> 'Z
  in expression:
    twice head
- 

Just to prove it is static typing at fault, here's the beta reduction done by hand:

Code: Select all

- head (head [[1,2,3], [4,5,6]]);
val it = 1 : int
- 

[PearOs]

Ok. I see now. I think I know how to make this work now guys. Thank your for your input and ideas.
This should be a fun one to implement in my operating system.

Thanks, Matt

[Love4Boobies]

we're really talking about a potentially dangerous and restrictive special case of static typing

No, static typing is restrictive as it is, by definition, "context-sensitive constraints on the well-formedness of programs" (Language Semantics and Implementation, Course Notes, Rod Burstall). With dynamic typing you can choose to avoid the constraints.
I'm also surprised to see dynamic typing being described as a special case of static typing.

That is a common and understandable misconception. I'm actually glad this came up. And, since you've mentioned SML, I'll direct Robert Harper's blog (one of the designers of SML, amongst other things) to avoid simply repeating his words: clicky.

[bwat]

I read it and I was quite disappointed. Sadly there was no detail, only opinion.

Since I don't have time to read the draft of the book that the blog entry was promoting, maybe you can tell us what type the expression

Code: Select all

twice head [[1, 2, 3], [4, 5, 6]]

should have? According to you it must have a type because dynamic languages are a "restrictive special case of static typing". Hence, there can be no dynamically typed expression can not be statically typed.

As another example, consider the following example from Types and Programming Languages, Pierce:

Code: Select all

if <complex test> then 5 else <type_error>

Now, if <complex test> always evaluates the true then this can be dynamically typed. It cannot be statically typed as <type error> is put there to introduce failure.

Is the above example a "restrictive special case of static typing" as you put it?

Edit: Rewrote some things so as not to put words into Harper's mouth.

[Combuster]

there was no detail (...) maybe you can tell us what type the expression

Code: Select all

twice head [[1, 2, 3], [4, 5, 6]]

should have?

Not sure if trolling, or just unable to read.

Any expression in a dynamic language has the same static type. Call it "object" if you have to.

[bwat]

Not sure if trolling, or just unable to read.

Looks like you've made progress. This time you're not relying on your imaginary friends to call me a troll.
http://forum.osdev.org/viewtopic.php?f= ... 48#p228648

Any expression in a dynamic language has the same static type. Call it "object" if you have to.

Do you think that Pierce's example with the conditional (given in my previous post) should be typed as "object"? Is there a statically typed language that would accept an instance of that example?

Taking this view, the static type is of no use as it is overly conservative(*) and the dynamic type checker will calculate a non-object type. I see this as an argument for my claim that dynamic typing is not a "restrictive special case of static typing".


*) Calling it "overly conservative" is being kind. Reducing the type checker to a function that can only return one constant means you don't really have a type checker any more.