OSDev.org

I understand how I can load relocatable to any memory location but how would I make my OS be able to run 16-nit DOS programs? Would I have to interpret the binary or could I just run it in it's own segment. How would I emluate BIOS interrupts as they are 13h, same as GPF. I don't want to implement v86 as this reduces performance and is over complicated.

Any ideas will be appreciated

You have no choice but to implement V86 mode (or switch to real mode) if you want to run DOS applications. DOS apps will not run in protected mode -- real mode and protected mode are different enough to break them.

or maybe you might run those applications in a software virtual machine like BOCHS, or vmware or something you write by yourself. However, it seems pointless to write a software VM for the 8086 as there is a hardware one in the processor.