OSDev.org

When I try to link my bootstub with cygwins ld I get a message that says the file format is not recognized.

The stub is compiled with nasm using -f aout

Should I try another linker, and where can I find one of those?

i rather suggest that you try to find out what input format your linker support (e.g. coff or elf) and ask NASM to output binaries in that format.

FAQ: GCC Cross-Compiler tells you how to set up a compiler / assembler / linker / disassembler toolchain that "speaks" whatever format you like. (Just set $TARGET for whatever you want.)