Question on calling conventions for syscalls/functions

[Ethin]

So, I know of the various calling conventions that x86 has (safecall, stdcall, vectorcall, ...). I also know that other architectures (RISC-V, ARM, ...) have dedicated registers for passing arguments to functions (e.g.: RISC-V has a0-A6). My question is mainly out of curiosity: vectorcall, for example, passes arguments in rcx, rdx, r8, r9, and then vector arguments in xmm0-xmm5. However, these register allocations seem arbitrary to me. X86 has 32 vector registers (and 96 if you have the full vector instruction set -- SSE all the way up to AVX-512). What I'm wondering is this:
1. Why don't we put all integer arguments in RAX, RBX, RCX, RDX, R8, R9, R10, ..., or use R8-R15 for arguments and RAX/RBX/RCX/RDX/RSI/... for return values?
2. For system calls, I've noticed that -- for example -- Linux uses EAX, EBX, ECX, and EDX fro arguments, but this is very different on x64. Why is this the case? Shouldn't it use the 64-bit RAX/RBX/... registers on x64 like it does on x86?
3. For vector arguments, vectorcall uses only XMM0-XMM5. So what happens to XMM6-XMM31, and why are they unused? Similarly, what about YMM/ZMM0-31?
4. Would it be okay for my OS to use the above (logical) calling convention, or would it break a lot, and how much effort would it require for me to add that to compilers/interpreters?
I know that this is primarily an x86-specific set of questions, but I'm just trying to understand why we don't use register arguments for parameter passing and return values as much as possible for all OSes, and use the stack for register spilling, whereas on other architectures registers seem to be used as much as possible before spilling onto the stack arguments that exceed the register count.

[nullplan]

1. Why don't we put all integer arguments in RAX, RBX, RCX, RDX, R8, R9, R10, ..., or use R8-R15 for arguments and RAX/RBX/RCX/RDX/RSI/... for return values?

Having non-volatile registers (registers the callee needs to keep the same) is good for calling functions in a loop. Having all registers as arguments would simultaneously make them all volatile, so before you can call a function, you have to save everything you currently have in registers. And if you need to call functions repeatedly, you have to do that every time. Add to that the fact that most leaf functions don't require all registers, and it becomes obvious that such a calling convention does not use the registers very well, and would over-use the stack.

2. For system calls, I've noticed that -- for example -- Linux uses EAX, EBX, ECX, and EDX fro arguments, but this is very different on x64. Why is this the case? Shouldn't it use the 64-bit RAX/RBX/... registers on x64 like it does on x86?

When Linus designed the system call mechanism for x86, it was early in the development, and he just sorta winged it. By the time AMD64 had come around, Linus was no longer the primary developer, having instead transitioned to a managerial position (he's the guy that approves the patches other people write). The system call mechanism on AMD64 was written by other people, and they thought it would be best to just stick to the calling convention they were going to use. Shuffling arguments around is a major part of calling syscalls on x86, and that is just not useful to anyone. So instead, they kept to the user space argument order, except for rcx, which will be clobbered by the syscall instruction.

3. For vector arguments, vectorcall uses only XMM0-XMM5. So what happens to XMM6-XMM31, and why are they unused? Similarly, what about YMM/ZMM0-31?

Yeah, this one doesn't really make sense to me, either. But then, functions taking more than six floating-point arguments are rare, so who cares? The XMM registers are all marked as volatile in the ABI, so this is not to allow the caller some space to save their FP variables.

4. Would it be okay for my OS to use the above (logical) calling convention, or would it break a lot, and how much effort would it require for me to add that to compilers/interpreters?

Going against the grain on calling conventions requires you to patch at least the compiler (and good luck with that). Unless you have an extremely compelling reason to do so, I don't think it is worth the effort.

[Octocontrabass]

I also know that other architectures (RISC-V, ARM, ...) have dedicated registers for passing arguments to functions (e.g.: RISC-V has a0-A6).

The registers are only "dedicated" for that purpose by the ABI. Even the register names come from the ABI. MIPS is a good example of this: it has two different sets of register names depending on which ABI you're using.

However, these register allocations seem arbitrary to me.

At least the AMD64 psABI is based on metrics instead of being completely arbitrary. For example, they intentionally avoided using RAX for function arguments because code generation would be less efficient that way.

X86 has 32 vector registers (and 96 if you have the full vector instruction set -- SSE all the way up to AVX-512).

No, there are not 96 vector registers. XMM registers are the low 128 bits of the YMM registers, and the YMM registers are the low 256 bits of the ZMM registers. SSE has 8 vector registers. Long mode extends it to 16 vector registers. AVX-512 extends it to 32 vector registers.

[Ethin]

Thanks for your input, guys, that helped a lot. I understand the situation a lot better now; before I was confused about the register order because it seemed arbitrarily chosen and I couldn't find any background on why it was done the way it was.