Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
You will not be able to load the kernel by name. You would have to somehow patch or reprogram GRUB to support your file system. How could GRUB possibly understand your new file system without being reprogrammed? It's not magic.
However, if you know the location of the kernel on the disk, you most certainly can sill load it. Something like, for example, "kernel 200+15", if the kernel were at sector 200 and 15 sectors long.
Another possibility is to also have a boot partition which includes grub & your kernel (or a loader for your kernel) on a grub-supported file system (such as ext2/fat).
Or more accurately, a stage1.5 and a stage2 of grub that supports your filesystem. Are you referring to Legacy GRUB or GRUB 2? I've happened to have added a filesystem to Legacy GRUB, but I don't know anything about GRUB 2.