OK, back to square one. Don't get hung up on names, especially not the name "virtual memory", because different people mean different things when they say that.
Let's say you have a really, really basic OS. No MMU, which means no virtual-to-physical mapping, no pages.
Step 1 - Physical Memory
You determine how much RAM you have at which addresses. You assign areas of this memory to processes as they start up. You assign additional memory to running processes as requested by their [tt]malloc()[/tt] implementations. You collect memory from userspace [tt]free()[/tt] implementations and ending processes. This is your memory manager.
Step 2 - Virtual Addressing / Memory Protection
Now comes the MMU, which enables you to provide each running process with a virtual address space starting at 0 and reaching up to 2^32-1 (on a 32bit system). This is usually done page-wise, because mapping each virtual address to a physical address would require too much memory for mapping data.

You get the added headache of maintaining page tables and directories, but on the upside you don't have to fiddle with executable relocations, and you can protect processes from each other. You might want to call this your
virtual memory manager, or you might reserve that for...
Step 3 - Virtual Memory
And because of some nice features provided by the MMU, you can map virtual memory pages to physical memory
not residing in RAM but on hard drive. When such page is accessed, you stop execution, and swap in the requested page from hard drive (possibly swapping out another page in exchange).
Note that everything beyond step 1 is completely optional. It's a common illness that people tend to lump step 2 and 3 together, and aren't very precise in their usage of terms like "virtual memory" or "paging".