Then two things happened: Windows, and 32 bit processors. Fast forward to today, and MS-DOS has pretty much disappeared from the landscape - Windows, and its rich graphical interface, is everywhere. The same is true for the 16bit processor. Most all contemporary machines are built on various flavors of 32bit processors.
What does all that have to do with COMMAND.COM and CMD.EXE? It's quite simple really: CMD.EXE is a command line processor for 32bit Windows. Much like MS-DOS before it, it provides a character-mode only user interface into which you can type commands to run programs. CMD.EXE is a 32 bit program that is fully a part of Windows - in fact it's what gets run when you select Command Prompt on the Windows Accessories menu.
COMMAND.COM, on the other hand, exists only for compatibility and 16bit programs. It exists solely for those programs which have not been updated since the days of MS-DOS. It's designed to run 16bit applications, and operates just like the old MS-DOS did.
On the surface, they're very similar. Even though CMD.EXE was a complete re-write, it's based on the same concepts and similar syntax. But the differences become quickly apparent. Let's used your "CD" example from above.
In CMD.EXE, you can change to the "Program files" directory by doing this:
cd "\Program Files"
Note the quotes around the directory name. Try doing that in COMMAND.COM and you'll get this:
cd "\Program Files"
Parameter format not correct - "\program
The problem is COMMAND.COM doesn't know about long file names (filenames longer than 8.3) or filenames that have spaces in them, and it doesn't have a way to quote the name. So if you still need to, how do you CD to that directory in COMMAND.COM? Well, first you need to find out the alternate, 8.3 name of that directory ... its short name:
dir /x
Directory of C:\
08/21/2005 09:50 AM
(I've removed some of the lines of output for brevity.) The "/X" parameter to DIR tells it to include the "shortname" in the output. In this case that's "PROGRA~1", a machine generated alternative name for "PROGRAM FILES". Now we can CD to that in COMMAND.COM:
cd progra~1
dir
Volume in drive C is
Volume Serial Number is B026-62CA
Directory of C:\PROGRA~1
08/21/2005 09:50 AM .
08/21/2005 09:50 AM ..
03/24/2005 04:42 PM Adobe
...etc...
The bottom line is actually quite simple: use CMD.EXE unless you know you need COMMAND.COM for compatibility with some older 16bit program. And even then, if you try and run a 16 bit program in CMD.EXE, it will typically run it within COMMAND.COM for you anyway, so you still may not need to know or care.
And in case you're wondering, yes ... something similar is coming down the road as 64 bit processors begin to arrive. The 64bit version of Windows does, I believe, include another command processor, one for native 64bit command-line programs.