XPCOM is like COM, for Windows. It lets you get pointers to components inside Mozilla, and it lets you talk about their interfaces as well.
XPConnect is what lets JavaScript talk with Mozilla guts, the XPCOM system. You can imagine it like a bridge between JavaScript code, and Mozilla guts.
xpcshell is a command line interface to Mozilla JavaScript. It's an interactive interpreter -- it gives you a shell that talks straight to Mozilla, via JavaScript's XPConnect bridge into it.