A GPS receiver discerns your position on the surface of the earth by measuring the length of time it takes to receive signals from satellites. It cannot, however, tell what direction you are facing while standing still (or moving slowly). So if you are navigating to a waypoint, your GPS can only point you in the correct direction once you are moving.
This is where an electronic compass comes in handy. It can tell what direction you are facing, even while standing still. Not only does this make navigating easier, it also helps with projecting waypoints and orienting paper maps.
Electronic compasses will shorten battery life somewhat. They are typically paired with a barometric altimeter.