Atheletes do in general have lower heart rates than sedentary people. Lance Armstrong famously has a resting heart rate of 32-34. There is a good explanation here http://www.runnersworld.co.uk/news/article.asp?UAN=1539
. I would not become concerned by a low resting heart rate in isolation, especially as you have noted that your heart rate increases appropriately with exercise.
Heart rate recovery after exercise is a decent measure of overall cardiovascular fitness.
Some people do develop dysfunction of their sinus node ("sick sinus syndrome") where their heart rate at times slows inappropriately and has frequent arrhythmias such as atrial fibrillation. Possibly your cardiologist is worried about this possibility. This is increasingly common with increased age. Such people typically eventually have symptoms such as lightheadedness or fainting spells that prompt treatment with a permanent pacemaker. In the abscence of such symptoms (not just occasional palpitations) a pacemaker is generally not indicated unless you have marked EKG abnormalities (such as AV conduction block).