There may be general genetic tendencies that you can do nothing against but I would tend not to agree with the "set point" theory (had never heard of it before actually). Consider my case: growing up I had always been a naturally skinny, tall and lean kid... an ectomorph. I'd always be under 8%, I was several times even measured at 4% with calipers. My main problem was gaining muscle mass, even when I tried I couldn't do it.
So last summer at the ripe and mature age of 19 I made a big and wise decision "what the heck I'm tired of being skinny, let's bulk up like crazy no matter what". I did, and gained mass fast. Too fast: my bodyfat percentage shot up to around 15-20%. My face wasn't the same anymore. I was amazed at the change, I couldn't believe my old athletic friends were now taunting me for being "fat" and having a belly. I was like a different character.
Since then I've gradually lost the fat and these days I'm under 10% for sure. I'll never bulk up again like an idiot the way I did last year, but I'm glad I did it at least once because it taught me that you can change and become pretty much anything if you do what's needed. Your body will adapt and reach new comfort zones in my opinion. When I became chubby, I felt I could've stayed there had I wanted to... My stomach had adapted to eating a lot. But then I became sensible and started thinking that lean and fit was better than big and fat even if it meant being on the skinny side of the fitness spectrum.