Higher damper makes it harder in the same way that taking off from a stop sign in 5th gear is "harder" than 1st gear. Harder doesn't always mean better in terms of maximizing power output. Plus rowing with too high of a damper with anything less than perfect form can lead to injury.
Also note that drag factor, not damper, is what you should really be worried about. Drag factor can vary for lots of reasons from rower to rower and day to day.