Honestly, I don't think adding weight would matter much. I sink like a freakin' rock as it is. Since you're underwater to begin with, it'd just keep you near the bottom...which isn't a big deal. At speed with the fins, just angling your hands a bit would keep you from scraping bottom...generating lift just like the wing of a plane.
Ditching the fins will make this a lot more difficult. I'm betting that times will come close to doubling, if not more. The other way to make it more difficult would be to insist on multiple laps (not coming up)...with the second/third/etc. lap only counting if you made it the whole way. If you bailed, you'd have to go back to the last wall.