As I recall, the fitness center beach has a wall that drops off to the ocean with little or no sand along the wall at the waters edge. It did not look appealing to swim there when we were there in April.
We went swimming there, no one yelled at us. Then again no one yelled at us for anything we did! If I remember correctly there is a spring around there that empties into the water which cools it off a little.
You can access the water from the area in front of the fitness center. You need to go down beautiful stony steps near Charlie's grotto and you will exit a lovely stone arch onto a very small sandy area. There are no chairs here, but you can swim. There is some lovely coral to the right of this area, so you'll need to be careful.