You can indeed. The best way to swim is to go out to the island, claim your loungers then proceed to the right side of the island and go down path/stairs that leads to the water. You can wade in there and cool off! On really hot days, I've seen almost every islander in the water instead of on the island! You can also get in near the pier.
You need to wear water shoes. There are a lot of spiny urchens about 15 feet into the water. And it gets very rocky. If you want to swim while you are at the island, I would suggest snorkeling. Since CTI does not condone snorkeling on you own near the island they would not let us use their equipment. Fortunately, we brought our own.
Just to follow on Bowser's comment about snorkeling from the island.... I did this in December, with my own gear (mask, snorkel & water shoes. Enter the water at the stairs along the southeastern margin of the island where there is a small sandy beach area. Swim toward the main beach across a very shallow reef environment (being careful to avoid the rocks and corals) until you reach a point where the water suddenly deepens to a level where you can swim more freely. I went east along this margin, which extends underwater for some distance, beyond the eastern limit of the island itself. The fish are plentiful and varied, and the reef environment along this margin is generally well protected from the wave energy that dominates the outer margin of the island and the open water that extends eastward and back toward the beach. One should watch out for boat and hydrocycle traffic coming to and from the watersports area. Enjoy!