You can use your car to get to the island, the entrance is here N 45 56.964 W 1 05.691. If you drive at low speed, then any car will pass. Though, there is not much parking space on the island, try to park here N 45 57.245 W 1 06.345 or here N 45 57.639 W 1 07.214.
You can also leave the car near the beginning of the spit and continue on foot. The ordinary sneakers are ok, but if you have waterproof hiking shoes, better use them. Low tide usually lasts at least 3-4 hours, so you will have enough time to walk.
Two steps from the island is the Logis Hôtel la Chaloupe (booking.com) where we stayed overnight. Overnight is needed if you do not know exactly the tides' schedule. And in general, this is an interesting place for one night.
The view from the window of the hotel at the middle of the low tide
And at low tide
There is a fort on the island, though opening hours are not clear, since they are depending on tides. Try to check this webpage (in French) https://www.ecomuseeportdesbarques.net/tarifs-horaires. Currently the fort is guarded only by a gang of rabbits.
Behind the fort and in different parts of the island you can see traditional fishing houses.
Oh yeah, why is the island called Madame? Perhaps because the ships entering the port of Rochefort landed sailors here and they spent time with ladies?
Getting there: entrance to the island N 45 56.959 W 1 05.699. You can take a bus I from Rochefort to the Les Rouches stop (route map, schedule).