Easiest is to drive there. There are car parks available at the ferry terminals. There are also buses and maybe minivans. Or a twice weekly flight (Tuesday & Thursday) on Berjaya Air from KL to the tiny airstrip on the island.
My GPS really did not want me to go there, and was determined to direct me to Lumut. You simply go straight instead of turning right to Lumut at the big shell sculpture.