You know that there's something special about a place when you've hardly arrived there, and already you are planning how you'll return. Soon. Flinders Island is like that.

The last remnants of an ancient land bridge between Victoria and Tasmania, this craggy island is just one of the Furneaux group, today reduced to awkward stepping stones in Bass Strait, with not a lot in the way of land between them and Antarctica.

Flinders Island itself is not really on the way to anywhere. You must decide to go there. Make an effort. Nor is this 90 by 30-something kilometre island teeming with locals. Jim, our tour guide was quick to point out there are only 800 residents, but 17,000 geese. That's Cape Barren geese, of course, named for the largish island across Franklin Sound to the south.

Some enterprising souls are breeding these birds now for export to Tasmania and the mainland. Other new industries – desperate attempts to put the land to other use now that wheat and sheep prices have plummeted – include crops of opium poppies and cauliflowers. And then there's abalone breeding along the coast. A vineyard has been planted, and with Flinders' moderate climate, there are optimistic hopes for a future with wines too.

Visitors can be as active or as laid back as they like here. For the energetic there is plenty to do – four wheeler tours, fishing, diving around some of the island's many shipwrecks, bushwalking and boating. Some people do a five-hour climb of Mt Strzelecki, the 760 metre peak that dominates many views on the island. The super-fit take part in the Easter Three Peaks race in Tasmania that combines boat racing with mad dashes up Mt Freycinet in Coles Bay, Mt Wellington overlooking Hobart, and Mt Strzelecki.

Yet you could come and just relax at a comfortable bed and breakfast or the beautifully restored family-run Flinders Island Interstate Hotel, in Whitemark, built in 1911.

The island's beaches enclose the land with white apostrophes – crunchy crescents splashed by cobalt and turquoise waters that could have skipped across from some coral atoll. At one, there is buried treasure too of a sort. Some older residents remember diving for it using hookah equipment in Killiecrankie Bay.

Those Killiecrankie 'diamonds' are really a sort of topaz, but their brilliance can be deceptive enough and even if you don't want to use the sieves and shovels that are for hire there so you can fossick for your own, you can buy some ready-cut and polished and set in gold or silver.

Today the bays yield up crayfish more readily and a sign at the gift shop in Whitemark, suggests you place an order for them at around $30 each. Here you can also buy the appropriately named Roaring Fort' pickles and sauces and honey. Another shop has local free range eggs and inland, at Latitude 40, the farm shop sells woollen pillows and softly luxurious wool filled vests.

Sometimes it's hard to believe that you're not on a remote island off Scotland. In fact with Mt Strzelecki reflected in water, cattle grazing nearby, and gulls flying overhead, the similarities are strong.

There's a mystery about Flinders. How the island has survived, why people stay. I'm going back, of course. I need to try to crack the code, and discover what the lure is. So, if you go, and find out first, please don't tell me.

I need to experience the thrill of finding out for myself.