How They Do It

The Golf Magazine panel (see post below) works from this criteria, posted online but not printed in the current issue beside the list:

Our 100-strong rating panel includes major championship winners, Ryder Cup players, architects, leading amateurs, administrators and others respected for their golf knowledge. Each is asked to assign a letter vote, from A+ to F-, to each of the 575 nominated courses they have played. A’s are reserved for the world’s top 10 courses, F’s for those that don’t deserve to be on the ballot. Write-ins are permitted. Consideration is given for strategic design, playability, setting, tradition, condition, examination, etc.

Examination? This is golf. Not school. It's supposed to be fun. Right? Why can't Fun be a category? Sorry, continue...

The panelist determines the importance of the criteria in each case. Votes for courses played during the past five years are weighted 1.5 times those that have not. Architects cannot vote on their own designs. Once the panel’s nearly 25,000 votes are received, letter grades are translated into numbers. Finally, we simply do the math by dividing a course’s total by the number of panelists voting on it. The result is the most respected ranking in the world.

They were going strong there until that last sentence. I still like the setup. But only prohibiting architects from voting on their own designs isn't enough. Golf administrators should be prohibited from voting on courses they host events at. Photographers shouldn't be allowed to vote on client courses they're also paid to photograph, etc...

And architects also should not be allowed to vote on older courses they restored/renovated, which I believe they currently are allowed to do (ex: Rees Jones can vote on Torrey Pines, Tom Doak can vote on Chicago Golf Club, etc... I bet you can guess which example I think poses a legitimate conflict of interest).