The news media has jumped on a new study of judges at the California State Fair wine competition as evidence that all wine tasting is baloney. But the author of the study says that's not what he meant. Not by a mile.
In the study, tasters sampling the same wine, hidden in several successive blind flights, could only get within 4 points of their previous scores on a 100-point scale. That's the difference between a gold medal and something less.
I reached Robert Hodgson, the study's author, by e-mail in South Africa, where he is making a presentation on testing judges at wine competitions. That's what the study was about: The difficulty of getting agreement among several different tasters who don't normally sit down and sample wines with each other.
"I am not against tasting and reviewing wines in general," said Hodgson, owner of Fieldbrook Wines in California and author of several studies on wine tasting. "I'm not intent on debunking the whole idea of tasting and judging wine. On the other hand, the tasting and judging of wines in the setting of a wine competition, in my view, leads to meaningless recommendations."
His motive for this work is personal. He wondered why his wines would earn gold medals in some competitions and nothing in others. He has been studying the results and, as a member of the governing board at the State Fair competition, the inner workings, for nearly a decade. "Year after year, the answer was that the variation among panel members could be mostly explained by the inability of judges to consistently grade replicate samples of wine poured from the same bottle."
The California State Fair judges are drawn from retailers, restaurant owners and wine directors, winemakers and others in the trade, and a few writers. This is fairly typical of American wine competitions in general. These are not people who taste blind, for publication, on a regular basis.
"I often point out emphatically that my conclusions do not apply to … persons who may be involved in rendering evaluations of wine," Hodgson said. "What I do say is that based on an enormous amount of data, that the results of the State Fair, and by extrapolation to many other competitions, do not provide the public with reliable information with which they might make a rational buying decision."
In order to deliver reliable advice, judges—whether for wines, restaurants, movies or art—must have the experience to understand a work (or a wine) and put it in context, as well as a methodology that allows for fair and consistent evaluation. These are the standards we strive to meet at Wine Spectator.
However, wine is personal, and circumstances can significantly affect our impression of its character. So another person's experience with the same wine, in different circumstances, probably will be different. Also, wine is a moving target. Certainly, by the time a review is published and you drink the wine, it will have changed in the bottle, maybe a little, maybe a lot.
Despite all that, I believe it's important for a review to provide as thorough a description and as precise a reflection as possible of how much I like it. This gives us a baseline, and it makes it possible to compare the wine with others reviewed at the same time, or from the same region. And it gives readers a point of comparison with other critics.
A professional tasting note is a good-faith effort to describe a wine, and place it in context with others that you, the reader, might consider buying or drinking. Nothing more, nothing less. It's not perfect, but it's not B.S. either.