The news media has jumped on a new study of judges at the California State Fair wine competition as evidence that all wine tasting is baloney. But the author of the study says that's not what he meant. Not by a mile.
In the study, tasters sampling the same wine, hidden in several successive blind flights, could only get within 4 points of their previous scores on a 100-point scale. That's the difference between a gold medal and something less.
I reached Robert Hodgson, the study's author, by e-mail in South Africa, where he is making a presentation on testing judges at wine competitions. That's what the study was about: The difficulty of getting agreement among several different tasters who don't normally sit down and sample wines with each other.
"I am not against tasting and reviewing wines in general," said Hodgson, owner of Fieldbrook Wines in California and author of several studies on wine tasting. "I'm not intent on debunking the whole idea of tasting and judging wine. On the other hand, the tasting and judging of wines in the setting of a wine competition, in my view, leads to meaningless recommendations."
His motive for this work is personal. He wondered why his wines would earn gold medals in some competitions and nothing in others. He has been studying the results and, as a member of the governing board at the State Fair competition, the inner workings, for nearly a decade. "Year after year, the answer was that the variation among panel members could be mostly explained by the inability of judges to consistently grade replicate samples of wine poured from the same bottle."
The California State Fair judges are drawn from retailers, restaurant owners and wine directors, winemakers and others in the trade, and a few writers. This is fairly typical of American wine competitions in general. These are not people who taste blind, for publication, on a regular basis.
"I often point out emphatically that my conclusions do not apply to … persons who may be involved in rendering evaluations of wine," Hodgson said. "What I do say is that based on an enormous amount of data, that the results of the State Fair, and by extrapolation to many other competitions, do not provide the public with reliable information with which they might make a rational buying decision."
In order to deliver reliable advice, judges—whether for wines, restaurants, movies or art—must have the experience to understand a work (or a wine) and put it in context, as well as a methodology that allows for fair and consistent evaluation. These are the standards we strive to meet at Wine Spectator.
However, wine is personal, and circumstances can significantly affect our impression of its character. So another person's experience with the same wine, in different circumstances, probably will be different. Also, wine is a moving target. Certainly, by the time a review is published and you drink the wine, it will have changed in the bottle, maybe a little, maybe a lot.
Despite all that, I believe it's important for a review to provide as thorough a description and as precise a reflection as possible of how much I like it. This gives us a baseline, and it makes it possible to compare the wine with others reviewed at the same time, or from the same region. And it gives readers a point of comparison with other critics.
A professional tasting note is a good-faith effort to describe a wine, and place it in context with others that you, the reader, might consider buying or drinking. Nothing more, nothing less. It's not perfect, but it's not B.S. either.
Vince Liotta — Elmhurst, Il — July 4, 2013 2:59pm ET
Robert J Sawicki — San Anselmo, CA — July 4, 2013 11:44pm ET
Robert Hight — CA — July 5, 2013 6:45pm ET
Glenn Keeler — SoCal — July 5, 2013 7:50pm ET
Mahesh Lekkala — Livingston, NJ, USA — July 6, 2013 9:16am ET
Harvey Steiman — San Francisco — July 6, 2013 3:08pm ET
Harvey Steiman — San Francisco — July 6, 2013 3:18pm ET
Kelly Carter — Colorado — July 10, 2013 1:48am ET
Jason Whyte — Cape Town, South Africa — July 10, 2013 3:09am ET
Karl — New York City — July 10, 2013 5:17pm ET
Steve Coleman — San Francisco, CA — July 12, 2013 3:36pm ET
Harvey Steiman — San Francisco — July 12, 2013 4:16pm ET
Terry French — Columbia, MO — July 23, 2013 12:18pm ET
Sips & Tips | Wine & Healthy Living
Video Theater | Collecting & Auctions