Log In / Join Now

harvey steiman at large

Behind the B.S. About Wine Tasting

How critics of the critics miss the point
Photo by: David Yellen

Posted: Jul 3, 2013 3:12pm ET

The news media has jumped on a new study of judges at the California State Fair wine competition as evidence that all wine tasting is baloney. But the author of the study says that's not what he meant. Not by a mile.

In the study, tasters sampling the same wine, hidden in several successive blind flights, could only get within 4 points of their previous scores on a 100-point scale. That's the difference between a gold medal and something less.

I reached Robert Hodgson, the study's author, by e-mail in South Africa, where he is making a presentation on testing judges at wine competitions. That's what the study was about: The difficulty of getting agreement among several different tasters who don't normally sit down and sample wines with each other.

"I am not against tasting and reviewing wines in general," said Hodgson, owner of Fieldbrook Wines in California and author of several studies on wine tasting. "I'm not intent on debunking the whole idea of tasting and judging wine. On the other hand, the tasting and judging of wines in the setting of a wine competition, in my view, leads to meaningless recommendations."

His motive for this work is personal. He wondered why his wines would earn gold medals in some competitions and nothing in others. He has been studying the results and, as a member of the governing board at the State Fair competition, the inner workings, for nearly a decade. "Year after year, the answer was that the variation among panel members could be mostly explained by the inability of judges to consistently grade replicate samples of wine poured from the same bottle."

The California State Fair judges are drawn from retailers, restaurant owners and wine directors, winemakers and others in the trade, and a few writers. This is fairly typical of American wine competitions in general. These are not people who taste blind, for publication, on a regular basis.

"I often point out emphatically that my conclusions do not apply to … persons who may be involved in rendering evaluations of wine," Hodgson said. "What I do say is that based on an enormous amount of data, that the results of the State Fair, and by extrapolation to many other competitions, do not provide the public with reliable information with which they might make a rational buying decision."

In order to deliver reliable advice, judges—whether for wines, restaurants, movies or art—must have the experience to understand a work (or a wine) and put it in context, as well as a methodology that allows for fair and consistent evaluation. These are the standards we strive to meet at Wine Spectator.

However, wine is personal, and circumstances can significantly affect our impression of its character. So another person's experience with the same wine, in different circumstances, probably will be different. Also, wine is a moving target. Certainly, by the time a review is published and you drink the wine, it will have changed in the bottle, maybe a little, maybe a lot.

Despite all that, I believe it's important for a review to provide as thorough a description and as precise a reflection as possible of how much I like it. This gives us a baseline, and it makes it possible to compare the wine with others reviewed at the same time, or from the same region. And it gives readers a point of comparison with other critics.

A professional tasting note is a good-faith effort to describe a wine, and place it in context with others that you, the reader, might consider buying or drinking. Nothing more, nothing less. It's not perfect, but it's not B.S. either.

Vince Liotta
Elmhurst, Il —  July 4, 2013 2:59pm ET
To me (and for me) the biggest challenge to evaluating wine is CONTEXT. What mood? What occasion? Which people are you tasting with? What other wines are being tasted? What food? What time of day? What time of year? What place (geographical)?

And then add the reality of bottle variation, not just two bottles of the same wine tasted at the same time, but the variation which occurs over 2 months, 4 months, 6 months. . .

Tom
Robert J Sawicki
San Anselmo, CA —  July 4, 2013 11:44pm ET
You cannot ovrrestimate the importance of context.
Robert Hight
CA —  July 5, 2013 6:45pm ET
Your last paragraph should be a required disclaimer for those who make their living as you do. What's important is to find a critic who has a palate similar to yours. For me, it's James Laube. I can't recall any wine we both have tasted and he reviewed that we differed by more than 3-5 points. You and I are nearly that close as well. Context? Mood? Occasion? The same bottle of Caymus Special Selection will taste no different whether it's at Del Friscos or Denny's.
Glenn Keeler
SoCal —  July 5, 2013 7:50pm ET
Mr. Hight, do you really think your bottle of Caymus will taste the same with a grilled rib eye from Del Friscos as it will with pancakes from Denny's?

Mr. Steinman, I've enjoyed many bottles that when first opened and tasted were just ok or holding back a bit, but with an hour of air and a good food pairing really changed the wine dramatically for the better. Most recently a bottle of 97 Brunello was thin and acidic on opening, but after some air and with food the wine showed amazing sweet fruit and complexity. I'm sure you have experienced the same. How do you account for that when tasting a small pour of a young wine in a flight of 10?
Mahesh Lekkala
Livingston, NJ, USA —  July 6, 2013 9:16am ET
Unless you are professionally tasting the wines and concentrating on just evaluating them, in my opinion it does make a difference on where you are drinking the wine and with whom you are sharing.
Harvey Steiman
San Francisco —  July 6, 2013 3:08pm ET
There is no doubt that setting, glassware, temperature and company affect our own personal experience, even if the wine itself doesn't change. When we make judgments, it's important to keep that in mind.

The classic example is the lovely local wine we drank over a delicious lunch of grilled fish and vegetables on vacation with our sweetie in Mallorca. The same wine never tastes the same at home, after a long tough commute.
Harvey Steiman
San Francisco —  July 6, 2013 3:18pm ET
Glen, your experience with the 1997 Brunello demonstrates that a wine that has been aging for years needs time to regain its equilibrium after being opened. Young wines on release usually aren't so closed-up, but I do set aside any that seem like they just need time it to open up. Occasionally, when I retaste one at the end of a flight (mine are usually more like 15-25 wines), it can show a different profile. If it does, I either adjust the tasting note (if I haven't already removed the bag, so it's still blind) or bring it back in another tasting and have it decanted first.
Kelly Carter
Colorado —  July 10, 2013 1:48am ET
I agree with Mr. Hight's observation that finding a reviewer who shares common palate traits with your own can be invaluable in effectively using the notes offered by the reviewer. It isn't that one WS reviewer is better than another, but palate alignment helps make the tasting more consistently enjoyable in my experience.
Jason Whyte
Cape Town, South Africa —  July 10, 2013 3:09am ET
I believe that a large part of the problem is the public's fixation with the precise points given to any particular wine, rather than treating the points given as a general indication of quality to be considered along with other considerations, such as whether you actually like the style of wine in the first place! This is particularly important as the consumer will (generally speaking)not be tasting blind, and will thus be influences by any number of other factors, not least of all that of how much he or she paid for the bottle in question!
Karl
New York City —  July 10, 2013 5:17pm ET
I think most comments are missing the point. Hodgson's analysis is not about whether a wine changes its taste over time or with the food you eat. It also doesn't discuss whether a wine should get 91 or 93 points.

Hodgson's analysis is very simple and straightforward. At the CA Wine Fair, tasters taste 3 identical wines, poured from the same bottle, at the same time. That is, influences of food, time etc. are ruled out. There is virtually nobody that assigns the same quality rating to each of these 3 wines. In fact, only 20% of all judges stay within a two-medal rank (e.g., gold-bronze). That is, 80% of the wine judges rate identical wines all over the place. And the judges (the 20%) who were relatively consistent in one year, were inconsistent the next year. Here is a link to the original analysis. http://www.wine-economics.org/aawe/wp-content/uploads/2012/10/Vol.3-No.2-2008-An-Examination-of-Judge-Reliability-at-a-Major-U.S.-Wine-Competition.pdf
Steve Coleman
San Francisco, CA —  July 12, 2013 3:36pm ET
Seems to me Hodgson's findings (which I''m not surprised by) at least somewhat generally discredit the 1976 Paris tasting that put California on the map.
Harvey Steiman
San Francisco —  July 12, 2013 4:16pm ET
Steve, I always thought the important point about the 1976 Paris tasting wasn't that California wines "won," but that the French tasters could not tell the difference. It showed that the wines belonged on the same quality level.
Terry French
Columbia, MO —  July 23, 2013 12:18pm ET
A woman came up to me in the Italian aisle of the wine shop in which I work and excitedly said, "I just returned from a trip to Tuscany, and I had some wonderful Chiantis. Are these going to taste as good?" I smiled and replied, "Not a chance. You were having dinner in a sidewalk cafe in Siena, watching the Italians walk by in their evening finery, eating terrific Italian food, and being waited on by a handsome, young Italian waiter. No, these aren't going to taste as good."
Also, when rating wines that we tasted as part of my job as a wine consultant, I used a six-point scale for my own reference. 0 for a wine that was not good (0 to 80 on Wine Spectator's scale), 1 for a wine that was decent but uninspiring (81-84), 1+ for a wine that showed some interesting aromas and flavors (85-87), 2 for a wine that showed depth and complexity (88-90), 2+ for a wine that was great showing depth, complexity, a long finish (91-93), and 3 for a truly outstanding wine (also, the number of Clydesdales it would take to pull the glass from my hand.) Six levels seemed to be enough. Face it, most consumers could not tell the difference between and 89 wine and a 90 wine, but they will certainly chase the wine with the 90.

Would you like to comment? Want to join or start a discussion?

Become a WineSpectator.com member and you can!
To protect the quality of our conversations, only members may submit comments. Member benefits include access to more than 315,000 reviews in our Wine Ratings Search; a first look at ratings in our Insider, Advance and Tasting Highlights; Value Wines; the Personal Wine List/My Cellar tool, hundreds of wine-friendly recipes and more.
Most Recent Posts

WineRatings+ app: Download now for 340,000+ ratings.