Log In / Join Now

Massive Tastings Have Diminishing Returns

Photo by: Greg Gorman

Posted: Feb 18, 2009 4:07pm ET

I’m often asked how many wines I taste a year and, for a long time, the answer has been 5,000.

That sounds like a lot, particularly to consumers and those not in the business. And it is. But when you break it down, that’s about 100 wines a week, and I try to taste 20 wines per day on average. Some days it’s a little more. Some days I skip.

I’m curious about what you think constitutes a big tasting. How many wines a day, or week, or month do you try to sample?

I mention this because I came across a story online about reviews, rating systems, scores, etc., and one of the publications mentioned was Wine Spectator. And at the end of the story, one of the persons interviewed offered a conclusion: "The problem [with tastings]," the person quoted said, "is when a critic tastes 100 wines in a morning, the bigger, bolder wines inevitably stand out and get bigger scores. The methodology stacks the deck against delicate wines."

No one at Wine Spectator tastes that many wines in a morning, and I doubt any of our tasters hit 100 in a day. Wine Spectator's blind tasting procedures aren't a secret.

So that’s one of those urban legends that circulates about tastings. People who do taste 100 wines (or more) in one sitting are pushing human limits and, in my experience, the margin for error in judging wine quality is greatly increased.

Jim Gallagher
Jim Gallagher —  February 18, 2009 6:39pm ET
James you have posed a great question. I think of it in terms of reliability of the taster ordering preference of wines compared. My experience has been that as the number of wines compared in a given time frame, say an hour or two, increases reliability in ranking diminish. I have tried from six to 20 wines. For our seminars we now use eight wines at a tasting and most can reliably discriminate, that is make forced choice rankings that are (relatively) consistent.
James Rego
Redding, Ca., Shasta County —  February 18, 2009 7:15pm ET
I envy you in your job but it must be difficult, at times, to remain completely objective. Nonetheless, your recommendations always seem sound.
Jason Thompson
Foster City, CA —  February 18, 2009 9:12pm ET
I am not a professional, but from the blind tastings that we do in our family/friends, anything over 12 wines in one sitting gets lost in the shuffle. So extrapolating, I think I could handle around 3,000 year taking weekends off and a couple weeks of vacation. Don't know though, a couple weeks of that could get old...NOT!
Jim Mcclure
DFW, Texas —  February 18, 2009 11:04pm ET
I'm not in the business, so wine tasting is a purely after hours and weekend pursuit. I may taste two during the week, and I go to my favorite wine bar a couple of times a month where I usually try a flight of five or six. With friends occasionally, I may taste up to a dozen on a busy night. A couple of times a year I go to a large walk around tasting, where I taste a couple of dozen wines, but by the end my palate is shot. The most, best day I've had was in Napa, where I tasted upwards of 20 in a day, spread out over several hours, all spit out with lots of water. No major effect on my palate that day, which was nice, though I wouldn't want to make any official concise judgements by the end either :-)
Apj Powers
Dallas, TX —  February 19, 2009 1:47am ET
4-6 is optimum for me at any one sitting. I have done many more. The recent 2006 Bordeaux tour comes to mind. I just do not believe I can accurately judge more than 10. And I do believe elegant & nuanced wines lose out in the mass tastings.When sales reps visit, I usually pass on 2 or 3 b4 we start. Even then, I feel a few swirls & sips in 5 minutes or less is not fair to the winery (although really bad wines stand out immediately). A recent tasting of a Rhone emphasized this last wk. Not too fond during rep visit, but he insisted I take it home. That night had a full glass, liked it, looked it up & JM had scored it 92. That shows both sides of the discussion. He probably evaluated many at once & I only a few. Shows who the real pro is. Guess I have a ways to go
Jason Gullion
February 19, 2009 10:50am ET
Hi James,Your discussion of how many wines you taste per day and per year has put me in mind of another question. After doing an advanced search here on the site, it appears that no California wine has ever received a 100 point score (I find three 99 point scores). Is this accurate? Has there really never been a 100 point California wine in your estimation? If so, I think this begs some interesting questions about the state of California viticulture and winemaking (are we not quite there yet?). Or coming at it from another direction, I'd love to hear your views on the subject of wine perfection.
Scott Collier
Texas —  February 19, 2009 11:40am ET
A very helpful post to demystify Wine Spectator's tasting process - thanks for that! As a new niche wine producer (we just submitted our first Rockroom wines to WS) it is good to see a number like 20 wines in a day - that should allow for catching subtle characteristics just fine. I personally enjoy a single tasting session with 5-7 wines as beyond that, especially with bigger reds, I personally get a little palate fatigued and risk missing subtle characteristics. But having spent time with more experienced tasters at trade events I think that, depending on the wines, time allowed and water/food breaks, 20 in a single tasting session is reasonable to still catch the intended complexities. I recently did a few chardonnay barrel tastings in the morning, tasted through a bunch of pinot samples right after lunch and then did some cab tasting a few hours later that afternoon, so 30-40 different wines in the course of the day. That's about all I would want to try and still rely on my opinions. And I like Powers' comment above about returning to questionable wines - first tastes are not always a good indicator.
James Laube
Napa, CA —  February 19, 2009 11:55am ET
Scott, I do trust first impressions. But I'm not a slave to my first sip or first wine. I prefer to keep the wines bagged and taste through them a couple of times if I have doubts. But usually after five or six wines I find my palate is tuned up and in sync. And painful as it sometimes is I taste questionable wines twice.
Dennis D Bishop
Shelby Twp., MI, USA —  February 19, 2009 12:25pm ET
When I have my friends over for wine tastings we rarely do more than 8 wines, and no more than 6 reds.
Morgan Dawson
Rochester, NY —  February 19, 2009 1:43pm ET
James,You're sort of attacking a straw man here. You don't link to the story in question, but regardless, the larger point is that there are plenty of people who are concerned that bigger, bombier wines tend to stand out in any multiple-wine tasting format. You didn't address that contention. I'm not saying I share it, but I am curious to hear your thoughts on it.
James Laube
Napa, CA —  February 19, 2009 1:46pm ET
Morgan, I think bigger "bombier" wines do stand out. But they don't necessarily get better marks than delicate, more elegant ones. One does have to pay close attention, though, to both excess (as in over-ripeness) and finesse.
Morgan Dawson
Rochester, NY —  February 19, 2009 4:55pm ET
James - Thanks for the frank remarks. I appreciate the quick response, and it's great to see WS putting so much information forward regarding wine reviews. Cheers.

Would you like to comment? Want to join or start a discussion?

Become a WineSpectator.com member and you can!
To protect the quality of our conversations, only members may submit comments. Member benefits include access to more than 315,000 reviews in our Wine Ratings Search; a first look at ratings in our Insider, Advance and Tasting Highlights; Value Wines; the Personal Wine List/My Cellar tool, hundreds of wine-friendly recipes and more.

WineRatings+ app: Download now for 340,000+ ratings.