I’ve started to see a statement crop up more and more in blogs, on Twitter, and in person on discussions about what does and doesn’t work in becoming agile: Anecdotes are not data. Typically, this happens because someone is explaining their views or practices as derived from their own experience. You’d think that wouldn’t be a controversial move, but agile coaching/consulting is one of the most armchair-dominated fields out there. It behooves people trying to make money in this field to cast aspersions on experience in favor of their “data.” You can tell story after story about what you’ve run into in the field, but it’s not “data,” and therefore can’t be used to establish anything.
Here’s the catch, though: where does that “data” come from?
Let me tell you where it doesn’t come from. It doesn’t come from a researcher, an organization, or a group of peers coming up with a hypothesis, defining the control parameters to test that hypothesis, then running experiments on a broad range of companies within the confines of those parameters, observing and recording the effects every day, then publishing those results after peer review. So, whatever agile consultants mean by the word “data,” they don’t mean what the scientific method means. They don’t mean actual empirical data.
Since no one has ever done an actual scientific analysis of various agile practices, what do people mean when they talk about data in this field? Well, they mean things like the Chaos Report or various organizations’ “State of Agility” surveys. In other words, they are referring to collated survey data, and unfortunately for the anti-anecdote wags, a survey response is anecdotal. People are reporting on their own experiences, and the way in which those experiences are reported is further filtered and conditioned based on things like the questions asked, the questions not asked, the answer options, the quantification mechanism chosen, the response rate, and the survey base.
So, data in the agile community is typically nothing more than a collection of anecdotes. One might argue that these collections of anecdotes yield more certainty than, say, one person’s collected anecdotes. Maybe that’s so, but at what point do anecdotes magically transform from “Bob’s experience” to “actual data?” Is it five people’s anecdotes? How about twenty? A hundred? A thousand?
The only difference between anecdotes and data in this context is one of degree, but not one of kind. Critiquing someone’s anecdotes because they aren’t data is like critiquing a full bathtub because it isn’t a swimming pool – maybe you can’t fit everyone into it, but there’s still a lot of commonality. All these various reports and surveys are just collections of stories. That doesn’t make them invalid, of course, because these stories are really all we have to go on until a hundred corporations volunteer themselves for experimentation by the agile community. But it does mean personal experience has both meaning and weight.
Even in what we think of as the hard sciences, the experiences behind anecdotes carry value. Do you think a heart surgeon who has never done surgery is just as good as a heart surgeon who has done a hundred surgeries because they both learned the same material? If the hundred-surgery surgeon started giving advice to the no-surgery surgeon about what the real deal is like, do you think, “Sorry old man, but anecdotes aren’t data” is a valid response?
If someone is being hidebound and closed-minded because of their past experiences, I think a much better observation to make is, “Your experience isn’t normative.” In other words, people may have experienced different things that lead them to different biases, actions, and even different ways of interpreting “data.” For instance, not too long ago, I criticized the popular use of burndown charts on the grounds of what I’ve actually seen happen. I would never, however, tell someone that burndown charts are universally inappropriate or should never be used under any circumstance. Why? Because my own experiences have led me to see them as a low-value activity, but another practitioner might have experienced a lot of benefit from them, and it serves the community as a whole for us to share those anecdotes and the views they espouse so that we can improve and the people who bother to listen can also improve.
Anecdotes are data, and if you discount what an experienced veteran says because it didn’t jive with someone’s survey, you’re taking a very risky path.