Most innovation is just stealing ideas from other fields. For example, peanuts used to be shelled by hand. Imagine a factory floor with hundreds of people doing nothing but shelling peanuts. Why would a person who works in the nut business know anything at all about canning vegetables? They wouldn’t. But one day (so the story goes) a fellow who owned a peanut business found himself visiting a bell pepper canning plant. He saw something amazing. Rather than core the bell peppers by hand, they had a big drum that they’d fill with peppers, seal, and pressurize. Bell peppers are slightly porous, so the pressure would rise inside the peppers to match the outside pressure. When they suddenly opened the drum, the pressure was released, and the bell peppers exploded, blowing out their cores. A pessimist sees a whole lot of peanut workers being out of a job soon. An optimist would invest in peanuts. I heard that story decades ago; so I might have gotten it backward.
I’ve worked in some very different industries, from prisons kitchens to marketing agencies and from event planning to competitive intelligence. So for me, sharing with the IT community something that everyone in another field already knows is the low-hanging fruit of blogging. But I never know when my observation will be someone else’s peanut epiphany, so why hold back? I did that two weeks ago when I shared something I learned from the world of TRIZ, and I’ll do it again today.
I was trained as an anthropologist. The most well-known tool of anthropology is participant observation, which seems simple enough. After all, it was discovered by accident. A fellow who went to school just down the road from the Krakow office where I’m writing now, went off to do a short study of people living on islands near Australia. It was 1914 and he, an Austrian subject, left Europe for his first big adventure abroad. As it was, something else really important happened in 1914. As an Austrian, Malinowski found himself on the wrong side of the war and had to stick it out with the Trobriand Islanders for rather longer than he’d expected. Years longer than he’d expected. When he came back and published what has become one of the most important texts in the annals of ethnography, he was rather sold on the idea of spending a lot of time in close contact with the subjects of a study. (In all fairness, Frank Cushing did something similar first, but it’s Malinowski who gets remembered for “discovering” participant observation because Cushing liked it a little TOO much and didn’t get around to writing most of his experiences down).
Why does any of this matter? I ran into the idea again in business school when I heard about the lean manufacturing practice of walking the gemba. That’s Japanese for getting off your butt and visiting the shop floor from time to time. A few years later, mostly thanks to Mary and Tom Poppendieck and later David Anderson, tech workers got keen to learn Japanese, too. Walking the gemba proved to be one of those lean manufacturing concepts that didn’t quite translate directly to knowledge work. For many IT managers it meant just being accessible to the engineers, which is rarely a bad thing, but they couldn’t really see how the work was being done the way a factory manager could. If they tried, following the “management by walking around” fad, the result was what one of my employees was overheard to call “management by looking over my shoulder.” The CEO of one of my companies took it to an extreme with his “flying office” which works great if your goal is to blur the distinctions between manager and employee (it was, in his case). All of these things, walking the gemba, management by walking around, and flying desks, allude to learning through participation. So just maybe some of the theories and practices of a hundred years of participant observation by anthropologists and sociologists might have something to contribute.
The first major distinction between walking the gemba and participant observation is that the latter is a research method guided by theory. The anthropologist has an idea about how things work, and they’re applying that idea to gathering data. The theory defines what is worth looking at and asking about and what isn’t. It makes sense of the complexity by imposing some constraints. A theory might be “social institutions emerge to serve survival goals” and so the ethnographer would concentrate on understanding social systems and their impacts on the life of the group.
In the 90’s, when the whole idea of objectivity in anthropology was being questioned and eventually dispensed with, theory got a re-think. Glaser and Strauss developed an approach to emergent theory called grounded theory (that isn’t as opposed to “ungrounded theory” but rather refers to developing a theory that is grounded in data), which very simply put is an approach to making sense of disparate data by finding patterns in it, and discovering a theory that can explain those patterns. I write about how I use it to analyze textual data in a post on one of my blogs.
So the anthropologist begins participant observation with a theory or with an approach to producing a theory and some research problem or question. These help them to know what to look for. Will they spend their days counting vegetables in personal gardens or mapping family trees? That’s why we have books like “Coming of Age in Samoa” and not a 10,000-page tome called “Samoa A-Z”. Would a problem, a question, or a management theory help to make time spent in the gemba more productive? Perhaps.
Next, anthropologists collect data, not just feelings and impressions. Their data is a mix of observations, interviews, polls, surveys, stories and artifacts. They take careful notes in the field and spend their evenings expanding on those observations while memory still holds. They understand that their ideas will change and evolve and that those ideas will shape their memories and so capturing data as accurately and promptly as possible is key to maintaining some bit of objectivity and to offset the risk of recency bias (wherein impressions are skewed more by recent observations than by earlier observations).
There are two fundamental rules of fieldwork that all good and decent anthropologists who don’t want to get drummed out of their cushy teaching jobs adhere to: informed consent and protection of their subjects. Informed consent means that the people we’re trying to understand know who we are, what we’re doing, how we’re doing it, and why we’re doing it and they give us permission to be there. Protection of subjects means that we take measures to ensure that no one gets hurt by helping us. Usually, that means anonymizing any data that could identify an individual. If a manager moves his desk into the dev room with the intention of watching the employees so he can stack rank them based on their work habits in preparation for the next round of corporate layoffs, but tells them that he just wants to understand their environment better, then he’s violating both of those core ethical rules. He’s misrepresenting himself with the intention to do harm. Please don’t do that.
Guided by a research goal and theory and constrained by a code of ethics, anthropologists entering the field begin by explaining their work and gaining permission. The permission can come initially from an authority like a police chief or village elder, but ultimately everyone who is interviewed or has any prolonged or close contact with the anthropologist has to understand that they are being observed and why and should have a chance to bow out if they think it’s creepy.
How might a manager explain their new desk in the team room? “I want to understand how things work here so that we can create policies that address real issues and work within the constraints you work under.” Or “I want to learn from you how the work gets done so that the process models that we use can better reflect reality.”
The first days in the field are mostly about getting settled and fitting in. The anthropologist pays attention to how people behave in much the same way a new student at school might be hyper-sensitive to social patterns to figure out how and more importantly, where, to fit in. When a new person enters a group from the outside, with the expressed purpose of gathering data for some cause, there will be people in the group who see that as a threat and others who see it as an opportunity. The anthropologist will be approached by people with their own agendas as well as by people who are naturally gregarious or curious. She has to start thinking about who to trust, who can provide deep insights, who has connections that could help and who is just pretending to have connections.
A manager entering the field will encounter much the same. Job titles and organizational charts of are of limited use when trying to determine the real pecking order in the dev pit. A junior dev may be the go-to person security issues just because she’s really easy to talk to and good at guiding the conversation in unconventional directions. You won’t know things like that until you’ve sorted the sycophants from the naturally helpful in the group and found your own place in the team and then eventually become so boring in your naturalness that you can sit quietly and watch.
Once you do start noticing interesting patterns, like the frequent visits to that junior dev’s desk, you’ll need those well-informed and well-intentioned informants to bounce ideas off of to test whether your conclusions are correct.
Aiming for accuracy, observation alone is not enough. Anthropologists in the field collect data in various ways. The make careful observations while in the course of participating in a group, but they also conduct formal and informal interviews, use questionnaires and take polls, take pictures and video, and look for objective data like who goes to lunch together, who plays foosball together, and who goes to whom for help, who is consulted on different types of decisions, and how often. They try to get multiple types of data to confirm or reinforce their conclusions.
One of the great things about informed consent is that since the people you’re studying know why and how you’re doing it, you can ask them to validate or question your conclusions. Key informants are usually people ideally suited to test ideas on, because they’re well-informed and you’ve spent a lot of time building up a rapport with them.
By its nature, participant observation results in subjective conclusions. It’s a research method that is difficult to reproduce. Two anthropologists spending time with the same people will arrive at different conclusions, neither of which is necessarily “wrong.” Ethnography produces no definitive answers. It’s used in the social sciences to provide context for other, more objective, data. For example, your HR department may have highly objective numbers on staff attrition combined with somewhat more subjective but still highly constrained data from exit interviews. They may paint a picture of a problem in one department without providing enough data for a policy decision. Fire the manager? Raise salaries? Start a softball team? Lay off troublemakers? The wrong decision could make the problem far worse. A month of working in the department and getting to know the staff there as peers could fill in a lot of the unknowns and suddenly HR’s data has context; it makes sense.
Participant observation provides the big picture that Clifford Gertz called “thick description.” Gertz described the idea of “thick description” best and quite cleverly when he introduced it using the example of a wink. Anyone seeing a wink for the first time might objectively describe it as a person briefly closing only one eyelid and then opening it again. That’s an accurate description. But how well you’d have to know a group of people and understand the social situation to distinguish between a lewd wink, a friendly wink, a sly wink, and a person making fun of another person’s lewd wink. The story that makes sense of the wink is the level of description that the ethnographer is aiming for, and it’s the level of understanding a manager needs to formulate policies that won’t be met with a wink and a nod.