Scrum is beginning to emerge as an increasingly popular choice for organizations seeking to improve software quality, decrease time-to-market, and stay competitive. Evolving in tandem a new generation of agile project management software and Scrum training providers like Danube. DZone recently caught up with Michael James, a software process mentor, team coach, and Scrum trainer at Danube who is also the author of the recently published DZone Scrum Refcard. In this interview, Michael talks about some of the adoption challenges and ultimate benefits of taking a Scrum-oriented approach. What is the optimal Scrum team size and why do some Scrum teams radically outperform others? What is the role of the product owner with respect to the project manager, and the Scrum master? Michael answers these questions and more.
The complete transcript of the interview has been provided below.
DZone: Michael, could you please tell us a little bit about some the work you're currently doing at Danube?
Michael: My job right now at Danube is to help software development organizations have breakthroughs, and the tools that I use for that are the Scrum framework itself, which is a good starting place, the Agile engineering practices - so engineering practices that lend themselves to iterative development - and what we've been learning about team psychology.
DZone: What are some of the driving forces behind an organization moving towards a Scrum-oriented approach?
Michael: Companies are interested in Scrum -- they're [also] tired of suffering. They're tired of having no idea where they are at the moment. They're tired of missing deadlines. They're tired of poor quality. Their customers are tired of it also, and traditional approaches to managing have only worked up to a point, and their competitors are figuring out better ways of managing their projects. These are typical complaints that will motivate people to be interested in Scrum or an Agile approach.
DZone: Would you say the motivations for adopting Scrum are the same for both the developer and the business unit? And if not, how would you differentiate the needs of those two groups from a Scrum perspective?
Michael: Well, everyone wants to succeed. That's what people want. But they see it from a different perspective. From a developer's perspective, and that's my background, I'm interested in doing Scrum because it's more fun, it's more rewarding, the work I'm doing. I'm on a cross-functional team. I'm learning from other people who have different skills. The work I'm doing is more aligned with the values stream, so there's less BS, in other words. Also, I'm liberated from micromanagement when I'm doing Scrum. I'm on a team we're a self-managing team. So me and my peers are managing each other rather than playing the boss-worker game. So that's the motivation typically from a developer, and if we're doing it right, we make it more fun and more rewarding for the developer.
From the business perspective, why would the business be interested in Scrum? The business has an opportunity to build the high value features first. Now this comes at a cost, but we build the high value features first, and always know where we stand, so we're always within one iteration of a shippable product. That gives us the ability to prioritize in ways that we haven't been able to before.
So the additional visibility about where we are with a product, additional control over the features, also additional scope control, and then of course early warning -- if things are going awry, we can find out sooner. Typical motivations from the business perspective to do Scrum.
DZone: What would you describe as some typical Scrum pitfalls?
Michael: A lot of the suboptimal implementations that we've seen have come out of the implicit assumption that the project manager should be the Scrum Master. It's not necessarily true. It's not necessarily untrue, but we should question the assumption of that, because the Scrum Master role is quite different from a traditional project manager. A lot of the project managers are actually better suited to be Product Owners, are better suited for the Product Owner role. Now, the way that this issue can show up, at the team level, some pitfalls would be the failure to take it to the next step. Now Scrum is just the first step of a journey, and the next steps for the team probably include learning engineering practices which lend themselves to iterative development, which are different than the ones that we learned in university, typically.
We need to get testing involved much sooner than before. We can't impose quality at the very end. We need to be thinking about requirements all the time, continually evolving understanding of that, and the architecture as well. The architecture and design are so important, that rather than do them at a phase at the beginning; we're stretching that out over the entire development cycle. So we're doing every iteration contains a mix of all these activities. So that's a typical pitfall.
Reluctance to collocate, to get people in one team room, or to think that we can get the same results with people spread out around the world. That would be a typical team level pitfall.
At the Product Owner level, a common pitfall, or maybe sub optimization anyway, is not aligning our product backlog items on business value. We still see product backlog items which reflect waterfall thinking. So we're not getting the full value of Agility. We're not getting the full value of the Scrum framework if we are uninformed on how to create product backlog items and collaborate with our team on creation of those, work with the stakeholders on those.
At the organization level, well, there are lots of things that organizations, and especially large organizations with a lot of entrenched habits can do to put in the way of teams. One example would be, D do we organize our teams along architectural components or on business value features?
There's a reasonable amount of evidence now that if we give a team responsibility for a feature all the way through, soup to nuts, from beginning to end, give them ownership of that, even if it spans architectural components, we're likely to get a better result than from teams organized on architectural boundaries that are doing components.
There's something called Conway's Law that suggests that the organization of a software system tends to resemble the organization that created it, the org chart of the company that created it, and that's probably not aligned with the optimal software organization.
DZone: You mentioned not making the assumption that a Scrum Master is necessarily the project manager. In teams where you do have a Scrum Master and a product owner, or project manager, how do those two interact? Also, what are the dynamics of that interaction?
Michael: The Scrum Master role we deliberately separated these roles. The responsibility for project management is split among the Product Owner, the Scrum Master, and the development team themselves. We want business people making business decisions, and we want technical people making technical decisions as much as possible, we want to separate those things. So the Product Owner is the person who makes those business decisions. The Product Owner makes decision about priorities, what's important in the business, how do we prioritize the product backlog in particular? And the Product Owner may not give us detailed requirements, but they are the final arbiters of requirements questions.
The Scrum Master's role: well, management within the constraints between the Sprint planning meeting and the review meeting, that's the teams' responsibility. The team is self-managing, self-organizing. How do we create the circumstances for that? That's part of a Scrum Master's responsibilities.
So the Scrum Master’s easier to say what the Scrum Master doesn't do than what he or she does do. The Scrum Master does not make business decisions. The Scrum Master does not make technical decisions. The Scrum Master does not commit to work on behalf of a team.
Now, what does a Scrum Master do? It's easier if I can get you for at least two days and teach you through hands on for some examples of what they do. I've written about this question when it came to me a while ago from a Scrum Master who wasn't using her imagination thinking, what could my responsibilities be as Scrum Master? Do I just organize meetings? And the result of that was a checklist, a list of things I wrote for her in that particular example, in that particular case of things that I thought she should be thinking about.
This took off and it'd been downloaded I don't know how many thousands of times now. It's been translated into Russian, it's been translated into Japanese now, I just found out. So this can help people who are wondering what the Scrum Master’s responsibilities might be.
DZone: Scrum takes measures to create an ideal team, cross functional members in groups of seven plus or minus two. In your opinion, why do some teams radically outperform others?
Michael: I think the main reason is collocation. Getting people together in one team room. I was afraid to try this in the beginning, because I'm the type of developer who needs to focus and working at three in the morning with total silence. Working in a team room is a lot better than that, but we have to try it to find out that. So the collocated teams are outperforming teams that are geographically distributed, whether it's in the same office, or even worse, with a planet sized impediment, the whole world is an impediment. So that's a big factor in team success, apart from just the skills of the people we get on there.
Now there are other factors to look at. Intelligence, raw intelligence has a value up to a point, and then that rolls off. Personality has a big factor, and we've been ignoring that up until now. Teams that have too few extroverts will underperform. Teams with too many extroverts also underperform. There's a lot of research now showing this. We need a heterogeneous team, definitely.
As you mentioned, in Scrum, we need a cross functional team. People sometimes forget that means that we need skills and testing on the team. Testing needs to be involved right from the beginning and continuously throughout development.
DZone: How long would you say does it take a team to reach peak performance?
Michael: The evidence suggests it's one to two years before a team reaches it's peak performance. One to two years. And then there's some evidence that around three years, three or four years, if we don't do something, if we don't mix it up, then they lose that edge. Then team performance, team ingenuity starts to drop off after that. In a large organization, we're usually lucky if we can get one to two years together, because our old habits, our traditional resource management habits have been to constantly reconstitute teams, move people around. And the irony is we're doing this for the sake of efficiency, but the result of that is that the team doesn't have a chance to gel. The team has to go through the forming, storming, norming, to get to that level of performance. It's an organic process, and it takes some time working together.
DZone: Michael, what would you say are some typical changes that an organization can expect to experience as they start scaling their Scrum adoption?
Michael: One thing that will change or should change if you're doing this right, we should see people working together more. So sometimes we can just look at a picture of what the company is doing, a physical picture, and we should see people in team rooms. We should see one of our clients, about two years into their Agile transformation, go back there and physically it's different. People are working together. They're collaborating. They're working in team rooms, getting the high bandwidth communication that's possible when people are together. So that's one change that we see.
If we're really doing the framework and we're doing the retrospective and we have courage to look at the things that are revealed by the constant retrospective process, then we should see changes in things like the HR policies. Are we incentivizing people to be individual performers, or are we encouraging group performance?
Another change is we'll have more visibility about the actual product itself. So we should see more "genchi genbutsu" which is "go and see for yourself," so we'll see more people getting out of their chairs, and instead of managing just by numbers, just by metrics, we should see executives, managers, stakeholders, customers, coming in to see the Sprint review meeting. So we actually will see physical inspection of the product that's being developed. Typical changes that we'll see.
We'll be breaking down silos in a typical large organization. We've got one department that's responsible for one particular discipline, and another department that's responsible for something else, and to do Scrum effectively, we've got to get people out of those silos onto one cross functional team, and this means crossing political boundaries, so it certainly has implications for our org structure.
DZone: Earlier, you mentioned certain personality types that are required to optimize Scrum team performance. In terms of building a successful Scrum team, what skill sets does a project manager or a software development manager need at hiring time to really select the right people to bring success to a Scrum project?
Michael: That's a tough one. We've done this a couple of different ways. The optimal way of doing it, and this won't work every time, but the best way of doing it would be to have the team themselves decide who would join them. We've had reports from one of our clients after doing Scrum for a year or so, they reported they would rather lose a team member from a given team than add someone. It's almost like antibodies form against foreign invaders that are coming into our teams. So if we asked the teams get their viewpoint on, who would you benefit from having? Who would you want to have on this team? We might get a better answer than we will that people don't have their hands dirty in the work might propose. So when we've done this right, the teams came up with answers that, even if they were the same ones that managers might have proposed, the fact that the teams have ownership of that makes a big difference in how the team is going to perform.
I've interviewed a lot of developers, and I'd say that most of them initially don't have the habits that lend themselves to Agile development. If we're going to transform our engineering practices aligned around interactive development, along the way we have to break down the habit of future proofing. We're doing the simplest thing that could possibly work and building the product in a way that allows us the most flexibility in the future.
This is different than I was educated. These skills aren't really taught in universities. Previous experience, like an Agile project, is a big factor as well in success. Of course, if we can get at least one person per team who does have these skills and can act as a mentor for the others, or we can get coaching to come in and help with those problems, that would make a big difference as well.
DZone: You mentioned an interesting topic, the notion of future proofing. Can you talk a little bit more about that and how it fits in? Is that a typical pitfall in software development projects?
Michael: Yeah. One of the key pitfalls here is, I'm a software architect, so my tendency to add in things that no one asked for, add capabilities that no one asked for, insuring against a future possibility that someone might ask for that in the future, that was a key thing I had to overcome, and it took years for me to really let go of that. That's an example of the kind of thing. Also, we're changing the rules a bit. Scrum is hard because we're picking a population where software engineers didn't typically get into software development because they liked human interaction. They got into this because they liked dealing with toys and technology and ideas and concepts, abstract ideas, places where they can be in complete control of their universe.
And now we're putting them in a team room and saying your effectiveness at building a done, done, done product increment is really going to depend on your ability to make this whole team work. So that's a new set of skills for people.
DZone: Michael, do you think postsecondary institutions training computer scientists of the future are doing enough to educate them on the Agile realities of the workplace today?
Michael: For the most part, they're not doing that. Now, we've been working with University of Washington here on an extension program to try and see if that can be handled in a university context, but the traditional computer science education is not lending itself to Agile development. The developers that we've seen do the best at this have had some real world experience. They've had some hands on experience doing this.
DZone: If you were to identify three core skill sets needed to make software development more Agile, what would those be?
Michael: The developers that we're seeing, greenhorns, I guess, they're thinking about a narrow specialty, and the programming and the design is just the fun part of software development, but it's not everything that needs to be done to get a shippable product out the door. We need to do product development right, we need to think about requirements, we need to think about testing as well. And to do this in Scrums means that we're doing a little bit of that every iteration. And it's the whole team's responsibility to do that.
Developers are not learning to test infect the software development process. The types of problems we're solving for school are a world of unchanging requirements, which is not realistic.
DZone: If I'm a waterfall organization looking to test the Scrum waters or even looking to try a different Agile methodology, how would you typically convince the decision maker to take that risk, or to move towards an Agile project or even move an entire department to Agile?
Michael: I probably would not start with an entire department. We have tried it a number of different ways, and we haven't seen one magic formula for success. I can give some general guidelines. It seems to work better to start with one team, and one team of volunteers rather than draftees, typically. And give us reiterations. Put seven smart people in a room and let's work with clear goals. This is part of Scrum, we pick out clear goals for each iteration. B, but give seven smart people in one room clear goals and give us reiterations, and if you're not astonished by the results, then we're doing something wrong.
The issues, if seven smart people with different skill sets that have negotiated their goals at the beginning of each iteration are not creating astonishing results, then it's likely that there's an organizational impediment which has been blocking us whether or not we decided to use Agile. One of the benefits of Agile is it brings these things to the surface so you've got to start dealing with them.
DZone: As someone who would approve the funding or the transition to a new approach, when could I expect to see results from the Scrum project, even if it is just an initial small experiment?
Michael: You should start seeing, if you do it the way we coach you to do it, within a few iterations you should start seeing a well, every iteration we're looking for a potentially shippable product increment. So rather than managing by reports and metrics, we can manage by physical inspection of the actual product. So right away we should know where we stand in a given moment. Every iteration we should see a potentially shippable product increment, and within a few iterations, we should start revealing the organizational impediments that are preventing us from doing that.
DZone: So, at the close of a Scrum project, what is the typical documentation in metrics that emerge?
Michael: Well that would depend on the circumstances. The appropriate method weight to use. Do we need a heavyweight approach or a lightweight approach? It varies based on the circumstances. We've got clients that are using Scrum to build videogames, and clients that are using Scrum to build medical systems, and the requirements are for documentation, for regulatory approval, things like that, are quite different. And that needs to become part of our definition of done so we can start measuring, so we really know what our velocity is very early in the beginning. So it will depend on the circumstances.
One thing that we found useful at Danube, which is a software development company as well, is the traditional metrics were not so useful. I've been advocating macro measurements instead. So instead of micro measurements of where every hour is spent, let's look at the big picture, because that will tell us more.
For example, the release burn down chart, or my preferred implementation of that, is a converging release burn down chart, it shows the rate of actual work being done, so potentially shippable product increment, how many product backlog items, how many user stories are in shippable form now compared to the rate of scope creep compared to the rate of new scope discovery.
Typically that reveals that the rate of new scope discovery outpaces the rate of feature completion somewhere between three and four times. Of course, this means right away we need active product ownership involved in the managing scope.
So that report can help. For example, and this is described in Mike Cohn's book. I've also written a white paper about this called "Macro Measurements" this is one measurement that I think is useful in most circumstances.
DZone: And Michael, what advice would you give to organizations that are looking to move to Scrum or that are just looking to learn a little bit more about it?
Michael: To learn more about Scrum, the best way is if we can get you for two days and have you do Scrum through immersion, we can step you through it, and it's a simple framework, but it brings up a lot of issues. So that will be the best way, if you can take a Scrum class. Other things which can help you learn about Scrum if you put your toe in the water, I've written on DZone, there's a reference card, which is a short illustrated guide to Scrum, this which can tell you something about what Scrum looks like.
Another resource I mentioned, the Scrum Master checklist, so if I know something about Scrum already, I'm on the Scrum Master role. This would be a diagnostic tool, which can help you uncover some opportunities for improvements of how you're doing Scrum.
Some great things have been written about Scrum. My favorite book about Scrum is still Schwaber's second book, the grey book, ""Agile Project Management with Scrum". And to get a quick summary of what Scrum is, go to the Appendix, start with the Appendix A, I think, is the Scrum Rules. Appendix B is the Definitions. Start with that, and then go back and read examples, so case studies of what Scrum has looked like in practice. Sometimes concrete examples work better than the abstract when you think about things.
Some other great things that have been written about how to do Scrum, this book is not about Scrum at all, this book is about team ingenuity. This is a good resource for Scrum Masters, "Group Genius" by psychologist Keith Sawyer. If I'm in Scrum Master role facilitating retrospectives, then I should be using this book. There's not just one right way to do retrospectives. So this is Agile retrospectives.
And maybe if I'm at the business level and I'm wondering, Is this really feasible? And we're talking about some radical changes to our organization, and some of them are different from traditional practices. Is this a safe thing to do? You can read an example of a large organization called Simcoe which wasn't doing Scrum, they were a manufacturing company, and they made some of the radical change that Scrum is likely to lead to, and they had success with that, so this book is about Simcoe, it's called "Maverick," I recommend this.
Other suggestions about how we run our organizations we have a lot of misconceptions about how human beings are motivated. This book, "Punishment By Rewards" by Alfie Kohn, has a lot of empirical studies, so a lot of controlled experiments demonstrating that people are motivated quite differently than we think they are. This has implications for how we set up our organizations, how we compensate people, our human resources practice. I don't like to refer to human beings as resources.
And if I've got a technical background, if I'm interested in airplanes and want to see a demonstration of the same principles behind Scrum, a book called "Skunk Works," a book about Skunk Works at Lockheed, the team that built the world's fastest airplane.
Now of course they weren't doing Scrum, they were building an airplane, many airplanes, actually. What was different about their development practices, radically different from traditional approaches, and this would be a good start for learning how the principle will supply.
So these are really universal principles about managing complex work or new product development. Scrum is a simple framework, which is based on those principles. It's not the only thing you need to learn, though, to have success at this.
DZone: Michael, on behalf of the DZone team, I want to thank you very much for your time today.
Michael: Thank you.