In-Depth

An Agile Primer

Agile development is growing in popularity for a reason: It results in faster time to market and, in most cases, better software.

• By Aaron Bjork
• 04/02/2012

Do you remember who the popular kids were in your high school? If you're like me, it was fairly easy to identify who was a part of the "in crowd" during your younger years. It's hard to describe what separated them from others, but in a subtle way it was obvious. The popular kids stood out. Others gravitated toward them, wanting to be included in the "inner circle." They were different.

In many ways, Agile practices are the "popular kids" of the software industry today. Most development shops have either tried them, are in the process of trying them or are planning to try them. Some shops are finding success; others are struggling to see what's different. Regardless, the popularity of Agile practices can no longer be ignored. They've arrived on the scene, and are becoming more and more mainstream.

If you're new to Agile software development, you'll want to start by reading and understanding the "Agile Manifesto."

In short, 10 years ago in Snowbird, Utah, a group of software professionals met to search for answers as to why some software projects are successful, and some are not. The result of that meeting was the creation of the Agile Manifesto: four value statements and 12 principles that define and guide the approach known today as Agile software development. The four pillars:

1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

The manifesto states up front that while there's value to each item on the right side of a statement (for instance, processes and tools), there's more value in the items on the left side of each statement (for instance, individuals and interactions). I won't comment more on the history of the manifesto, but will point out that the creation of the Agile Manifesto set the stage for the growing movement to adopt Agile practices. That movement has taken root and is now a mainstream part of the software development world.

Last summer, Dave West of Forrester Research Inc. published an article touting the rising popularity of Agile in today's organizations ("Water-Scrum-Fall Is the Reality of Agile for Most Organizations Today," July 26, 2011). In that article, West states: "Forrester has observed Agile changing from an approach practiced only by Agilists to one that many organizations employ. We continue to see Agile methods gaining popularity; in a Q3 2010 survey fielded by Forrester Research and Dr. Dobb's Journal, about 39 percent of 1,023 IT professionals said that they follow an Agile method." (See Figure 1 for a graphical breakdown of the survey results.)

[Click on image for larger view.]

Figure 1. Agile development is the most popular methodology today, according to Forrester Research Inc.

Evolving Environments
Let's first take a look at why Agile is catching on. What's driving the popularity? What's different? The place to start is by looking at the environments in which software is used, purchased and developed.

I clearly remember my software buying habits back in the late '90s and early 2000s: I'd drive to one of the "big box" stores and peruse the shelves, flipping over product boxes to stare at the screenshots on the back. I'd then glance at the "requirements" section to make sure my PC at home had enough RAM and hard disk space to keep up. After making a decision, I'd buy the box, drive home, and spend the next few hours installing and configuring the software for use. This was normal. This was how we made software decisions. A lot has changed.

I honestly don't remember the last time I bought a piece of software that came in a box -- or even on a piece of physical media, for that matter. It's just not the way things operate in today's world. Instead we download, we "try" for free, or we buy from our phones or tablets with the touch of a finger. What used to require a car trip and a cardboard box now requires simply an Internet connection. If I don't like the result, I try something else. Switching is easy. The barriers and investment that used to exist are gone. Again, a lot has changed.

Because of this change, anyone building software has been forced to look for new ways to operate. There's an increasing need to make better decisions and eliminate waste. Diego Lo Giudice and Dave West from Forrester Research said it best in their article, "Transforming Application Delivery": "Firms today experience a much higher velocity of business change. Market opportunities appear or dissolve in months or weeks instead of years. This increased business velocity demands much greater agility from business systems."

A need has developed for a new approach to building software. Enter Agile processes and methods. They're new, they're lighter-weight and they provide the perfect environment for building software in today's world. In the remainder of this article, I'll outline a few key factors I feel are absolutely necessary to be successful with Agile. I won't spend any time focusing on specific practices (daily stand-up, pairing and so on) or methodologies like scrum and extreme programming. Instead, I want to examine the environment an Agile team works in and lay out a few concrete steps necessary for an Agile transition.

Building an Environment
I live in the Pacific Northwest and have for all of my life. Yes, the rumors you've heard are true -- it does rain a lot. Despite the rain, we get a reasonable amount of sun each summer and generally have a very pleasant climate. A few years ago my wife and I decided to experiment with a vegetable garden. We had some extra space in our yard and it seemed like a fun experiment to see if we could grow something that we would actually eat. This experiment led me to build a series of raised gardening beds in our yard where our bountiful harvest would thrive. At least, that was the plan. I actually ended up building the beds twice after learning that treated lumber has high amounts of arsenic in it. Arsenic and edible vegetables are not a good mix. Lesson learned.

The point is that to successfully grow vegetables, it was necessary to create a healthy environment for the vegetables to grow. This included picking a spot in our yard that received the right amount of sun. It included creating a place for the garden and constructing a place for the plants to live. It also included adding rich soil to the garden and eliminating weeds that would choke out the plants. In the end, we were quite successful. We had more tomatoes than we knew what to do with, zucchini coming out of our ears, and enough lettuce and carrots to keep us going all summer. The garden was a success, and the environment we created was critical to that success.

I have no doubt that if we had taken a different and more cavalier approach, our harvest would have been significantly less. Imagine if I'd simply wandered into my backyard and scattered seeds in random places, "hoping" they'd grow. We might have ended up with a zucchini or two, but I'm sure the harvest would have looked much different. Moving to Agile is no different. You can't just say, "We're going to be Agile," and expect good things to happen. You've got to go out and create an environment where new practices and new thinking will work.

3 Motivations
So, where do you start in creating a healthy environment? I believe the No. 1 place is motivation. Agile is about the team: the people building the software. To build a great environment for teams, you need to understand what motivates them. This brings me to a book I read a few years ago by Daniel Pink: "Drive: The Surprising Truth About What Motivates Us" (Riverhead Hardcover, 2009). It was a book I couldn't put down; it captivated me and opened my eyes to some truths about motivation.

In "Drive," Pink contrasts traditional motivation techniques (rewards and punishment) with three new essentials critical to motivation in today's environment: autonomy, mastery and purpose. These three essentials lead to what Pink describes as "intrinsic motivation." The idea is that we all crave autonomy, mastery and purpose in everything we do. When individuals and teams find these things, good things happen. Let's define them.

1. Autonomy. The first essential presented by Pink is autonomy: The ability for one to make his own decisions. For a software development team, this is critical. A team needs to feel empowered, and a team needs to be empowered. Why? Because Agile teaches us that change is inevitable during the software development process. Learning about the product, the architecture and the customers will occur after the team has started executing. The team must feel empowered to react to change and communicate what it's learning. This doesn't mean the team should have the authority to do whatever it wants; rather, it gives the team freedom to do what's right. A team lacking autonomy is more likely to blindly follow a plan handed down from above while ignoring critical learning.


2. Mastery. The concept of mastery is fairly simple: People want to get better at what they do. We find this in all areas of life. Part of our basic DNA involves the desire to improve. Toddlers work tirelessly to master the art of walking. They usually need very little encouragement. By nature, they have a deep-rooted desire to master walking, and they work at it until it becomes second nature. This doesn't change as we get older. I love to play golf and I have a desire to be a better golfer year after year. Improvement is part of my love for the game. I'm certainly not a master (my career on the PGA Tour hasn't yet materialized), but I continue to strive to improve hole after hole, round after round, year after year. The process and the results are exhilarating.

   Software teams are no different. Every software team has a basic desire to find newer and better ways to accomplish the goal set before it. Members of the team love to build software and love to solve difficult problems. If they didn't, they wouldn't be in a difficult industry. The challenge and art of building an elegant yet functional solution drive them. Too often, however, we treat teams and the people on them as if they're pieces on a game board that can be moved, shifted and changed without consequences. We shuffle resources, change goals and randomize their priorities. By doing this we remove their ability to become masters. It's a pattern I see repeated again and again in the industry.

3. Purpose. The final essential laid out by Pink is purpose. Teams have a deep-rooted desire to understand how their work fits into the big picture. As an engineer, I want to understand why the software I'm building matters. We need a reason, and a purpose. As a leader in your organization, find ways to ensure that your team understands how the value it produces is contributing to the business. Communicate the importance of new features delivered in the last sprint. Show them delighted customers benefiting from their solutions. Too often we disconnect our development teams from the business and from customers. We do this because we think they don't care, or we're worried that we'll distract them. However, more often than not we end up alienating them and destroying their sense of purpose.

Less Is More
The second key to creating a healthy environment for Agile is to eliminate waste and unnecessary steps in your process. I'm talking about those things you've done forever, but nobody is really sure why. I often refer to them as taxes, because more often than not, doing these things feels like paying taxes. They are things that "must" be done (at least according to some handbook or list of rules somewhere), and yet they are things the team rarely understands, likely cares little about and receives very little value from. To the team, these tasks feel like a black hole. High taxes will kill an Agile team.

If you're part of the leadership team in your organization, this is where you take a long look in the mirror. What taxes are your teams subjected to? Why are they necessary? If you're having a hard time coming up with a list, I can guarantee that if you walk up to any member of your team and ask for a few suggestions, you'll receive a lengthy list. Most lists like this are filled with steps designed to reduce risk. Don't get me wrong: I'm all for managing and minimizing risk. It's absolutely necessary to ship high-quality software on time. However, risk-management techniques can quickly lead to a high-tax environment.

I've found in all the organizations I've worked for that the process the organization follows tends to "snowball" over time. You pick up a new guideline or rule in each release, and before you know it your teams are spending so much time checking all the boxes on the release checklist that they end up with little time to create value for customers. The goal should be to create an environment where your team members can maximize the value they produce. The process should be as absolutely simple and streamlined as possible. Try eliminating a few of the taxes on your teams and see what happens. If they prove necessary, add them back; or better yet, engage the team on how to solve the problem long-term. Whatever the case, ensure that you're not burdening your teams with unnecessary or outdated taxes. A simple process is always a better process.

Steps to Becoming Agile
Now that we've established a few key principles to building an environment for Agile to succeed, I want to outline a few clear and concrete steps you can take in your own Agile transformation. With the popularity of Agile increasing, there has been a corollary increase in the amount written on the topic. The fact that you're reading this article proves that point. However, among all that "guidance" it's often difficult to understand where to start. How do you actually put any of this into practice? The statements from the Agile Manifesto sound great, but what steps can you take to make them a reality? In the sections that follow I hope to lay out some practical guidelines you can use in your own shift toward an Agile approach to software development.

The Finish Line
The first step is to establish a clear finish line for everything your organization is doing. A finish line can never be crossed if it doesn't exist or is constantly changing. How gratifying would it be in a foot race if each time you approach the end of the race, someone ran out and informed you that the line had changed and was now another half-mile down the road? Crazy, right?

Things are no different with software teams. It's absolutely necessary that everything a team works on has a clear definition of "done" (or finish line). This doesn't just apply to project milestones or even iterations of work. It applies to everything from daily tasks to the completion of user stories from the backlog.

Try this exercise. Sit with your teams and talk about what "done" means for everything they do. When is a task from the backlog done? Is checking in the code enough? Does it require a code review by another peer? How about testing? Likewise, what does the finish look like for the next iteration of your current project? Does it include setup and upgrade logic? What about end-to-end testing? How about support and documentation?

This is important because, without a clear finish line, you won't have any semblance of predictability on your projects. Most of us are very good at creating and doing work. Finishing work is a lot harder. There are countless examples of projects that were started flawlessly but never finished because there was never a clear finish line.

Shorten Your Iterations
The next step to becoming more Agile is to shorten the iterations in which your teams work. I don't think there's a "one size fits all" iteration that works for every organization, so it's up to you to figure out the correct length based on your circumstances. There's no doubt, however, that shortening your iterations can greatly benefit your team.

For starters, shorter iterations help a team develop a habit of finishing what they start (see the previous section). Finishing a one-mile run is a lot easier than finishing a marathon. The problem is we often send our teams off to run marathons instead of more realistic distances. Start your teams with a few 100-yard dashes before advancing to longer races. The point is, you want your team and your organization to have an ingrained sense of the importance of finishing. Value can't be delivered until you're done.

Second, shorter iterations aid in achieving a higher yield on time spent planning. A traditional approach to planning requires spending large amounts of time up-front studying requirements, architecture and design options. This is done to produce a plan that will guide you to success. Without the correct initial plan, the project won't succeed.

Agile teams think differently. An Agile approach to planning also involves studying requirements, architecture and design -- but an Agile approach puts a stronger emphasis on getting started with the well-known and well-understood requirements and architecture

over continued planning. The expectation is that the team will derive greater value from starting on the known requirements than it will from continuing to invest in additional planning. Said differently, the value of any learning that results from getting started will have a greater return on investment (ROI) than the value from continued planning.

Figure 2 and Figure 3 illustrate this point. Notice how in Figure 2 ROI begins to diminish over time. This is because continued planning is based completely on assumptions made in the early stages of planning.

[Click on image for larger view.]

Figure 2. Traditional development processes see a high ROI early, but it quickly tails off.

Figure 3 shows an Agile approach. Planning begins in the same manner, but shortly after the planning starts it's stopped in order to begin execution. Why? Because the team anticipates learning more from getting started than it would learn from continued planning. After the team has executed on the well-known and well-understood, it returns to planning. This process continues throughout the duration of the project.

[Click on image for larger view.]

Figure 3. Compared to Figure 2, Agile development ROI remains high throughout the process, resulting in more efficient production.

Try shortening the iterations on your teams and see if you don't see immediate results when it comes to your investment in planning.

George Patton said it best: "A good plan executed violently today is better than a perfect plan executed next week."

Use the Front Door
Before a team can execute and finish work, they have to receive it. For an Agile team, the only way to receive work is through the product backlog. I call this the Front Door Principle, and it's absolutely critical to becoming Agile. Start by thinking of teams as occupying houses on a city street. The street represents the organization; the houses represent the teams. Each team member lives in a house on the street, and every house has a front door. There are additional openings on the house, just like a physical house -- windows, back door, garage and so on.

The front door represents the only way a team can receive new work. In Agile terms, the front door is the product backlog. When new work is created, it's placed on the product backlog and prioritized with all other known work. This helps protect the team from a few common distractions:

• New requirements can't be thrown at the team mid-iteration. If something new pops up that requires the team's attention, add it to the next sprint. If that's too long to wait, you're probably building good evidence to shorten your iteration length. The team should not be distracted from its commitment by new requirements.
• Second, the team is protected from wasting time on poorly groomed or incomplete stories. Every story (requirement) the team works on must be prioritized and groomed on the product backlog. There should be no exceptions to this rule. An incomplete or poorly groomed story almost always results in waste.
• Finally, using the front door helps build trust between the team and leadership.

  There's nothing worse to the team than conflicting priorities, and when new work and new priorities start to appear through side doors, the teams starts to lose trust in their relationship with leadership.

Pay Down Your Debt
Becoming Agile requires a commitment to getting out of debt. Not financial debt, obviously, but bug debt. It seems the software industry has fallen into a bad habit in which carrying weeks -- if not months -- of debt has become an acceptable practice. If you've followed the global economy over the past decade, you understand clearly the negative effects of carrying large amounts of debt. One of my colleagues here at Microsoft, Mario Rodriquez, coined a phrase during our own Agile transformation that has stuck with me: "quality for agility." Said differently, you can't be Agile until you've got quality under control. Simplified even more, it's imperative that your teams and products aren't dragging a mountain of bug debt behind them.

The traditional project management triangle we all know and love recognizes three constraints that apply to all projects: time, cost and scope. Shifting one impacts the other. Problems arise, however, when you introduce a fourth constraint: quality. It's very difficult to deliver value to customers when your features are littered with bugs. Quality should never be adjusted to meet time (schedule), cost (budget) or scope. Pay down the debt and fix the bugs. Your customers will thank you, and you'll be in a better position to move on to the next thing in your backlog.

Fine-Tune and Adjust
Finally, becoming Agile requires a desire by the team to improve over time. Agile teams use retrospective meetings to achieve this goal. Retrospectives aren't new to the software industry; they've existed as an improvement mechanism for years. A team holds a meeting at the end of a project or milestone to discuss successes and failures, then looks for ways to create more success and less failure in future projects or milestones. It's a fairly simple concept.

Agile teams in particular have embraced retrospectives as a way to drive continuous improvement into the team and the process. As stated in one of the 12 principles behind the Agile Manifesto, "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly." Let's examine this phrase in a bit more detail.

The first key statement in the principle is about regularity of reflection. The principle states up front that a team will reflect on being more effective at regular intervals. If the team "looks back" at random times in the product cycle, or only at the end of a long release, the value of any learning from the retrospective is diminished. I've participated in numerous retrospectives at the end of long release cycles that achieved nothing more than the creation of a long list of items that everyone wishes had been done differently (you might have these lists hanging in your offices, too). The problem with this type of reflection is that the scope is so large and so broad that very little of the learning is actionable. In contrast, regular time for reflection helps to establish an environment of natural, continuous improvement.

Scrum prescribes that teams perform retrospectives at the end of every sprint. The idea is to instill in the team that "becoming more effective" is a natural part of the team's behavior. Whether the sprint is one or four weeks long, the team reflects at the end of the sprint on how it can become more effective.

The second statement in the principle is a clear action item: the team will tune and adjust. It doesn't say the team will start over, and it doesn't say the team will experiment or radically change its approach.

These two words (tune and adjust) have particular significance because they imply a precise set of actions.

If you have a music background, you know tuning an instrument takes a well-trained ear. It's a precise action. You tune with precise movements and careful actions to ensure that the instrument produces the required pitches. To this end, a retrospective should produce a set of precise changes for the team. A retrospective should not result in drastic action that fundamentally changes how the team operates. Too much change, or too big a change, will only disrupt the team and get in the way of producing value for the customer.

Finally, and perhaps most important, the last few words in the principle state that the team will tune and adjust its behavior. It says nothing of the behavior of other teams; it says nothing of the behavior of the organization as a whole; and it says nothing of the behavior of anyone outside the team. This is a statement of personal responsibility. The team changes its own behavior. Be careful to avoid identifying all the external problems to your team. It's easy to blame poor execution on factors outside the control of the team. There is benefit to understanding those factors, but a common trap many teams fall into is blaming all their difficulties on those around them. Be sure to keep the retrospective focused on those things that are within the team's control.

It's clear that Agile methodologies and practices are no longer just buzzwords in the software industry. They've entered the mainstream and are helping teams realize the goal of producing more value for their customers. As you embark on your own Agile journey, remember that all this "Agile stuff" is very easy to say (or write about), but much more difficult to follow through on. That's not to say it can't be done; it can and is being done by software teams around the world. Start small, fine-tune and adjust as you go, and reflect regularly to ensure that you've got an environment built for success.