Sunday, May 31, 2015

Scaling Agile: Hardening Sprint and the Problem of Procrastination

“Yes, we know that this is not done right yet. We have some rework to do and we have three more feature Sprints with a significant backlog. We will take care of this rework in the hardening Sprint.” 

Sounds familiar? 

What is a hardening Sprint? Why do we plan for a hardening Sprint?  Hardening Sprint is an iteration that happens before release readiness. It is a common practice irrespective of the methodology one follows. The term ‘hardening’ comes from ‘code-hardening’ – providing some finishing touches, making some minor adjustments to make things better on code base.  The goal is to improve structural quality and fix some defects too. It is not about delayed refactoring. Hardening Sprint does not include implementation of new features – obviously!  Hardening Sprint is to tie all the loose ends, fix defects and make sure that the product is ready for release. No new features but finishing touches, key improvements, and defect fixes. Period.

In large-scale Agile projects, even if we adhere to this norm, we know there are enough challenges. This is mainly because of what is involved in hardening the software created by multiple feature teams. And when we violate the purpose by introducing new features or major changes, we eventually face a whole lot of challenges.

When we go through a whole lot of challenges in a hardening Sprint because of reasons such as introduction of new features or user stories or major changes to architecture/design components or any change that rattles the compatibility of components, we must admit that probably it is because of the problem of procrastination.  Procrastination results in some of these -  insufficient planning or re-planning, ineffective prioritization, inadequate story grooming, weak design, postponed decisions, too many changes, fuzzy acceptance criteria, and so on. All these could make a hardening Sprint harder to execute.

When we are in feature Sprints to deliver user stories or features, we tend to give importance to things that are immediate and urgent (not necessarily important).  We tend to do quick fixes. We accumulate technical debt.  Eventually, we lose focus on long-term goals or vision.

“Giving up on our long-term goals for immediate gratification is procrastination.”  This is what Dan Ariely says in Chapter-6 of his book ‘Predictably Irrational’.  In this chapter he has given multiple examples on why we can’t make ourselves do what we want to do.

In large-scale Agile or projects cross-functional teams align and exhibit certain behavior.  Sometimes, the power of dissent is forgotten or ignored.

What can we do?  We can do many things to make a hardening Sprint meaningful and valuable.  Here some ideas.
  1. Identify the factors that make a hardening Sprint harder.  Learn from experience. Check if certain bad smells are repeating when you are working on a subsequent release. If yes, find a fix.
  2. Define the scope of Hardening Sprint 2 or 3 Sprints ahead in the release cycle.  Assess the need for any corrective action – such as one more feature Sprint or abandoning low-priority features.
  3. Assess technical debt (including testing debt – such as lack of adequate testing or lack of adequate automation).
  4. Assess environment issues that may lead to a backlog of work items in the hardening Sprint.
Try all these at the end of Sprints.  Don’t procrastinate and look at all these just before the hardening Sprint begins!

Saturday, May 30, 2015

Scaling Agile: Transitioning to Kanban

Large-scale agile projects include multiple feature teams working in rhythm over short iterations or time-boxes one after the other.  That happens when features for the first release are coded, integrated and tested in time-boxes.  When the first release or a subsequent release goes to production, we see an emerging need for forming one or two teams to work on daily iterations unlike 2 or 3-week iterations of feature teams.  This is because of the need to provide timely support and daily fixes to product owners and end-users.  Here, the transition from iteration-based or Sprint-based development into Kanban becomes essential.  Obviously, those one or two teams that support end-users cannot afford to make their stakeholders wait for 2 or 3 weeks for defect fixes and patches. They need to be even more agile, and they need to respond daily by fixing critical issues and responding to end-users.  Potentially, they must be able to understand and implement Kanban.
Transitioning to Kanban comes with two key aspects. The first one is about learning and implementing Kanban and the second one is about aligning the engineering processes such as configuration management, build and release management, tools etc.
When you and your team are transitioning to Kanban, you need to do a good mix of unlearning and learning.  This is because, when you do Kanban, you are not going to do Sprint Planning or Story Point Estimation.  You are not going to do daily stand-ups with the anticipation of delivering something at the end of the Sprint. You are going to learn about visual boards or more specifically Kanban boards. You are going to experience how team members ‘pull’ work items.  You are going to learn the concept of ‘Work in Progress’ and WIP Limit.   And of course, when you work with virtual teams, you will need tools that support distributed Kanban teams and offer electronic visual displays.
Coming to the second aspect on engineering processes, the first step is to think through configuration management.  How are you going to establish configuration management across multiple feature teams that synchronously work on 2 or 3-week iterations and Kanban based production support teams? An obvious solution is to have 2 branches – one for ongoing development and the other for production support activities. How frequently are you going to merge? And what is going to be the efforts involved in merging them together?   Next, how are you going to perform independent validation or testing in the new model? What are going to be the revised overheads or estimates? Finally, how will this new model impact your build and release management process?
Are you looking at all these aspects while transitioning to Kanban?  Are there any other key factors or issues?

Saturday, May 23, 2015

Scaling Agile: Disastrous Software Factories

Quite often, we hear the buzz word ‘Software Factory’ in Scaling Agile conversations and conference sessions.  Those who promote or sell the concept of ‘Scaling Agile’ attempt to indoctrinate their audience with the belief that Scaling Agile is a definitive way to setup an Agile Software Factory. Yes, when you add ‘Agile’ to ‘Software Factory’, it becomes even more attractive.  While there are several case studies and white papers about how to go about setting up an Agile Software Factory, the challenges involved in doing so are numerous.  Because of such challenges and several other reasons, initiatives to setup such factories become disastrous.  Why do such disasters happen? This post is to bring out a list of reasons.  Here is the list.
  1. Vision and Leadership Commitment - Lack of vision and leadership commitment is a primary reason.  Well, in many projects and programs there will be vision and leadership.  However, when the vision is not set right and shared among all stakeholders including team members and when there is no inspiring leadership and leadership commitment to support and nurture the vision, results can be disastrous.
  2. Investment in Coaching - Not investing in Agile Coaching is a definite recipe for disasters.  In large programs it is necessary to invest in coaching at different levels right from teams to middle and senior management.  Obviously, this means that you need a coachable ecosystem. Learnability is important. When there is rigid mindset and inability to learn and collaborate, cultural issues fuel disasters.
  3. Focus on Cost Savings Only - Just because your ultimate goal is to set up a software factory, your focus cannot revolve around cost savings all the time.  You cannot optimize your team members just as you optimize your machines in the production line.  You cannot replace them just as you replace your machines. You cannot operate in shifts to save costs. You cannot forget about people and their emotions.  Obviously, ‘that’ factory is different from ‘this’ factory.
  4. Physical Infrastructure – Lack of infrastructure including open workspace, video conferencing, phones, meeting rooms, computers with necessary configuration, network connectivity, tools, etc., is the next reason for such disasters.
  5. Intellectual and Emotional Infrastructure - Intellectual infrastructure is about skills, competencies, expertise, special groups, knowledge forums, technical excellence and so on. Emotional infrastructure is about team bonding, trusted leadership and the likes that enhance the feeling of belonging.  When intellectual and emotional infrastructure are not adequate you will see symptoms such as  loopholes in process implementation,  poor or weak automation strategy, poor execution, lack of technical excellence, etc. leading to pool quality.
  6. Inclusion - In virtual teams or geographically distributed teams, lack of inclusion can create chaos. We know that there are several challenges in distributed agile.  When there is lack of inclusion, you see one site dictating or directing the other sites, double standards, and so on.
  7. Governance - Though this comes as seventh, it is as significant as the first reason.  Governance can make or break things.  Collaborative, inclusive and supportive governance teams can do timely course correction and deliver value to customers.  When the members of Governance team do not have a unified agenda, they may pull each other in different directions. That is disastrous.
Do you see your team stretching or burning out? Is that becoming a trend? Are your team members demotivated? Are you calling your project or program a Software Factory? Probably you do not sit through with your team members and empathize on what is going on. You are not identifying the root causes. Revisit the seven reasons mentioned in this blog and see how you can make some course correction to come out of a disastrous Software Factory and make it a success. 

Saturday, May 16, 2015

Scaling Agile: Succeeding in Release Management

This post on ‘Scaling Agile’ is to present a list of critical factors to succeed in release management.  What makes large-scale Agile teams capable of ensuring successful product releases? Here is the list.
  1. Roadmap and theme for releases – Have a road map and theme for all releases or as many you can envisage.  This is one of the key responsibilities of product managers and sponsors.
  2. Tech debt management -  In addition to having measurable Definition of Done and Acceptance Criteria to measure the success of Sprints, have a consistent focus on technical debt management.
  3. Don’t let a hardening Sprint result in a chaos!  - Plan hardening Sprint well and execute them to preserve the integrity of your product release.  Don’t let a hardening sprint result in a big mess leading to an unexpected upsurge of defects.
  4. Test Automation – The strength or weakness of test automation will reflect directly on the team experience, challenges and results of release management.  Keep this in mind.
  5. Risk-based Testing – It is important to focus on risk-based testing by identifying risk prone areas or epics or modules and testing them first.
  6. Continuous Integration - You may have continuous integration working very well in development and staging environment.  Make sure that continuous integration is applied in other environments too (e.g. Testing, User Acceptance, and Production).   When you don’t do this, you are going to carry out several manual steps to complete integration. That is going to be time consuming and error prone.
  7. Continuous Delivery and DevOps – Do you want to increase the ‘speed-to-deploy’ in product environment? Think two things – Continuous Delivery and DevOps.  These will improve efficiency and result in cost savings.
  8. Defect Management - Is your defect management process helping you in slicing and dicing data to understand product quality? Does it allow you to predict? Does it allow you to spot missing test cases?  Or are you relying on a basic defect management process that helps you understand defects by priority and severity alone?  Effective (and efficient) defect management is an enabler for release management.
  9. Dependency Management – Understand all dependencies, categorize them and manage them effectively. Do not allow any dependency to create a big surprise and impact release schedule or quality.
  10. Communication & Coordination and Stakeholder Management – These are the non-technical but essential aspects not only for release management but for all software engineering activities. Make sure that the way you communicate and coordinate among teams enables osmotic communication. Also, keep your stakeholders informed of what to expect and how release management activities are progressing. Manage their expectations well.

      Friday, May 1, 2015

      Scaling Agile: The Value of Independent Testing

      In Agile projects of all sizes including large-scale projects, software testing plays an important role.  In our industry, there is a school of thought that proclaims that there are no developers or testers in an Agile team and everyone is a ‘team member’. Also, it challenges the need for independent testing.  My intention is not to argue on that but to share my thoughts on the value of independent testing in this post.
      First, in reality we do not develop standalone applications every time. A vast majority of software applications or products we create are complex with dependencies or integration needs on internal or external systems.  They involve multiple technologies and compatibility requirements too. Second, nowadays, many projects involve two or more cross-functional teams.   As the number of teams increases the situation becomes complex in spite of cross-functional teams focusing on the ‘Definition of Done (DOD)’.  It is not enough if multiple cross-functional teams deliver working software that satisfies DOD.  We need an integrated build that works!  We need an integrated build that satisfies all functional requirements.  We need to make sure that it satisfies all non-functional requirements as well – everything related to performance, security, integration, and so on.  We need an independent testing team or independent verification team to take care of this.
      That is obvious, known and make sense. However the questions are, 
      1.       What should be the size of this independent testing or independent verification team?
      2.       When should we introduce this team - in the first iteration itself or closer to the first release?
      3.       Do we need this team in all iterations throughout the project?
      4.       How can we justify the costs?
      5.       What can go wrong even if we have an independent verification?
      The size of the independent team typically varies between 10 to 15% of the total team size. For example if the size of your large-scale agile team is 200, that will include an independent testing team of 20 team members.  That is just a guideline.  And this team will include automation experts who focus on test automation for end-to-end functional tests, performance tests etc. The size of the team will increase based project complexity (multi-browser, multi-device, complex architecture, complex business rules), dependencies, non-functional requirements, target users (number of countries, regions, languages, etc.).
      ‘When should we introduce this team?’ is a great question. It is very important to explore test strategies and identify an optimal one.  Your test strategy exploration needs to address several questions - What is our need?  What are the possible approaches to meet our needs? What are the tools and techniques? What are the measures? What is going to be the governance?  What are the pros and cons of strategy A, B, and C?  When you do this you will get a clear idea on when to introduce this team.
      ‘Do we need an independent verification team throughout the project?’ The answer is ‘Yes’. You have multiple cross-functional teams that deliver features in short iterations.  You need an independent verification team to test the integrated builds.  You need them all the time.  How do we justify costs? Well, don’t you have to test the integrated builds? Of course, yes.  Independent verification directly relates to the cost of quality.
      ‘What can go wrong even if we have an independent verification team?’  Good question. Several things can go wrong.   First, feature team can become lax on delivering tested code to meet DOD or product owners may relax DOD to speed up feature delivery assuming that independent verification team can find all defects.  With that you enter a vicious cycle and independent verification team will slog in finding all kinds of defects.  Feature teams will do firefighting to fix defects and that will potentially open more defects. This is a huge risk on release quality as well as timelines.  Second, with no or weak test strategy, independent verification team may lose traction on test automation.  Third, with lack of focus on fundamental testing principles, the team may dilute or miss traceability and coverage. There can be many other things that can go wrong.  Please feel free to add those in your comments.
      ‘Are we going to have an Independent Verification Team? That sounds like waterfall!’  You must have heard this too. It is not waterfall. It is large-scale Agile.  All team members are involved from the initial stages. The goals are clear. They follow Agile principles and practices.  Teams do not operate in silos. Their timelines are not squeezed.  And they don’t compromise in any way in terms of quality to make a release happen!