Software Engineering: Scaling Agile: Emerging Architecture and Technical Debt

In Agile projects including large-scale projects, emerging architecture offers immense benefits and it can save lot of big-upfront efforts. That is an advantage. However, emerging architecture can lead to technical debt and significant rework when there is lack of orchestration in terms of timely decision making, validation and dependency management. I am writing this post to discuss how emerging architecture and technical debt or interconnected and how even if you ensure emerging architecture there is a possibility of accumulating significant technical debt.

As discussed in my previous posts (see ‘Related Posts’ at the end of this post), we need a great deal of communication, coordination and collaboration among business users, architects and designers to create and maintain a prioritized product backlog and provide groomed user stories to feature teams. That is about the four gears and they need to work synchronously. When they are disconnected or out of rhythm, one or more of the following can happen.

Architects make decisions with a high-level understanding of business requirements. But those decisions may not take into account any recent changes in business requirements.
Solution Designers and Architects do not come together. They depend on assumptions or design-by-contract decisions that are not documented and agreed up on.
Decisions are not validated to firm up feasibility by constructing and demonstrating PoCs.
Architects and designers let features teams move forward for implementation by feature teams with placeholder decisions or designs. Later they firm up their decisions and leading to significant rework by feature teams.

These are obviously the symptoms of disconnected teams or teams that lack communication, coordination and collaboration as shown in this diagram. Yes. When they are disconnected, they do not produce a backlog that can optimize technical debt and reduce rework.

So, what do we need to do when we work on large-scale programs? Here are some tips.

Look ahead architecture and design is important. Let your architecture and design mature and be ready to cater to at least 2 to 3 upcoming iterations. That is not big upfront design.
Construct architecture prototypes or PoCs to validate your architecture and design decisions. Do not pass it on to feature teams if you want to reduce rework and technical debt.
Let us assume that you have four or five iterations per release in your project or program. Most probably there will be a hardening Sprint at the end of releases. And there will be some architecture or design work that may cascade into rework by feature teams. Do not let this ripple effect happen your hardening Sprint. When you do this you will see code changes across all layers leading to an upsurge in defects. One way to avoid this is by firming up your architecture and design decisions at least two iterations or Sprints before your release ends.

Have you been part of large-scale programs and attempted to optimize the accumulation of technical debt through emerging architecture and designs? How have you accomplished this? Would you like to share some practices or tips here?