Competitive market drives modernization
For the last 7 years, Levi9 has been successfully building and maintaining a partnership with a client that provides Core Banking Software. During this time, we helped a lot of banks run their daily operations smoothly by using the software we created and following the processes we established.
As this product was growing and technology trends were changing, it was becoming more challenging to keep the quality, processes, and organization at the desired level while maintaining a healthy business. It became obvious that we needed to improve and modernize in order to stay ‘alive’ on the extremely competitive market of core banking solutions.
With this in mind, we gathered a group of experts to assess the current state of the product and create a proposal on the best solution for its future. After a 6-week-long discovery phase, it was agreed to build a completely new cloud-native cutting edge technology Core Banking Platform. The idea was to help banks by giving them a system that is flexible, easy to work and integrate with, but still robust and secure. The solution is based on an event-driven and extendable architecture. The SaaS platform is built from scratch and it takes advantage of a broad range of Azure Cloud services.
What is SAFe®, and why did we choose it?
In addition to technical expertise, the decision to build a could-native banking platform from scratch required a new way of organizing work on a larger scale. We examined several popular scaled frameworks and opted for SAFe® based on criteria such as productivity, quality, time-to-market, seamless integration, reduced costs, and focus on security.
Scaled Agile Framework or SAFe® provides principles, practices and competencies for achieving agility in large-scale software projects. It draws its ideas from Lean, Agile and DevOps and arranges them in a coherent whole which helps teams and other stakeholders make sense of their place in the bigger picture. It also facilitates, planning, execution and evaluation of software development projects. It has built-in quality practices ensuring a fast flow across all levels and agile teams.
Having in mind that this project includes 8 scrum teams working on one product from four different counties in three time zones, the complexity and dependencies are at a high level. SAFe® provided a mechanism to keep the teams aligned with the overall product roadmap. The continuous collaboration gives the teams a high level of transparency — they clearly understand the requirements and the deliverables and in turn, can manage the expectations of all stakeholders.
Specifically, because SAFe® was designed to maintain a big picture view of software (product) development, it can easily handle a coordinated strategy for large-scale and complex projects, with teams that number into hundreds. However, because it is rooted in agile and lean principles, it remains more efficient than traditional approaches to software (product) delivery.
The flavor of SAFe® we opted for is Essential SAFe®, as it contains the minimal set of roles, events and artifacts required to deliver the solutions.
How we implemented Essential SAFe®
The key element of SAFe® is the Agile Release Train (ART) tasked to deliver one or more desirable, feasible, viable, and sustainable solutions or parts of a solution. The ART’s long-lived, flow-based, self-organizing nature is what powers SAFe® and enables Business Agility. These trains are virtual, spanning organizational and geographic boundaries; others follow a line of business or product line management reporting structure.
SAFe® roles
The following SAFe® roles have been positioned to facilitate the fit-to-purpose product implementation structure:
- Product Managers: Represent the internal voice of the customer and work with customers and Product Owners to understand and communicate their needs, define system features, and participate in result validation. They are responsible for the Program Backlog.
- Architects: The group of three people in charge of creating and governing the approach to solution development, incorporating all aspects of a system and its environment into the design, development, deployment, and maintenance of the system itself.
- Agile Teams: Cross-functional groups of 5–11 individuals who can define, build, test, and deploy an increment of value in a two-week-long time box, supported by a Scrum Master and a Product Owner. There are seven development and one system team working on our platform.
Next to the SAFe® roles, for the purpose of this project’s implementation, following roles have been established:
- Release board: The purpose of the Release Board is to decide whether a release meets defined quality gates and can be released to Customer/Production environments.
- QA guild: The guild consists of all Quality Assurance engineers on the project and is led by a Quality Master. The goal of this guild is to share good testing practices, enhance the knowledge about the domain and identify improvement points when it comes to the quality of the platform.
- Steering committee: The committee consists of Scrum Masters, Product Managers, and C-level representatives. They meet bi-weekly and decide on the priorities or order of business of an organization and manage the general course of its operations.
Program Increments
Program Increments (PIs) consist of 4 to 5 two-week Sprints that deliver an incremental value in the form of working, tested software, and systems. For PIs, Product Owners of each team together with Product Managers define priorities for teams based on the anticipated capacity and past performance of the team.
PI ends with two-week Innovation and Planning (IP) iteration. During that period, we finalize refinement of the work for the next PI, organize knowledge sharing sessions and work on innovation and technical debt.
6 events in the IP iteration
1. PI Kick-off session
IP iteration starts with a Kick-off session where planned features, enablers, and infrastructure improvements for the upcoming PI are presented. Features are presented by the Product Manager, Enablers by the architect, and infrastructure improvements by the Head of DevOps. In this meeting, every member on the platform has an opportunity to see what the plans and goals of the platform for the next 4–5 sprints are.
2. Retrospectives
The Retrospective is used to acknowledge what we did well and define points to improve. Once we implemented SAFe®, we started with Retrospectives on a platform level, which means that members from each team, each Scrum Master, Product Owner, and relevant stakeholder were in the meeting. After a few retrospectives in this setup, we concluded that this was not the most efficient way of working. The reason was that not all the teams faced the same challenges, and it was unproductive to keep everyone engaged in such a big group.
To mitigate the challenges, we replaced these big Retrospectives with Retrospectives per team at the PI level. We start with Team Health Check — a specialized self-assessment tool that uses a set of common dimensions that teams can use to evaluate and discover the current health of the team. Based on the discussion, we create conclusions and actions per team.
Later on, Scrum Masters of each team gather to collect inputs, define mutual topics and discuss what actions are required from people outside of the Agile teams.
3. Refinements
After the Retrospective, the entire first week is reserved for team Refinements. Product Owners are defining Epics for each team throughout the PI and bring partially refined Epics to the IP refinement meetings. A JIRA epic represents a software feature. During the refinements, teams clarify requirements and break them down into stories. The goal of team refinements is to make stories clear and ready to be picked up in the upcoming PI.
In addition to team refinements, we organize cross-team refinements. They help us define cross-team dependencies at an early stage and create a timely plan to address them.
4. Planning
Planning is done based on the velocity of the previous 6 Sprints. We use historical data to predict what amount of work can be pulled into Sprints.
Once we know our baseline capacity and have clearly defined stories we want to work on, together with Product Owners we organize stories per Sprint, based on the Product Roadmap and anticipated team capacity. In Sprint backlog, there is 60% of the capacity is reserved for features and 40% for technical enablers.
5. Confidence voting
Confidence voting is not an event that is part of SAFe® framework by the book. We introduced it to help us identify the risks that the teams recognize as threating the execution of the PI.
Team members vote on a scale from 1 to 5 (1 — not confident at all, 5 — very confident) and thus state how they feel about their commitment. Team members who give lower marks explain why they are not confident that they’ll be able to deliver the scope for the coming PI. The risks can include uncertainties, complexities, bottlenecks, etc.
After risks are identified, action points are created and assigned so we can manage the risks and stay on track with the PI scope.
6. System demo
We finalize our IP weeks with a big system Demo. On that Demo, each team shows what they completed in the previous PI. Participants of the Demo are team members, Product Owners, Scrum Masters, Product Managers and relevant stakeholders. In this meeting, we collect feedback on our work.
Challenges in implementing SAFe®
Overcoming challenges isn’t easy, but it’s often rewarding. Listed below are the most important ones with our solution to each.
Changing mindset
Years before we adopted SAFe® as a framework, we worked in Scrum. Once we started with SAFe®, the biggest challenge was changing the mindset from planning Sprint by Sprint to planning the entire PI upfront. When it comes to a big change, everyone needs time to digest it.
To help each other adapt to this change, we introduced a Kick-off session at a platform level, where the Product Manager explains to everyone the roadmap and goals of the coming PIs, so that everyone knows that we have overall goals and what these goals mean for the platform and for the teams themselves.
Aligning dependencies
Architecture in our software is built with the intention to create teams that would be as independent as possible; however, in reality the teams are dependent on each other to some extent. Additionally, we work in a remote, geographically dispersed model from 4 different countries and 7 different offices worldwide.
To overcome this challenge, we set up cross-team refinements. Once we identify a feature that impacts the work of more than one team, a cross-team refinement is organized to identify dependencies upfront and clarify who needs to do what and when. Cross-team refinement is facilitated by a Product Owner who leads the feature, and participants are members of the teams that will work on it.
To stay on top of dependencies we use Jira dependency maps — that help us visually represent dependencies, follow their progress, and adjust to changes if needed.
Setting up the release train
The release train is one of the essentials of the SAFe® framework. SAFe® says that the release train departs from the station at the same time in each iteration and picks up those who are ready to go. If someone is not ready at that point, it means that they need to wait for the next train. Our train departs every Friday. We use Feature flags to help the train move on schedule and manage internal dependencies.
Communication
Over the course of two years, the number of people engaged on the project has been constantly growing. We are now at the points where we are working with 7 different teams, in four different countries and four different time zones. Setting up the optimal communication structure is an ongoing challenge and our response to it has been changing along the way as we search for the best fit for any current situation.
At some point, there were Team Lead meetings and big-group dependency checks, and now many of the alignments are taking place on Scrum of Scrum meetings and the weekly PO sync. This is serving the current purpose, but if the situation on the project changes we will make sure to adapt to it to keep the flow of information as fluent as possible without putting too much communication pressure on any single group on the project. All the meetings are listed on a Confluence page and everyone has access to the meeting notes of interest. This gives us the opportunity to clearly know the place where we can communicate our challenges, dependencies, and ideas for improvement outside of the team.
SAFe® benefits
Once implemented, SAFe® framework gave us more than one valuable benefit.
Clear roadmap
SAFe® as a framework ensures a clear roadmap on what we are going to do at least 4 sprints ahead. Usually, we have a high-level plan for the coming PI. This helps the teams understand the big picture and the goals we want to achieve. To assure everyone is aware of the platform roadmap, Kick the tires meeting is organized during the PI, to present planned features and enablers for the next PI.
Continuous delivery
Continuous delivery is achieved via the release train scheduled in the same, commonly known, and agreed release frequency.
All teams are in the same iteration cadence
SAFe® gives the opportunity to set all of the teams in the same rhythm. This means that the sprint for each team starts at the same time, has the same length and the same end-date. Everyone has the same release schedule and clear integration moments. This helps us keep all teams aligned and helps with planning and execution of the plan.
Enablers
SAFe® as a framework identifies Enablers as an entity. This means that activities needed to improve the existing code, components and technical infrastructure to improve future business functionality are treated like every other value-added development activity. They are the subject of estimating, visibility, and tracking which increases the quality of the product. In our case, 40% of work in each PI is focused on enablers.
There’s no one-size-fits-all solution
Implementing a new framework can be a rocky road for the entire organization, as it requires changes and adaptation at all levels. To experience the benefits of SAFe®, it is important to set up a product architecture that enables scalability and minimizes the inter-teams dependencies. SAFe® has a built-in mechanism that allows to see a bigger picture and how the product is going to look in the future. This is one of the main advantages to be used.
What we have learned in the past two years of SAFe® practice on project is that there is no silver bullet. However, there is a constant need to listen to the project organization and teams, as well as to identify the challenges and adjust the framework implementation to targeted improvements. If some predefined structures are not the best fit, find a mechanism to tailor them to your project needs, monitor the outcomes, and improve accordingly.
We hope you can learn from our experience. Good luck in finding your own solution!
Special thanks to Levi9 Delivery Managers Dejan Ačanski, Mira Dulić and Dragana Bodroski for successful SAFe® implementation!
