Measurement is never been an easy job. It’s not just about the metrics but measurement also acts as a motivation for the team to achieve a better goal.
There are two kinds of measurements commonly used: the measurement of a product and a project.
You may wonder why we separate it like this. It is because the measurement of a product is quite common in most companies, whether the product was developed in an Agile way or traditional way. Usually it includes following contents:
Bugs found in production (with severity);
Customer Rating (if applicable);
Customer Engagement: e.g. bounce rate, retention rate, conversion rate, duration, interval;
Number of Users: e.g. active users, new users and updated users;
Visits: e.g. page view/PV, unique visitor/UV, total visits;
Revenue: e.g. average revenue per user/ARPU, average revenue per paid user/ARPPU, monthly payment ratio/MPR, lifetime value/LTV.
All those measurements are data driven, thus they are objective.
But when it comes to measurement of a project, things are different. The measurement of a project combines two aspects:
how do we review the project team?
how do we review each individual team member?
Traditionally, we use Key Performance Indicators (KPI) or Return on investment (ROI) to evaluate the team, using KPI and manager’s opinion to evaluate team members.
In an Agile world, especially in a company like ThoughtWorks, which doesn’t have a product itself but delivers value to clients through product delivery, the measurements are combined.
Firstly, the product success metrics are aggregated into how we measure the project; unlike the traditional way, the measurement of a product and the project to deliver it are separated. That is to say, if the measurement of the product met or went beyond the client’s expectation, the team will get a high client satisfaction score. Hence, the project team is continuously monitoring product metrics , mostly by integrating them into CI pipelines and analytical platform such as New Relic.
In the meantime, the project team is also focused on delivering value to customers (end-user, stakeholders and the client’s company), which can also improve the client’s satisfaction. When implementing a feature, the team will always figure out the hidden problem to be solved and the value delivered when it’s done and then try to brainstorm all the possible solutions before actually working on it. This approach forces everyone to think about the value they bring to product and team in daily work, rather than just piling up an endless backlog and try to make themselves busy but never really think about the product.
Secondly, as part of the measurements, the client’s relationship is also taken into consideration. In this aspect, ThoughtWorks expects the team to provide the best solutions based on the experience and expertise, even if this does not align with what the client originally asked for. This benefits the relationship in the long run and from a big picture because the client understands that we are focused on the things that bring them value.
Thirdly, ROI or margin also plays an important role in the review. Every company’s primary objective is to survive, so only ‘Reasonable’ margin can help the company to grow sustainably. Why does ThoughtWorks think that the margin should be ‘Reasonable’ rather than as much as possible? That’s due to the rest of aspects we need to think about in the review listed below and then do a trade-off.
Fourthly, the influence on client is also considered to be an important aspect in the review. In most cases, ThoughtWorks is not only considered as a vendor to just deliver a product or a piece of work, but more importantly – leveraging the client’s team, bringing new thinking and technical expertise as well. To meet this expectation, usually the team conducts brown bag sessions, pair programming and knowledge sharing to influence more. As a result, measurement of influence is served as feedback to those actions.
Fifthly, the satisfaction of each team member is also a critical measurement among all the measurements. It takes great efforts to make everyone in the team happy, since everyone is eager to improve their own skills, to work on challenging tasks, to contribute more to the product, to work more efficiently, and, of course to not have to work overtime. This is why ThoughtWorks values health of both project and team members and believes it can bring more efficiency and energy to the team.
Sixthly, the leveraging of team members is important not just from team member’s personal perspective, but also as a company like ThoughtWorks. The more experience and expertise the employee has, the more challenge situation and project they can handle, thus the more influence on client and industry the company can have.
Last but not the least, how we change the world is also largely considered in the measurements. ThoughtWorks always gives the products and projects which can have more social and economic justice influence a high priority, even for sometimes, it means less profit.
In summary, ThoughtWorks assesses a delivery projects from 7 aspects:
- Client’s satisfaction and value delivered
- Client Relationship
- Margin
- Influence on Client
- Team members’ satisfaction
- Leveraging of team
- Social and Economic Justice
If you are familiar with ThoughtWorks’ three pillars Business Model (Pillar 1: Sustainable Business; Pillar 2: Software Excellence; Pillar 3: Social and economic justice), you will find the measurements just reflect the business models. The measurements 1 to 3 are aligned with Pillar 1, 4 to 6 are aligned with pillar 2 and 7 is aligned with Pillar 3.
To my experience, just like in Conway’s law:” organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.”; an organization is constrained to produce design the measurements of a project which are copies of the business model and culture of that organization.
What are the measurements in your company?