How we evaluate and adopt new technology

This is the first of two articles on how we adopt and evaluate new technology at TrueLayer. This post is an introduction to technology governance at TrueLayer: the next will be an in-depth case study on one of our riskiest technology choices to date – the adoption of Rust.

• "Why don't we try out NewTech instead?"

You’ve been there before.

You’ve heard that question.

And you know what happens next: that dreaded discussion on whether or not to adopt new technology.

NewTech's advocates sing its praises, explaining what a leap forward it would be compared to the OldTech currently being used. Skeptics, instead, minimise the advantages, stressing pressing deadlines and existing OldTech expertise within the organisation.

These debates can be stressful — emotions are often running high and the fallout can be felt for weeks if anybody leaves the table feeling unheard.

At the same time, you should be having plenty of these conversations in a healthy engineering organisation: you cannot improve if you shy away from scrutinising your tools and processes. In a sizable organisation, it is bound to happen on a daily basis — both at the micro ("Should we use this library?") and the macro level ("Should we migrate from AWS RDS to AWS Aurora?").

TrueLayer is no stranger to this challenge and we have done our own experiments — some succeeded, others did not.

This article details our current approach to technology governance. We were inspired by the documented experience of other organisations – we hope that sharing our learnings can help others who are facing similar problems.

Seeking clarity

We're making a decision: should we use NewTech or not?

Before going any further, there are other questions to ask: what are we optimising for? Who is impacted by this decision? Who gets the final say?

We're doomed to fail if we don't have a clear and agreed upon answer to all those questions.

Animosity often stirs up as a result of an unclear decision-making framework – each participant fills in the gap as they see fit and is left puzzled or raging when reality doesn't match their expectations.

At TrueLayer, when it comes to technology adoption, we've had constructive conversations by framing the discussion in terms of risk.

Each line of code is a liability

Several different technologies are required to bring a complex product to life.

To succeed, everything must line up – you need enough expertise in every single technology you choose to adopt and in the way they interact with each other. Complexity is combinatorial!

For each technology, we try to quantify the risk that we're taking onboard.

It's useful to break it down into categories:

• Known knowns, a.k.a. facts: work that will be necessary to successfully use this technology in our product, according to the knowledge available within the team

• Known unknowns, a.k.a. known risks: areas of uncertainty that the team is aware of but chose not to investigate because they're not considered necessary for a successful delivery

• Unknown knowns, a.k.a. hidden facts: additional work that will be necessary that the team is not aware of, but information and help can be found in the wider community around this technology

• Unknown unknowns, a.k.a. unforeseen risks: nobody knows about these, neither your team or the wider community. You will be on your own.

A good understanding of product requirements, coupled with extensive discovery, can help a team to minimise the number of hidden facts – they will either become facts or be de-scoped as known risks.

There is very little you can do about unknown unknowns – if you are the first to take a particular technology into uncharted territory, there is no way to forecast what challenges you might have to overcome (or if you will be successful at all).

This is the core thesis behind Choose Boring Technology, a popular talk from a few years ago: it takes a long time and broad adoption to reduce the number of unknown unknowns for a specific technology, therefore you should be wary of introducing novelty into your tech stack.

Choose Boring Technology puts forward a pragmatic proposal for technology governance: each project should have a finite number of innovation tokens to take a chance on newer technologies; once you run out of the tokens, you must rely on battle-tested technologies to fulfil the remaining requirements.

Technology pyramid

Innovation tokens sound like a sensible approach, but the devil is in the detail: how many innovation tokens does a new database cost? What about a programming language?

It helps to break down and visualise your technology stack as a pyramid.

You will only use a handful of technologies from the categories closer to the top (often just one!), while you will have several examples for the categories closer to the bottom.

The position within the pyramid roughly correlates with how expensive it would be to change your mind: swapping out an authentication library is going to be orders of magnitude cheaper than migrating your whole stack to a new Cloud provider.

We can therefore use the relative position of a new technology in the pyramid as a first-level approximation of its cost in terms of innovation tokens.

The pyramid helps us to determine who should be involved in the conversation – a team can autonomously decide to introduce a new library, but the adoption of a new database most definitely requires a broader audience. The CTO/VP of Engineering must be involved if we are discussing the adoption of a new programming language or experimenting with a new cloud provider.

Looking again at the technology pyramid, you may ask – is it ever reasonable to bet on a new programming language or a new cloud provider for a project?

Unique requirements are often the driver for early adoption – the new technology is a key enabler for a unique value proposition and adoption is seen as a safer route compared to the risks and cost of building and operating a custom in-house solution.

Looking back, the genesis and adoption of Erlang within the telecommunications industry is a textbook example. More recently, the migration from in-house data centres to public cloud providers.

Not all risk is tech-related

There is another variable in the equation that we should consider: people.

It's TrueLayer’s first Engineering principle – “[Software is] Made by humans”.

The impact on people should be a key factor when choosing whether to adopt new technology.

Does introducing a new database increase the cognitive load on our engineering teams beyond what they believe to be reasonable?

Do we have the time to properly train and upskill our engineers to work effectively with a new templating tool for Kubernetes’ manifests?

Considering the impact of technological choices on people, we can even understand frowned-upon practices such as resume-driven development – a bias to choose very modern technologies when embarking on a new project because they make a good impression when listed on a CV.

The average tenure in the tech industry is 3 years – it's perfectly rational, as an engineer, to try to make sure that your skill-set remains marketable over time.

Using these lenses, adopting a newer programming language or retiring an old database can be seen as an investment in hiring and retention, benefiting both parties involved (the company and its engineers).

People – their knowledge, their aspirations, their relationships – are active parts of the system that powers your product offering. Technology choices must take them into account.

Going forward

Organisations are not static – they are continuously evolving.

While the philosophy underpinning our actions might remain consistent, we do expect to see evolution in our technology governance practices, driven by the environment around us and the stage of our organisation – few processes survive the impact of scaling from 10s to 100s to 1000s of developers.

Even now, as we write this down, we are experimenting with something new – namely a company-wide, formalised technology radar.

We're also mindful of the structural gap between theory and practice – between how we expect an organisation to function and how it actually functions.

This article will be followed by an in-depth case study on one of our riskiest technology choices to date – the adoption of Rust.

Stay tuned!

We're hiring! If you're passionate about technology, take a look at our job opportunities