What is screen scraping and how does it work?

At its most basic, screen scraping is the process of collecting display information from a ‘screen’ (typically a webpage) to use elsewhere or to perform actions that the user would normally carry out.

But the technology and ethics behind screen scraping has made its usage quite controversial. The practice of sharing security credentials for banking portals in order to enable screen scraping has implications for user security, while even small visual updates can disrupt the user experience.

While it was once the only real option for collecting certain types of customer data automatically, other options have since grown in popularity. Screen scraping was previously a common practice for powering open banking use cases, but following changes in regulation, application programming interfaces (APIs) are increasingly replacing it.

This article explains how and why businesses use screen scraping, the benefits and pitfalls of the process, as well as how it relates to open banking.

How does screen scraping work?

Screen scraping works using a programme or ‘bot’, which gains access to a customer account and automatically captures the data on the screen in the background, without the customer being present.

Specifically, screen scraping works like this:

1. The customer shares their login details with a third party provider (TPP)

2. The TPP uses these details to log in to the customer’s bank account

3. The TPP then copies or ‘scrapes’ the customer’s bank data for use outside of the customer’s banking portal

In effect, the business carrying out the screen scraping is impersonating the user (with their permission).

A common example of screen scraping you may encounter is allowing a TPP to access and scrape your financial data as part of a smart budgeting app, so it can use insights from the data to suggest better ways to budget and save.

What is screen scraping used for?

Screen scraping has several use cases, which can be split into two categories. There are examples that collect sensitive data, where the user must share login or account information for a business to carry out the screen scraping (also known as credential sharing).

There are also several use cases that simply scrape publicly available information, powering comparison websites, verifying ad placements and translating information from legacy applications to modern applications.

When looking specifically at use cases involving credential sharing, common examples include:

To access and analyse bank account information: Probably the most common use of screen scraping, financial services can scrape a customer’s account information to log into that customer’s bank account and then collect the customer’s bank data for use outside of their app.

To initiate payments: This is an example of a business ‘performing an action’ rather than simply gathering data. Say a provider has consent to access your bank account. It could initiate a payment to another account. A smart budgeting app may need to transfer money into a different account you own to take advantage of a better interest rate.

Affordability checks: If a business wants to check your financial history and spending habits, it may ask you to consent to scraping your bank account for relevant information. For example, if you wanted to take out a loan, the loan provider may use screen scraping to quickly check whether you can afford the loan.

To store data for later use: A lot of credential sharing via screen scraping is about building a more complete picture of your financial footprint. A business may gather this data to store and use at a later date.

To steal data: while the majority of screen scraping is carried out by legitimate companies with the consent of their customers, cyber criminals can also use it to steal data from unsuspecting web users.

What are the benefits and drawbacks of screen scraping?

The main benefit of screen scraping is that it allows companies to collect customer information automatically and at scale.

But there are several drawbacks to using screen scraping to collect sensitive information:

It’s expensive to upkeep

From a business’ point of view, screen scraping is time consuming to maintain. Because the screen scraping technology needs to recognise every minor visual detail of a webpage, even the smallest update can completely disrupt or outright break the user experience. This can lead to the customer not being able to log into their bank or use other essential services.

Screen scraping doesn’t include data minimisation

While a customer may consent to their bank information being used for specific purposes, screen scraping doesn’t allow for data minimisation. Data minimisation is where a user consents for only the specific data that is necessary for a specific service to be accessed. Screen scraping generally pulls out any data that is on the screen, making it very difficult for consumers to control exactly what is accessed, and how it will be used.

Is screen scraping legal?

Under PSD2 — the EU law designed to increase competition in the payments industry — screen scraping is still legal as long as certain security steps are followed, including identification of the TPP to the bank it is accessing. However, most banks have moved to providing APIs to enable access to account data and payments, so screen scraping is not necessary.

There has been plenty of debate on whether screen scraping should be banned entirely. In the UK, most banks provide APIs for access, and some banks that have supported screen scraping have now been required to transition to APIs. In Europe, the European Banking Authority (EBA) has advocated to end the practice, but industry consultations are still ongoing.

What’s the difference between open banking and screen scraping?

Open banking is a way of giving regulated companies secure, limited access to your bank account, with that customer’s permission. Previously, that information would only have been accessible by banks. Open banking has led to several examples of new and innovative services that help consumers and businesses make the most of their finances. Open banking can also include payment initiation, where TPPs make payments on behalf of their customers with their consent.

Screen scraping is effectively one way to power open banking. While other technologies are becoming more and more common, screen scraping is still accepted under PSD2 when more modern and secure API technology isn’t available or working.

Screen scraping vs APIs

The main alternative to screen scraping in open banking is API technology. APIs connect different applications together so they can exchange data. But, unlike screen scraping, they do so in a secure, uniform and entirely encrypted way. They also enable data minimisation, meaning that subsets of account data can be accessed (with the customer's consent), rather than all of a customer's data being accessed at any one time, as is the case with screen scraping.

Banks provide their own APIs for other businesses to connect to. In the UK, these APIs must conform to standards set out by the Open Banking Implementation Entity (OBIE). Those wishing to connect to these APIs must also be authorised, in this case by the Financial Conduct Authority (FCA). In the EU, there are several different standards for APIs, all of which allow providers to comply with PSD2.

Find out more about open banking

Since PSD2 came into effect in early 2018, open banking APIs built by the banks have steadily improved, along with the use cases powered by them.

Read our comprehensive guide to open banking, to learn more about how APIs work, what they can be used for and how your business can benefit.