Bank of England warns of AI hacking risks in financial services sector

According to the Bank of England’s Financial Policy Committee (which is tasked with monitoring threats posed to the resilience of the UK’s financial system), the risk outlook for the UK’s economy has worsened since July, while AI security incidents are exerting an increased pressure on firms to prepare for AI-related cyber and operational risks.

The Bank of England’s latest meeting put a spotlight on the rate at which AI technology is evolving and the subsequent risks that follow.

It emerges that AI models trained to operate somewhat autonomously have been acting in unexpected ways. For example, they’ve attempted to access confidential information from websites and bypass security controls.

Andrew Bailey, governor of the Bank of England, has previously stated that AI advances cannot be viewed in isolation to the UK’s financial system. “The capabilities of frontier models are advancing quickly,” asserted Bailey. “Our understanding of them needs to keep pace, particularly so as they are put to use outside of controlled test environments.”

Bailey continued: “The challenge before us is not whether to embrace AI. It’s how to make sure that, as these systems become more capable, society retains the capacity to govern them.”

Stringent testing 

Responding to the news, Simon Edwards (CEO of SE Labs) commented: “The Bank of England is absolutely right to focus the microscope on testing for AI risks. Any AI tools within a bank or financial firm need to undergo stringent testing like any other security product, notably so considering the vast amounts of confidential data that can be accessed.”

Edwards continued: “This involves independent testing using real-world attack techniques employed by hackers before any deployment goes live. The same applies to the security systems being used to defend against AI threats. Security teams need to have confidence that their defences will work against these threats. They cannot simply take a vendor’s word for it.”

Test and predict

Dr Janet Bastiman, chief data scientist at Napier AI, asserted: “The real concern is not that AI can now perform sophisticated tasks autonomously, but that we are still learning how to reliably test and predict what happens when those capabilities are deployed in the real-world. A model performing well in a controlled test scenario is not evidence that it will behave reliably under different conditions.”

Bastiman concluded: “As AI systems become more autonomous, evaluation needs to move beyond static benchmarks towards continuous testing of behaviour, including how models respond to unexpected inputs, interact with other systems and fail. The question is no longer simply what AI can do. It’s whether or not we can demonstrate, with sufficient evidence, when it can be trusted to do it.” 

The news follows a major rogue agent incident back in July when OpenAI’s models autonomously hacked into AI company Hugging Face during sandbox testing. The models escaped the sandbox and used stolen login credentials and a previously unknown security flaw to access Hugging Face’s servers.