Who’s holding the remote control? [4/9]
Exploring the dangers of blindly allowing personalised AI to shape our future
A blog series by Dr. Alex Bowyer and Iain Henderson
AI is infiltrating our financial lives
Welcome to the fourth in our series ‘Who is holding the remote control?’
In our daily lives we use cash less and less each year; the pandemic was a massive shift in that direction, and that change will continue. This means that more and more of our spend and the habits around it are being data-fied. Every search, card swipe and phone touch is recorded, ingested and turned into ‘intelligence’.
Open banking has meant that our bank transactions data are fair game for ingest into the machine, albeit with our ‘consent’. Obviously we will all have read the 30 page terms and conditions and separate private policy, and confirmed we have done so; that we have understood and agree with them, and consented to making our data available (checked the dreaded box) before trying that cool app 🤷
Open Finance is coming next, doing the same to mortgages, pensions, investments, insurances and more. Current statistics suggest that less than 9% of UK adults take formal financial advice. For many, paid advice is in the ‘too hard’ or ‘not for me’ bucket.
And yet at the same time, fraud seems to be exploding. I guess we are all paying increased fees to cover that then….
That means it is only a matter of time before ‘Big AI’, the personalised variety, targets financial services as the next big area to be ‘disrupted’. And ‘autonomous agents’ are apparently coming down the track; wandering around, doing stuff.
The problem is, I’m not sure I want my money disrupted…. I’d just like it to be secure, under my control, invested according to my choices and preferences, growing steadily, in line with my risk profile, and available to support me through life events good, bad and somewhere in the middle. And where I am paying fees for help in managing my money, I’d like those to be transparent, and fair value for what I am paying.
Artificial Intelligence, deployed in a genuinely personal way, could be a very good thing for people in the context of their financial affairs. However, if it heads off in the large scale, ‘personalised’ direction then we may get more than we bargained for.
For a refresher on the difference between personal and personalised AI, see Part 1.
To see what AI-driven financial life might look like in the future, meet Alice:
PART 4: Whose Money is it Anyway?
“After your committed spend, you have £14.98 left…” – Alice’s Story
Who is doing the nudging; my AI or someone else’s?
Alice had always felt she was good with money. Her salary as a nurse in the NHS could be better; but she was used to living within her means. She had some money put aside for a rainy day. She was saving up to get married; she and her fiancé Josh were to marry next summer, and planned to buy their first home soon after.
One day, she saw an advert for a new AI-powered money manager app from Banana Bank - a spin out of one of those big tech firms. They somehow knew she was getting married and looking to buy a new house; their messaging was uncannily real. Their offer was interesting - one single account covering daily spending, savings, investments, a mortgage, and a range of insurances– all managed and optimised by a personalised AI-powered financial guide.
Of course it had all the usual get out clauses….’Investments can go down as well as up’; but Alice decided to give it a try. After all, she was in control; wasn’t she?
All went fine for a few months. The AI moved money around every now and then; but in overall terms the savings and investments seemed to perform well. She received adverts in her feeds thats seemed eerily accurate about what she was planning to buy; but from brands she had never heard of. She just ignored them.
Then things got weird. Banana announced a new OptimiZer AI module in the app. It was not optional, and had an updated terms and conditions (which she didn't read - ‘what’s the point if I have no choice?’). When that kicked in she noticed:
- A lot more transactions were happening in the investment pots. She used to see 3 or 4 trades per month in investments. Things seemed to be speeding up; more like 20 trades or switches per month.
- She seemed to have begun investing in ‘cryptocurrencies’, despite having only the minimum knowledge of what they are.
- She was switching providers of core services such as home energy, car insurance and similar more or less every month in order to save a few pennies; but often moving to brands she had never heard of.
Alice was beginning to feel that this was no longer just her bank account…..
…. and then one day, it all went wrong….
When she woke up she noticed a whole stream of notifications from the Banana app on her phone. Once she had swiped them all away she found that she had started the day with £25k, including her wedding and new house savings; and ended it with £14.98. And bills still to pay.
Two things had gone wrong….
1. A crash had happened in Far Eastern markets overnight for a reason that the AI had not been trained on, and that set off a whole chain of negative financial events.
2. The Banana Bank app developers had been in such a rush to get their new release out that it had not been tested on all scenarios.
…. ‘but remember, investments can go down as well as up….’
She was devastated.
After major stress, a postponed wedding and a change in bank, Alice eventually got her money back. As the industry regulator, The Financial Conduct Authority stepped in. They found that Banana Bank had been making investments that served their interests ahead of Alice’s. Their ‘value’ algorithms had been drawn into a shadlily backed crypto-scheme. They suspended Banana Bank from trading and set up a compensation scheme - the whole process took 18 months… The regulator, in their post-crisis report noted that ‘technology moved, and was allowed to be deployed, much faster than the regulations that provided protection’.
AI-Powered Money Management - in Whose Service?
When money is almost exclusively seen and used in digital form, who is in control of the AI is of critical importance.
Personalised AI may offer some attractive insights, and even make good suggestions. But ultimately it has a fiduciary duty to its owner; not its user. A fiduciary duty is a legal obligation for one party to act in the best interests of another, often involving a relationship of trust and confidence. This means the fiduciary must prioritize the other party's interests above their own, acting with loyalty, good faith, and transparency. Breaching a fiduciary duty can lead to legal repercussions.
In complex areas such as financial services, where the pace of AI invention runs well ahead of regulation then subtleties such as ‘financial advice’ (regulated) and ‘financial guidance’ (lightly regulated) can be mis-understood or ignored. That line can between advice and guidance can be difficult to define.
Personal AI is characterised by its fiduciary duty to you, its user. The quality and accuracy of guidance can’t be guaranteed; but its orientation is very clear and as it should be.
Financial services are not a safe place to explore personalised AI; the consequences could be catastrophic.
→ If we must mix money and AI - let’s ensure that is the Personal AI flavour.
We hope you’ve enjoyed this week’s post in our “Who’s holding the remote control?” series. Each story explores a different fork in the road between AI that works on you, and AI that works for you.
If you missed the earlier parts, here’s some links:
Part 1: TRAVEL
- “The Lure of Convenience”Part 2: HOME
- “Relax, Just Let Me Handle Things”- “Now Entering a Pre-Approved Natural Moment”
Come back next week to explore what effects personalised AI could have in Education .
Thanks for reading — please add your comments below!