Overview

Australia's superannuation industry is a robust and essential part of the national economy, with total assets exceeding $3.9 trillion as of March 2024. 

Customer churn is a problem in many industries, none more so than in superannuation due to its compulsory nature and low barriers to switching. Superannuation customer value is proportional to their super balance, so it follows that the longer someone has been a customer (and the larger their super balance becomes) the higher their value. Put another way, having customer churn close to retirement is costly.

Active Super is a prominent player within this thriving sector, known for its strong performance and member-focused approach. With a substantial share of the market, Active Super has consistently delivered competitive returns and innovative investment options. Deepend have been working with Active Super on their technology and platforms for many years, and we wanted to see if we could move the needle on the customer churn issue by applying a new approach that hasn’t been tried before.  

The current crop of LLM based AI systems have a unique ability to understand human language. This insight formed the core of an idea that was developed into an experimental program. At a high level, the questions we wanted to test were:

1. Could AI leverage new and existing data sets to understand customer sentiment?
2. Would this sentiment over time have predictive value for customer churn?
3. Could generative AI be leveraged to then generate personalised retention communication strategies?

What We Delivered 

• AI experiment design and methodology 
• Data analysis and preparation 
• Custom Python data pipeline 
• Feature engineering and data cleansing 
• Custom LangChain application 
• OpenAI API integration 
• Churn signal identification 
• Prediction testing and model refinement 
• Personalised retention communication generation 

Impact 

The experiment established that customer interaction data could contribute to churn prediction alongside demographic and account information. 

• Prediction accuracy exceeded the initial target by 5% in round one 
• Later testing achieved 75% churn prediction accuracy 
• Churn indicators were identified across demographic, balance, tenure, sentiment and keyword data 
• The application generated personalised retention communications linked to likely causes of dissatisfaction 

Problem & Insight 

Active Super had extensive customer data, but much of it was siloed or difficult to analyse at scale. Free-text CRM notes and service interactions contained valuable signals that traditional analysis could not easily surface. 

Deepend’s core hypothesis was that language models could interpret changes in sentiment over time and combine these signals with known factors such as tenure and balance. 

Strategy & Experience Design 

The team framed the work as an experimental program rather than a production deployment. Three questions guided the program: whether AI could understand sentiment, whether sentiment could predict churn, and whether generative AI could support personalised retention. 

The methodology used known churn outcomes to identify signals before testing the application against new curated datasets. 

Technology & Delivery 

Deepend built a Python data pipeline to merge, clean and prepare the source datasets. The process removed erroneous records, expanded acronyms and created new features from existing information. 

A custom application was then developed with LangChain and OpenAI APIs. This gave the team control over the testing environment, data handling and iterative model refinement. 

Results 

Testing showed that the model could identify churn indicators across demographic and interaction data. Round one exceeded the target accuracy by 5%, and subsequent refinement increased prediction accuracy to 75%. 

The experiment also exposed an important constraint: the model occasionally predicted churn for non-existent customer IDs. The source notes that limiting prediction-set size was being investigated as a possible control. 

Frequently Asked Questions

How did Deepend use AI to predict member churn for Active Super? 

Deepend built a custom AI application that combined demographic, account and customer interaction data to identify signals associated with member churn. The experiment also analysed sentiment over time from unstructured CRM notes and service interactions. 

What technologies were used in the Active Super churn prediction project? 

Deepend created a custom Python data pipeline and an AI application using LangChain connected to OpenAI APIs. The setup provided a controlled environment for preparing data, testing churn signals and generating personalised retention communications. 

What churn prediction accuracy did the Active Super AI model achieve? 

Round one testing exceeded the initial accuracy target by 5%. After further refinement, later testing reached 75% churn prediction accuracy. 

What data signals were used to identify potential member churn? 

The experiment identified churn indicators across demographic and account data, including tenure and balance, as well as customer interaction signals such as sentiment over time and keyword usage. 

What limitations did the Active Super AI churn experiment identify? 

The model sometimes predicted churn for customer IDs that did not exist. Deepend identified this as an AI hallucination risk and continued investigating controls, including limiting the size of the prediction set.