Time Taken: 2 months
Market: 'Health & Wellbeing' & 'Sports & Recreation'
Average Revenue Increase: 45%
Executive summary
Goals and measurement
- The intervention goals we defined and tested as part of this project were:
- Increase Urgency
- Increase Trust, Remove Blocks to Sale
- Increase Awareness
- Close a Sale
- Each goal was measured against the following core KPIs:
- Conversion Rate
- AOV
- Revenue (Total Revenue and Product/Promotion specific revenue as appropriate)
- Exit Rate
Key results
- In every scenario tested, interventions contributed to less frequent site exit, improved conversion rates and strong revenue growth.
- Direct A/B tests showed that users who have seen interventions can be up to 4 times more likely to convert than users who haven't seen interventions.
- Across our core tests, we calculated an average revenue growth of 45% in periods when interventions were used.
Introduction
This research paper will define a variety of IRP intervention strategies and analyse their revenue impact. This is a metric-based study carried out on IRP commerce client websites over the course of a two-month period in 2017.
In this document we will firstly outline our initial hypothesis and define the project scope. We will then summarise our project methodology and finally move on to highlight key results and suggestions for future research.
The paper will provide a blueprint for future intervention campaigns and prove that well-executed IRP intervention strategies can generate an average of 45% ecommerce revenue growth.
Hypothesis and scope
The intervention functionality on the IRP has been created to enable programmatic intervention into a user journey. The end goal of these interventions is always to increase website revenues. Aside from collecting customer data for future use, revenue growth with interventions can be achieved in two principal ways: by increasing Conversion Rate, or by increasing Average Order Value.
Our hypothesis was that basic and logical intervention strategies could positively affect the Conversion Rate and Average Order Value levers of the core ecommerce equation. We believed that by reinforcing the correct sales messages at the correct time, customers could be reassured, seduced or accompanied to checkout, much in the same way that an experienced Sales Assistant would do in a physical store.
The aim of the study was to test the above hypothesis. To do this, we created intervention goals with specific KPIs. Depending on the goal, we measured Conversion Rate, Average Order Value, Exit Rate and Revenue as appropriate. The results are highlighted below.
The scope of the study and timeframe did not allow for optimisation and iteration on specific messaging or strategies. Further studies, analysis and multivariate testing will be required to improve on results in the future.
Methodology
Project guidelines
We created an Interventions project group within Export Technologies to carry out this project. This team established the following rules and guidelines at the outset:
- Client websites should be chosen based on the following criteria:
- Willingness to test interventions
- Revenue potential of the intervention
- Traffic levels (more traffic allows for faster, conclusive results)
- Interventions should always have clear goals and KPIs in mind when they are conceived.
- Where possible, similar interventions should be run across a range of different clients. Cross-referencing of results between clients will increase result confidence.
- Interventions should be run for as long as possible to get a definitive data set. For this project, an intervention must have a minimum of 2000 user sessions to be considered for analysis.
- For the purposes of this project, we should not run multiple concurrent interventions. I.e. a single customer should only see a single intervention. This will allow for easier results tracking and reporting.
- Seasonality, ad hoc website promotions, product launches, stock holdings, etc. make it difficult to compare results of one short-term period versus another. There are too many variables at play to measure week1 vs. week2, or week1 vs. week1 last year. As such, and given the short-term nature of this project, interventions should be a/b tested against the 'control' of 'no intervention' within exactly the same timeframe. To do this, all interventions run as part of this project should be set to show 50% of the time and tracked using Google Analytics event tracking as explained below. We will still use comparative periods to put results in context, but it's important to have 'a/b' data on 'interventions' vs 'no interventions' under otherwise identical conditions.
- Google Analytics should be used to analyse data and results and the performance case study will be written based off findings.
- Our project plan and methodology should be documented, iterated upon and reused for future intervention projects.
Intervention scenarios
With guidelines established, our project team went on to explore and document the process for creating a successful interventions strategy.
Based on research and real world use cases, the project team established that interventions would generally always take place against the backdrop of different scenarios. In effect, these 'Scenarios' set the scene of the website problem to be solved and are the perfect starting point for forming intervention strategy and goals. For example:
- Poor Conversion Rate in country X/device Y
- Possibility for higher AOV in country X/device Y
- High basket drop off rate
- Overstock of a particular brand
- Effective email marketing but relatively small Mailing List
- Under-performing promotions
Intervention goals
Each scenario will point to logical intervention goals and KPIs to measure.
For example:
- Scenario: The website has excellent traffic in Germany but a poor Conversion Rate vs. UK.
- Goal: Remove blocks to sale and increase trust for German customers.
- KPI: Conversion Rate and AOV for German customers.
- Scenario: Promotion results tend to tail off at the end of a long promotion period.
- Goal: Increase urgency for all customers on the last day of sale.
- KPI: Conversion Rate on sale items.
The team created a reusable list of goals, example actions and measurable KPIs. These are outlined below:
| Goal |
Example Action |
KPIs To Measure |
| Improve Customer Data |
Mailing List Signup prompt in popup |
Mailing List growth.
Average weekly mailing list signups with intervention active versus average weekly mailing list signups without. |
| Get Feedback via survey/poll |
Poll completion rate with intervention versus without |
| Increase Urgency |
Highlight a soon-to-end flash sale or voucher code |
CR with intervention versus CR without |
| Increase trust, remove blocks to sale. |
Highlight country specific value proposition and customer service ratings |
CR with intervention versus CR without
Exit Rates |
| Increase Awareness |
Promotions – highlight category/brand/country specific promotions |
Promotion specific sales with intervention versus without
CR with intervention versus CR without |
| Product Focus – highlight best sellers, new products/brands/categories etc. |
Product specific sales with intervention versus without
CR with intervention versus CR without |
| Upsell |
Make suggestions based on basket contents or value |
AOV with intervention versus AOV without
Sales of upsold product |
| Close a sale |
Offer free shipping or an voucher if a user with buying intent attempts to leave basket page
Tempt a leaving customer with over £X basket value to continue browsing |
CR with intervention versus CR without
Site Exit Rate with intervention versus without |
Performance measurement
Specific KPIs were agreed against each intervention goal. In general, the KPIs that should always be considered when running interventions were defined as:
- Conversion Rate
- AOV
- Revenue
- Exit Rate
NOTE: When using Google Analytics Event Tracking, Bounce Rates cannot be measured since the display of an intervention along with a Google Analytics Event automatically tracks as a non-bounce.
- It was agreed that the use of an intervention should be a/b tested against the 'control' of 'no intervention' within exactly the same timeframe. To do this, all interventions run as part of this project were set in the IRP to show 50% of the time. Performance was then tracked using Google Analytics event tracking and segment analysis as explained in Appendix 2.
Implementation and results
Clients were chosen from a variety of markets. Selection was based on revenue potential and volume of traffic (for faster and more conclusive results). Client permission and participation was also required.
Given the scope and time limitations on this study, the team decided to only test the following intervention goals:
- Urgency
- Awareness
- Increase trust, remove blocks
- Close a Sale
Intervention goal 1: increase urgency
The urgency goal was tested on 1 client from the 'Health and Wellbeing' market over a two-day timeframe.
The client had the scenario of a two-week sale coming to a slow close, and the requirement for a strong last day of sales to clear stock.
We used an Interventions 'PopUp Lightbox' to promote and create a sense of urgency around the 'Last day of Sale'. This was shown to both mobile and desktop users from all countries except Germany (Germany was concurrently running another intervention).
Comparing the KPIs of users who saw the intervention versus those who didn't, results were comprehensive.
A/B test results
Users who saw the intervention posted a Conversion Rate of 3.62%. Users who didn't see the intervention posted a Conversion Rate of 1.87%. This equates to a 93% increase in Conversion Rate when the intervention was displayed.
Average Order Value increased by 19% from £29.24 to £34.93.
Exit rates dropped from 18% to 7% when an intervention was shown.
Results in context
When we compared the overall Conversion Rate on the days of the intervention versus the two days previous, we also saw a significant boost in conversion of 17.22% (from 1.8% to 2.11%).
Average Order Value did drop, but the site still showed a revenue increase of 7.6%.
Revenue of sale items sold during the intervention increased by 16% versus the previous period, highlighting the effectiveness of the awareness campaign.
Intervention goal 2: increase trust, remove blocks, answer needs
The trust goal was tested on websites from the 'Sports and Recreation' and 'Health and Wellbeing' markets. Interventions were shown in specific countries where it was believed that an improved immediate understanding of the value proposition would impact Conversion Rates and lower site exit.
Trust test 1
We used an Interventions 'PopUp Lightbox' to highlight the website's customer value proposition to UK customers landing on any page on the website. This was shown to both mobile and desktop users and included key points around security, delivery and customer service.
A/B test results
Users who saw the intervention posted a Conversion Rate of 1.27%. Users who didn't see the intervention posted a Conversion Rate of 0.34%. This equates to a 306% increase in Conversion Rate when the intervention was displayed.
Average Order Value did however drop from £325 to £217
Exit rates dropped from 35% to 20% when an intervention was shown.
Results in context
When we compared the overall Conversion Rate on the days of the intervention versus the previous period, we also saw a boost in conversion of 23% (from 0.54% to 0.67%).
Revenue versus the previous period grew by 95% (from £118k to £200k).
Trust test 2
We used an Interventions 'PopUp Lightbox' to highlight the website's customer value proposition to USA customers landing on any page on the website. This was shown to both mobile and desktop users and included key points around security, delivery, and customer service. Messages were segmented and customised depending on whether the user was new or returning.
A/B test results
Users who saw the intervention posted a Conversion Rate of 0.89%. Users who didn't see the intervention posted a Conversion Rate of 0.24%. This equates to a 270% increase in Conversion Rate when the intervention was displayed.
Average Order Value increased by 50% from £103 to £154.
Exit rates dropped from 24% to 19% when an intervention was shown.
Results in context
When we compared the overall Conversion Rate on the days of the intervention versus the previous period, we also saw a boost in conversion of 8% from 0.5% to 0.54%. Over the same period, AOV increased 27% and revenue grew by 45.35%.
Trust test 3
We used an Interventions 'PopUp Lightbox' to highlight the website's customer value proposition to UK customers landing on any page on the website. This was shown to both mobile and desktop users and included key points around security, delivery, and customer service. Messages were segmented and customised depending on whether the user was new or returning.
A/B test results
Users who saw the intervention posted a Conversion Rate of 3.33%. Users who didn't see the intervention posted a Conversion Rate of 0.65%. This equates to a 412% increase in Conversion Rate when the intervention was displayed. It should be noted here that different messages were used on this intervention depending on whether users were new or returning. The effectiveness of this segmentation strategy can be seen in this impressive result.
Average Order Value dropped by 13%, from £134 to £117.
Exit rates dropped from 20% to 15% when an intervention was shown.
Results in context
When we compared the overall Conversion Rate on the days of the intervention versus the previous period, we also saw a significant boost in conversion of 14% (from 1.25% to 1.43%) and a massive uplift in revenue of 45%.
Trust test 4
We used an Interventions 'PopUp Lightbox' to highlight the website's customer value proposition to German customers landing on any page on the website. This was shown to both mobile and desktop users and included key points around security, delivery, payments and customer service. Messages were displayed in German Language to accompany the translated website language.
A/B test results
Users who saw the intervention posted a Conversion Rate of 1.05%. Users who didn't see the intervention posted a Conversion Rate of 0.47%. This equates to a 123% increase in Conversion Rate when the intervention was displayed.
Average Order Value stayed relatively constant at the £35 mark.
Exit rates dropped from 37% to 13% when an intervention was shown.
Results in context
When we compared the overall Conversion Rate on the days of the intervention running, versus the previous period, Conversion Rate was 20% higher (0.81% vs. 0.67%). It must also be noted that this particular intervention was run over a long period of time in order to gather sufficient test data. It would appear that the effectiveness of the intervention decreased the longer it was displayed. Results in the first week were significantly higher, with German Conversion Rates jumping as high as 6% overall. This did however coincide with a sale — showing the difficulty of comparing results from different time period.
Revenue increased 195% versus the previous period. This was down to a combination of the increased conversion from the intervention and the increase traffic to the German version to the website.
Intervention goal 3: increase awareness
The awareness goal was tested on 1 client website from the 'Sports and Recreation' market over a two-week timeframe.
The client had the scenario of a having excess seasonal stock to clear. They were planning to run a 2 for 1 promotion to shift stock and interventions was seen as a perfect tool to increase awareness of the special offer.
We used an Interventions 'PopUp Lightbox' to highlight the 2 for 1 promotion to customers landing on any page on the website. This was shown to both mobile and desktop users and prompted the user to click through to the promotion product listings page.
A/B test results
The intervention was eventually shown to 65% users. Those who saw the intervention accounted for over 80% of the promotional product revenue (£18k vs. £5k).
Users who saw the intervention, posted a Conversion Rate of 1.87%. Users who didn't see the intervention posted a Conversion Rate of 0.9%. This equates to a 108% increase in Conversion Rate when the intervention was displayed.
Exit rates dropped from 52% to 22% when an intervention was shown.
Results in context
When we compared the overall Conversion Rate on the days of the intervention running, versus the previous period, we also saw a boost in conversion of 114% from 0.72% to 1.54%.
Overall site revenue increased by 44.55% during this combined intervention + promotional period (from £16k to £23k.) A good proportion of this increase can obviously be attributed to the very good deal on offer, but the strong take-up of the promotion by users who saw the intervention shows how effective this tool can be in stimulate awareness and revenue.
Intervention goal 4: close a sale
The goal of 'Closing a sale' was tested on one client website from the 'Health and Wellbeing' market over a two-week timeframe.
This client had the scenario of Conversion Rates and AOV consistently falling in the past year across all countries. The UK in particular had seen the largest fall in Conversion Rate. Traffic had increased 100% but revenue had remained virtually static due to the lack on conversion. The lost revenue sitting in dropped baskets on this client site was extremely significant.
We used an Interventions 'PopUp Lightbox' to display a message if a customer with over £105 in their basket tried to leave the site. The Free Shipping threshold on this site was £120. The intervention was used to block the leaving of the site and inform the customer that they were not far away from getting free shipping. This was shown to both mobile and desktop users and gave the user the choice to continue shopping to claim their free shipping or to or dismiss the intervention and leave the site.
A/B test results
Users who saw the intervention showed excellent AOVs and Conversion Rates. This is understandable since they already had high value baskets. However, when we compared these users to other high value customers (over £105 revenue) we can see that those who saw the intervention had AOVs (£213 vs. £148) and Conversion Rates (45% vs. 33%) that were roughly 40% better. Exit rates were also significantly lower.
Results in context
When we compared the overall Conversion Rate on the days of the intervention running, versus the previous period, we also saw a significant boost in conversion of 22% from 2.17% to 2.65%.
Revenue only increased 0.75% but this can be attributed to the 21% fall in traffic, which effectively negated the conversion increases over the same time.
Conclusion
This research project and paper has outlined a first approach for creating and measuring Intervention strategies. Results of the project strongly validate the original hypothesis that 'the strategic use of interventions can significantly increase ecommerce revenues'. Key details are summarised below.
Intervention strategy
The key strategy takeaways from this research are as follows:
- Interventions will always take place against the backdrop of a business 'scenario' or a problem/opportunity to be addressed.
- The scenario will point towards an appropriate intervention. This intervention should have a clearly defined goal. The intervention goals we defined as part of this project are:
- Improve Customer Data
- Increase Urgency
- Increase Trust, Remove Blocks to Sale
- Increase Awareness
- Upsell
- Close a Sale
- Each goal should be measured for effectiveness against the following core KPIs:
- Conversion Rate
- AOV
- Revenue (Total Revenue and Product/Promotion specific revenue as appropriate)
- Exit Rate
- Results should be measured using Google Analytics Events and Segments. Supporting product and promotion sales data should be taken from the IRP.
The impact on revenue of interventions
- In every scenario tested, interventions contributed to reduced site exit, increased conversions and strong revenue growth.
- There is some evidence to suggest that interventions do not always increase AOV. This is understandable where AOV is not the main goal of the intervention, i.e. if the strategy is to get a conversion, not a higher value one.
- Outside of specific promotions, results were strongest where Interventions were able to have an impact on both AOV and Conversion.
- Interventions gave particularly impressive revenue results when combined with on-site promotions. Creating additional awareness of such promotions saw overall revenue increase by 45% versus the previous period.
- Direct A/B tests showed that users who see interventions can be up to 4 times more likely to convert than users who haven't see interventions.
- The more segmented and targeted an intervention, the more effective it is — specific messaging for specific user profiles creates better results.
- The longer an intervention runs, the less effective it is. Interventions should be split between new and returning users and kept as fresh as possible.
- Chosen correctly and wisely, interventions should drive significant revenue growth. Across our core tests, we calculated an average revenue growth of 45% in periods when interventions were used. It must also be noted that due to a desire to a/b test prove 'interventions' vs. 'no interventions', our interventions were not shown to all segmented users. Results could have been even more significant otherwise.
In summary, using a carefully planned strategy in combination with the IRP intervention features is a guaranteed route to increased conversion and revenue.
Appendix 1: Areas for improvement and future research
- Google Analytics was required to split test interventions and accurately measure their performance. More complete reporting options in the full release of IRP interventions will make this task easier in the future.
- To allow for a/b testing, all interventions tested in this project were set to display a 50% frequency to the selected user base. The actual final display frequency varied between 30% and 70% to the selected user base. This would suggest that further testing and understanding is required of this feature or that perhaps sample test size were not sufficient for the IRP algorithm to even out at 50%.
- This study simply tested the effectiveness of different intervention goals. Whilst initial results were excellent, there is no doubt that they can be further improved further multivariate tests on the specific context and content of interventions.
Appendix 1: Google analytics tracking setup
Track intervention views
If tracking intervention views, add the below script at the end of your intervention:
Track intervention clicks/acceptance
If tracking intervention clicks/acceptance, add the following code to the link to be clicked:
And add the following code to the bottom of the intervention:
Usage notes
- The argument 'ID 10 - Trust Messaging DE', should be updated with your intervention ID and title. For consistency, this should be the same format as above.
- The argument 'viewed' / 'accepted' is the log of the user action taken, so is set to 'viewed' if you're logging the fact that the intervention has been seen, or 'accepted' if you classify that the user has read, understood and specifically accepted the intervention.
- For clicks/acceptance tracking, you need to update the destination URL both in your link and JavaScript if you are changing this from 'default.aspx'. If you don't want to redirect anywhere, leave these URLs blank by swapping out 'default.aspx' for ''.
- Be careful when editing interventions. The IRP editor occasionally strips out JavaScript code on edit — ensure this still exists after you save.
- The above Google Event Tracking code will automatically populate Google Analytics > Behaviour > Events with the details defined above. This data of these events can then be analysed directly in Google Analytics using Segment reporting.
