Mediation AB testing
A/B testing is a simple and effective way to transparently compare the performance of Appodeal with other mediation services.
To exclude the influence of external factors on our results (such as seasonality, for example), we recommend running parallel rather than serial tests.
During parallel A/B testing, two mediations are running in your app at the same time, and users are randomly divided into two groups. The first group works with Appodeal and the second one works with another mediation service.
To obtain a statistically reliable result, it is necessary to exclude the influence of the groups on each other, i.e. the user (or device) should be strictly assigned to one group.
The percentage of traffic distribution between groups is configured on your own server, using Firebase Remote Config or another service for A/B testing. This will allow us to dynamically adjust the distribution of traffic during the test, as well as direct all traffic to Appodeal if the test result is satisfactory.
Alternatively, Android apps can use a staged rollout of updates.
A/B testing using Firebase Remote Config
- First of all you need to integrate Appodeal SDK (Android/iOS/Unity) with the Firebase adapter included.
Contact our support team via email [email protected] or the live chat for additional setup.
Go to Firebase Console -> select your project -> Remote Config -> press Create configuration.
Parameter name - mediation
Data type - String
Default value - appodeal (or one of your mediation partners)
Press Add new (Conditional Value -> Create new condition) to add new condition.
In the example below we will divide our users by half.
Name the condition as 50% (or any other percentage you want to divide your users into).
For the condition choose User in random percentage and select from 0 to 50.
Press Create condition.
As the second value enter the name of the mediation provider you are going to have tests with, for example max.
Now, you can observe the data in Appodeal and in Firebase Reports.
Please find the code sample in our demo projects below :
A/B testing using your own server
For cases when Firebase Remote Config is not used, the necessary settings for dividing user groups can be set on the server:
- When the application is launched for the first time, generate a random number for each user in the range from 0 to 100. Let’s denote this value by userPercentage. Store the value of userPercentage on the device and use it in all subsequent application sessions.
- Get a value from the file on the server that indicates what percentage of the advertising traffic will work through Appodeal. Let’s denote it by appodealPercentage.
- If userPercentage < appodealPercentage, initialize and use Appodeal. Otherwise, initialize and use another mediation service.
A/B testing using a staged rollout of updates to Google Play
In the Android app, it is not necessary to integrate two mediations into one build at once and use services for A/B testing.
It will be enough to prepare a separate version of the app with the SDK of the third-party mediation and deploy the update to a part of the audience.
You can see a more detailed description of this process in Help - Play Console.
Recommendations to avoid problems during A/B testing
- Both mediation services should not be initialized in one session.
- To avoid errors during the build and run time of the application, the versions of third-party SDKs in the application should coincide with their versions in Appodeal SDK. Android SDKs versions are available in the archive with the Appodeal SDK distribution. iOS SDKs versions are located in the CocoaPods release of the Appodeal SDK.
- When building an Android application, we recommend using MultiDex.
- In cases where only one network account is used for both mediation services, traffic distribution must be organized by using different ad units (placements/zones) for each group of users. Otherwise, you can get incorrect statistics for both mediations.
Performance comparison of the mediation services
ARPDAU is used to evaluate the performance of advertising since it is the the most objective metric for evaluating results for different impression volumes.
ARPDAU (Average Revenue Per Daily Active User) shows how much you earn on average per day on one active user.
This metric is calculated by the formula:
ARPDAU = Daily Revenue / Daily Active Users (Daily Revenue / Number of Active Users per day)
We will consider Daily Revenue as Ad revenue thereafter.
You can get Ad Revenue in the mediation dashboard. In Appodeal this metric is called "Ad Revenue", in other Mediations it can be "Network Rev", "Earnings", etc. It is better to use the actual value of ad network revenue rather than the estimated one, as it is more accurate.
DAU for mediation-specific versions of the app should be obtained from a third-party analytics service (Google Analytics, Firebase, etc.), as mediation services may track app active users differently.
For a more accurate assessment, we recommend to count not only the total ARPDAU by app, but also the ARPDAU by the country with the highest revenue for each type of advertising separately.
We also recommend comparing the following metrics (for app users with Appodeal SDK and third-party mediation SDK, respectively):
- retention rate day(n) - from analytics,
- crash rate - from analytics,
- time spent by the user in the app (daily) - from analytics,
- impressions per user (daily) for each ad type. This metric can be calculated by dividing the number of daily impressions for each ad type (this metric is called "Impressions" in the Appodeal dashboard) by the DAU from analytics.