Skip to main content

Mediation AB testing

A/B testing is a simple and effective way to transparently compare the performance of Appodeal with other mediation services.

To exclude the influence of external factors (such as seasonality) on our results, we recommend running tests simultaneously (this is called parallel testing) rather than serially.

During parallel A/B testing, two mediation services are running in your app at the same time, and users are randomly divided into two groups. The first group works with Appodeal and the second one works with the other mediation service.

To obtain a statistically reliable result, it is necessary to exclude the influence of the groups on each other, i.e. the user (or device) should be strictly assigned to one group.

The percentage of traffic distribution between groups is configured on your own server, using Firebase Remote Config or another service for A/B testing. This will allow us to dynamically adjust the distribution of traffic during the test, as well as direct all traffic to Appodeal if the test result is satisfactory.

Alternatively, Android apps can use staged update rollout.

A/B testing using Firebase Remote Config​

  1. First of all you need to integrate the Appodeal SDK (Android/iOS/Unity) with the Firebase adapter included.
Attention

Contact our support team via email [email protected] or live chat for additional setup instructions.

  1. You need to set up Firebase according to this guide. (Android/iOS/Unity)

  2. Go to Firebase Console -> select your project -> Remote Config -> press Create configuration.

Parameter name - mediation

Data type - String

Default value - appodeal (or one of your mediation partners)

Press Add new (Conditional Value -> Create new condition) to add new condition.

note

In the example below we will divide our users into two equal groups.

Name the condition 50% (or any other percentage you want the groups to have).

For the condition choose User in random percentage and select the value from 0 to 50.

As the second value enter the name of the mediation provider you are going to test, for example, max.

  1. Initialize Firebase using this guide. (Android/iOS/Unity)

  2. In the application get the value of the key "mediation", initialize and use only the corresponding mediation service using this guide. (Android/ iOS/ Unity)

Now, you can observe the data in Appodeal and in Firebase Reports.

Please find the sample code in our demo projects below :

A/B testing using your own server​

For cases when Firebase Remote Config is not used, the necessary settings for dividing users into groups can be configured on the server:

  1. When the application is launched for the first time, generate a random number for each user in the range from 0 to 100. Let’s denote this value by userPercentage. Store the value of userPercentage on the device and use it in all subsequent application sessions.
  2. Get a value from the file on the server that indicates what percentage of the advertising traffic will work through Appodeal. Let’s denote it by appodealPercentage.
  3. If userPercentage < appodealPercentage, initialize and use Appodeal. Otherwise, initialize and use another mediation service.

A/B testing using a staged rollout of updates to Google Play​

In the Android app, it is not necessary to integrate two mediations into one build at once and use services for A/B testing.

It will be enough to prepare a separate version of the app with the SDK of the third-party mediation and deploy the update to a part of the audience.

You can see a more detailed description of this process in Help - Play Console.

Recommendations to avoid problems during A/B testing​

  • Both mediation services should not be initialized in one session.
  • To avoid errors during the build and run time of the application, the versions of third-party SDKs in the application should coincide with their versions in Appodeal SDK. Android SDKs versions are available in the archive with the Appodeal SDK distribution. iOS SDKs versions are located in the CocoaPods release of the Appodeal SDK.
  • When building an Android application, we recommend using MultiDex.
  • In cases where only one network account is used for both mediation services, traffic distribution must be organized by using different ad units (placements/zones) for each group of users. Otherwise, you can get incorrect statistics for both mediations.

Performance comparison of the mediation services​

ARPDAU is used to evaluate the performance of advertising since it is the the most objective metric for evaluating results for different impression volumes.

ARPDAU (Average Revenue Per Daily Active User) shows how much you earn on average per day on one active user.

This metric is calculated by the formula:

ARPDAU = Daily Revenue / Daily Active Users (Daily Revenue / Number of Active Users per day)

note

We will consider Daily Revenue as Ad revenue thereafter.

You can get Ad Revenue in the mediation dashboard. In Appodeal this metric is called "Ad Revenue", in other Mediations it can be "Network Rev", "Earnings", etc. It is better to use the actual value of ad network revenue rather than the estimated one, as it is more accurate.

DAU for mediation-specific versions of the app should be obtained from a third-party analytics service (Google Analytics, Firebase, etc.), as mediation services may track app active users differently.

For a more accurate assessment, we recommend to count not only the total ARPDAU by app, but also the ARPDAU by the country with the highest revenue for each type of advertising separately.

We also recommend comparing the following metrics (for app users with Appodeal SDK and third-party mediation SDK, respectively):

  • retention rate day(n) - from analytics,
  • crash rate - from analytics,
  • time spent by the user in the app (daily) - from analytics,
  • impressions per user (daily) for each ad type. This metric can be calculated by dividing the number of daily impressions for each ad type (this metric is called "Impressions" in the Appodeal dashboard) by the DAU from analytics.