Improving Instagram notification handling with machine learning and causal inference
- We share how Meta applies statistics and machine learning (ML) to improve the personalization and management of notifications on Instagram, especially on daily push notifications.
- By using causal inference and ML to identify highly active users who are likely to see more content organically, we were able to reduce the number of notifications sent while improving the overall user experience.
On Instagram, notifications play an important role in providing effective communication channels between Instagram and our users. As the types of notifications have increased, it has become necessary to provide users with personalized notification experiences to help them avoid receiving excess notifications or those that they might not find important.
At Meta, we have applied statistics and machine learning (ML) to the personalization and management of notifications on Instagram. Today we would like to share an example of how we used causal inference and ML to control the sending of daily push notifications.
Go beyond click-through rate models
A daily stories digest push notification is a type of notification that lists a digest of stories that are shared and ready for someone to view. When such a notification is sent to a person’s device, that person can click on the notification to view the content. Traditionally, an ML model called the click-through rate (CTR) model is used to predict the likelihood of someone clicking on a notification. CTR models have performed well in many industry applications. The predicted click probability is used as a proxy to indicate the quality of notification to the user. If the predicted click probability is too low, the notification will be dropped in the middle of its send flow and the user will not receive the notification because it was deemed low quality.
Filtering based on the CTR model worked well for the daily digest notification in the sense that the actual average click-through rate when using the CTR model was significantly higher than without the model. However, we also noticed that using the CTR model meant that a large portion of the daily digest notifications were sent to users who were relatively active in terms of Instagram usage. For many highly active Instagram users, even without sending those daily digest notifications, they could see related stories organically. This actually opens up the possibility of providing a better user experience by sending fewer notifications to active users who are likely to see stories listed in notifications organically.
The difficult aspect is how to identify these users. If we reduce the notifications sent to active users based on the receipt of such notifications, these users may become less active. In other words, if the right users are not properly selected to send discounts, we risk creating a drop in user engagement.
Causal Inference and ML
This is essentially a user selection problem. We would like to maximize the effectiveness of the notifications sent by selecting the appropriate user cohorts. The solution we adopted to solve this problem is the combination of causal inference and ML.
The next question is how to estimate the incremental value before making the send or delete decision. This is a difficult question because for the same notification, we can either send it or delete it; there is no way to observe both scenarios. This is essentially a problem of causal inference and uplift modeling techniques can be used. To apply uplift patternsWe designed a randomized experiment in which each notification was sent or deleted at random, as shown in Figure 1.
Based on data collected from this randomized experiment, we developed a neural network-based uplift model to predict the incremental impact between not sending and sending daily digest notifications on user-level stories. Given the incremental impact estimates for all notifications, the solution to the above budget allocation problem is trivial. However, in practice, notifications are generated and scored online and therefore we cannot have incremental impact estimates for all candidate notifications in advance.
Therefore, we need an inline approach to determining which notifications to send or delete. A simple but effective solution is to compare the score generated online with a fixed threshold – if the score is above the threshold, we can send it. In doing so, we intend to maintain a fixed notification sending rate. r where 0 .
When we applied this approach in the online tests, we observed fluctuations in the sending rate, because the estimates of increase (incremental impact) generated from the ML models can change from time to time for various reasons. To stabilize the send rate, we use an online quantile calculation service to transform the raw uplift estimates to a standard uniform distribution while preserving orders. To maintain the send rate to rwe simply compare the transformed uplift estimate with r to make the send decision, since the transformed uplift estimate Z~U(0.1), Pr( Z >= 1 – r ) = r. This process is illustrated in Figure 2.
Better notifications with causal inference and ML
By applying this model and targeting users + notifications with high incremental impact, we significantly reduced send volume compared to using the CTR model and also saw no drop in user engagement. The benefit of this work is twofold: improved user experience and reduced resource usage.
In the Instagram Notifications Systems team, ML and statistics have been applied in different areas to improve users’ notification experience. If you would like to learn more about this job or are interested in joining one of our engineering teams, please visit our careers pageand follow us on Facebook.