Person, Flooring, Wood, Floor, Hardwood, Furniture, Plywood, Couch, Sphere

User state-based notification volume optimization

Arun Nedunchezhian | Software Engineer, Growth

Notifications play an important role in helping Pinterest’s 335M+ monthly active users discover and engage with inspiring content. It’s crucial to send the right number of notifications with high quality content. If we send too few notifications, Pinners might miss out on great opportunities to discover ideas, whereas sending too many notifications could cause fatigue, and lead them to unsubscribe from notifications completely.

Types of Notifications

For us, notification budgeting means calculating the optimal number of notifications to be sent in a week for each person, and we train a model to determine that optimal number.

Current system

Our current notification budgeting system is based on “Notification Volume Control and Optimization System at Pinterest”, which was published as a paper in KDD 2018. The system consists of two steps:

  • Utility Prediction: Use an ML model to predict a utility value for a given user u
  • Budget Allocation: Decide the optimal budget for the week for a given user u
  • Utility prediction

    Let p(a|u, ku) denote the probability of the user ‘u’ performing action ‘a’ this week given notification volume ku. The action ‘a’ can be any metric that we want to optimize. We train a model to predict the probability of action ‘a’ given k notifications for a user u. We make the prediction for values in the range kmin to kmax. From the predicted values, we select the optimal budget ‘b’ in the Budget Allocation step (described below). Our current production system uses daily engagement as the action to maximize for all users since optimizing for it helps us increase our daily active users (DAU).

    Daily engagement might be a long shot to reach for low-engagement users. However, converting more low-engagement users to Weekly Active Users (WAU) is a more reasonable goal. The main difference from our existing budgeting system is that we now have separate models for low-engagement and high-engagement users instead of one model for all users. A custom model for low-engagement users with a different metric to optimize will help increase overall engagement for those users underserved by the previous model.

    Budget Allocation

    The incremental utility of sending the k+1 notification to user u is p(a|u, k+1)−p(a|u, k). In order to achieve the maximum utility with a limited number of notifications, we need to remove the potential notifications with lowest incremental utility.

    Notification Budgeting overview

    For each user, we first compute the optimal budget imax in the allowed range [kmin, kmax], and then we gradually increase budget from kmin to imax until the average incremental value of the remaining notifications is below the threshold . The value i, which gives the maximum incremental utility, is chosen as the user’s budget for the week. The algorithm to find the optimal threshold can be found in the KDD’18 paper.

    User groups

    We want our notifications to increase engagement for users based on their current activity (for example, if they’re returning weekly, our goal is to show engaging content that brings them back daily).

    Based on a user’s activity with Pinterest, we assign them one of six possible user states:

    User states and their descriptions

    The less engaged a Pinner is, the harder it is to develop accurate models, so we have separate models based on how active the user might be.

    We cluster the users into two main groups based on either weekly or daily engagement.

    Thus based on each user’s level of engagement, we decide which metric to use for the model’s objective function. We then train a gradient boosting decision tree (GBDT) classifier to predict if the user will be active in the next week given a notification budget ‘b’ this week.

    Training setup

    We frame this as a machine learning problem by having two different objectives to learn.

    Although there will be two models optimizing two different objectives, the features used in the current production system are retained. More information about the features used can be found in the KDD’18 paper

    The training labels are generated differently even though the same features are used for both models. For WAU, we set the label as 1 if the user has an engagement action on Pinterest at least one day in a week, otherwise 0. For DAU, if the user was active 4 days in a week, we set 4 positive labels and 3 (7–4) negative labels. We then train two separate models for the WAU and DAU user groups using the appropriate labels.

    Learning & results

    We ran A/B experiments on a significant fraction of users who only subscribed to push notifications.

    Our system was able to reduce the volume of notifications sent to low-engagement users who wouldn’t benefit from them without affecting the overall push open rates or any other key metrics. The system was able to send more notifications to users who would benefit from receiving them, increasing WAU counts by 1% in the enabled group. After doing deeper analysis on user state levels, it turns out the system was able to achieve the WAU increase by cutting down notifications to dormant users by 3% and increasing notifications to resurrected users by 17% and marginal users by 7%. Having different objective functions based on user activity states resulted in more efficient notification budget allocation.

    Distribution of notification among different user states for the existing prod and the new retrained model

    Future work

    User state based notification volume optimization has yielded very good results in improving user engagement. We’re working on other ways to improve our system, such as:

    • Creating additional groups if necessary for optimizing other metrics
    • Based on users’ preferred channel (email notifications, Push Notification, or both), moving some user states from WAU group to DAU group
    • Considering other metrics like revenue, impressions, clickthrough rate, or engagement to better optimize for highly-engaged users

    Acknowledgements:

    This is joint work between Yuxiang Wang and Koichiro Narita from the Growth Notifications Team. We would like to thank Bo Zhao for building the first version of the Budgeting System and providing valuable feedback as well as Yunsong Guo for valuable suggestions throughout this project.


    User state-based notification volume optimization was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

    Read more >

    Jobs people like you viewed

    UX Writer, Measurement & Insights

    Design 1399408 Seattle Washington United States Seattle, Washington, United States Design Product Design > Product Design
    About Pinterest: Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspi...

    Software Engineer, Core Product Data

    Engineering 1395273 San Francisco California United States San Francisco, California, United States Engineering Engineering > Eng
    About Pinterest: Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspi...

    Data Scientist, Ads Measurement

    Engineering 1398387 San Francisco California United States San Francisco, California, United States Engineering Engineering > Eng
    About Pinterest: Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspi...

    Fullstack Engineer, Advertiser Growth

    Engineering 1398229 San Francisco California United States San Francisco, California, United States Engineering Engineering > Eng
    About Pinterest: Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspi...

    Sales Coordinator (6 Months Fixed Term)

    Finance and Accounting 1397922 Dublin Ireland Dublin, Ireland Finance and Accounting Finance and Accounting > Finance
    About Pinterest: Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspi...

    Editorial Manager, Madrid or Milan (6 Months Fixed Term)

    Product > International Product 1396836 Milan Lombardy Italy Milan, Lombardy, Italy International Product Product > International Product
    About Pinterest: We’re looking for an Editorial Manager, with a passion for fashion, home decor and beauty to join our team. You’ll work closely with the Growth Manager as well as the wider international editorial team owning and developing the ed...

    Please register your interest

    Not You?

    Thank you