Maria de Angelis | Summer 2019 Pinterest Engineer Intern
HTTPS should be everywhere. Your website likely runs over HTTPS, giving your users a secure, encrypted connection, but what about all the outbound links? Can you guarantee that they all use HTTPS? In this post, we describe the steps we took to upgrade Pin links from HTTP to HTTPS by leveraging DuckDuckGo’s Smarter Encryption technology.
While Pinterest runs entirely on HTTPS, it’s not the final destination for most users. As a place for Pinners to discover and do what they love, Pinterest is the launching pad for reaching other websites. As such, we have a responsibility to redirect Pinners to HTTPS sites whenever we can, including upgrading outbound Pin links from HTTP to HTTPS when possible and maximizing the percentage of traffic through HTTPS.
The reason that any HTTP traffic exists is simply that many Pins were created that way. When a Pinner clicks on a Pin link, she’ll be sent to the HTTP site if the content was created with an HTTP URL. However, if the site supports HTTPS, we want to send the user to the HTTPS version instead. We decided to perform an online upgrade to HTTPS so we could maximize the result on all Pinterest surfaces.
After making improvements, about 80% of outbound traffic is now through HTTPS, an increase of over 30%.
In order to ensure this change did not interfere with important metrics, we ran an experiment to release these changes to 1% of users and compare them to an equivalent control group. We found there was no change in benchmark Pinterest metrics, so we’ll continue to release the experiment to more Pinterest users.
Figure 1: This Pinterest board’s Pins are color coded by the protocol of their links. HTTPS links are green and HTTP links are red. When the experiment is enabled in the second picture, most red Pins become green because their links can be upgraded to HTTPS.
Smarter Encryption by DuckDuckGo
To work as efficiently as possible, we integrated DuckDuckGo’s Smarter Encryption technology which automatically uses encrypted connections to websites when available. DuckDuckGo was the perfect fit for us because they maintain a comprehensive list of upgradable sites, generated by comparing the HTTP and HTTPS version of a site, and adding a site to the HTTPS upgrade list if the two versions are identical. We can then regularly pull and ingest their list.
When a user scrolls through their Pinterest home feed, a request to fetch Pins is made in the API layer, which then calls the Apache Thrift service PinAndBoardService, which subsequently fetches Pins from the MySQL database.
Figure 2: A simplified representation of Pin fetching architecture. The API layer makes a request to the PinAndBoardService which then fetches the Pin from the cache or MySQL.
To implement Pin link conversion, we:
- Trigger whether or not to run the experiment in the API layer.
- If the experiment is enabled, call a newly added endpoint in PinAndBoardService in the API layer.
- In the new PinAndBoardService endpoint, check if a Pin’s URL begins with HTTP when it is fetched from the database or cache, in which case it should be upgraded to HTTPS if possible.
- Check if a Pin is able to be upgraded to HTTPS by stripping the domain from the URL and checking if the domain is contained in a list of secure domains provided by DuckDuckGo’s Smarter Encryption.
- Perform the conversion and upgrade the link so that all downstream services will receive the secure version.
- We chose to perform the link conversion in PinAndBoardService instead of directly in MySQL database for several reasons:
- If we upgraded a HTTP link to HTTPS in the MySQL layer and later on that domain no longer supported HTTPS, the Pin link would break and we would not be able to track the change since the link would have been permanently modified. To solve this issue, we could have stored both the old and new links in MySQL, but that would have been less space efficient than the current approach.
- Although we have to perform the conversion everytime a Pin gets fetched from the database (regardless of whether it has already been upgraded), we can generally avoid redundant fetching because Pins are highly cached.
As we move forward, we’re exploring permanently upgrading Pins to secure HTTPS links upon creation, a big step towards universal HTTPS usage and increased security for all users on our platform. We are also tracking the links that we cannot currently upgrade to potentially contribute to DuckDuckGo’s Smarter Encryption in order to improve the user experience for all users.
Huge thanks to Emanuele Cesena (my awesome intern mentor) and the rest of the Security team at Pinterest! Also huge thanks to the CoreService, Storage & Caching, and API teams for providing support and DuckDuckGo for providing early access to their Smarter Encryption technology.
Privacy Power-Up: Upgrading Pin Links from HTTP to HTTPS was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.