Person, Flooring, Wood, Floor, Hardwood, Furniture, Plywood, Couch, Sphere

 

Data Labeling and Relevance Program Manager

 

Data Labeling and Relevance Program Manager

San Francisco | California | United States

About Pinterest:

Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspiration and create a life they love. As a Pinterest employee, you’ll be challenged to take on work that upholds this mission and pushes Pinterest forward. You’ll grow as a person and leader in your field, all the while helping users make their lives better in the positive corner of the internet.

We're looking for a talented Program Manager to help us grow Pinterest internationally, focusing on human computation and content relevance. We need a dedicated person to run this program, which includes setting and running evaluations at scale for any team at Pinterest and managing our testing vendors. In order to meet the needs of engineers, PMs, data scientists, and designers who need data to be tested by our vendors, the Data Labeling and Relevance Program Manager will own the full process from inception to data delivery. 

What you’ll do: 

  • Meet with the human evaluation requester to define what needs to be tested and what the outcome of the testing will be (validation of an experiment, data labeling, or content quality assessment) 
  • Provide a complete testing interface with guidelines, a training guide, and label recommendations, to be validated by the requester
  • Select the best vendor to perform the judgments 
  • Launch tests and tracking progress; You will regularly meet with our vendor to calibrate results to ensure a high quality bar 
  • Provide results and analysis to the requester 
  • Localize the labeling guidelines into several languages, leveraging our localization team 
  • Lead our relevance efforts across organic and ads products, and represent the voice of users when providing results and suggestions of improvements to the product team
  • Manage our human computation budget 

What we’re looking for: 

  • 3-5+ years of experience with human judgment platforms, human computation, survey design, crowdsourcing, or experimental design
  • Ability to translate and synthesize data and complex information into a compelling narrative
  • Excellent written and verbal communication skills
  • Experience with HTML, XML, CSS, JavaScript, and SQL or specialized database query languages a definite plus
  • Knowledge of concepts like precision/recall and f-scores
  • Active Pinterest user with a strong interest in our product 

#LI-REX

Apply Now

Not You?

Thank you

Recommended Jobs

Senior Privacy and Data Protection Counsel

Legal and Policy 1350633 San Francisco California United States San Francisco, California, United States Legal and Policy Legal and Policy > Legal
About Pinterest: Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspi...

Finance Associate, Infrastructure

Finance and Accounting 1351705 San Francisco California United States San Francisco, California, United States Finance and Accounting Finance and Accounting > Finance
About Pinterest: Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their inspi...

IT Client Platform Engineer (Contract)

IT 1350945 San Francisco California United States San Francisco, California, United States IT IT > IT
About Pinterest:  Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their insp...

Technical Program Manager, Shopping

Engineering 1350632 San Francisco California United States San Francisco, California, United States Engineering Engineering > Eng
About Pinterest:  Millions of people across the world come to Pinterest to find new ideas every day. It’s where they get inspiration, dream about new possibilities and plan for what matters most. Our mission is to help those people find their insp...

Recommended Reading:

Our Mission

A couple words from CEO Ben Silbermann on how Pinterest chooses to do the right thing on the internet

What's new on the platform

Want to stay up to date on what Pinterest is up to? Check out our company blog!

Building an Inclusive Company

In 2015, Pinterest first published our diversity and inclusion hiring goals to keep ourselves accountable. Five years later, check in on how we're doing and how those goals have evolved.

Our Employee Resource Groups

Pinterest is home to 10+ ERGs, hear from them on how they're inspiring and staying inspired at Pinterest.

We’re looking for all kinds of people


To build an app that’s used and loved by people all around the world, we need a team with all kinds of different perspectives, experiences and backgrounds.

To put it legally: Pinterest is an equal opportunity employer and makes employment decisions on the basis of merit. We want to have the best qualified people in every job. Pinterest policy prohibits unlawful discrimination based on race, color, religion or religious creed, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender, gender identity, gender expression, age, marital status, status as a protected veteran, physical or mental disability, medical condition, genetic information or characteristics (or those of a family member), or any other consideration made unlawful by applicable federal, state, or local laws. It also prohibits discrimination based on a perception that anyone has any of those characteristics, or is associated with a person who has or is perceived as having any of those characteristics. All such discrimination is unlawful. Pinterest is committed to compliance with all applicable laws providing equal employment opportunities. This commitment applies to all persons involved in the operations of Pinterest and prohibits unlawful discrimination by any employee of Pinterest.

Pinterest is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or an accommodation due to a disability, please contact us at accessibility@pinterest.com.

Pinterest will consider qualified applicants with arrest and conviction records for positions in accordance with the San Francisco Fair Chance Ordinance, the Los Angeles Fair Chance Ordinance, and other applicable state or local laws.

Pinterest also participates in the E-Verify program in certain locations, as required by law. See https://www.e-verify.gov/sites/default/files/everify/posters/EVerifyParticipationPoster.pdf and https://www.e-verify.gov/sites/default/files/everify/posters/IER_RighttoWorkPoster.pdf