Person, Flooring, Wood, Floor, Hardwood, Furniture, Plywood, Couch, Sphere

 

Software Engineer, Big Data Workflow Platform

 

Software Engineer, Big Data Workflow Platform

Palo Alto | California | United States

We’re looking for Big data workflow/orchestration  platform engineers (across levels) to help us build and lead the next generation of ETL orchestration platform at Pinterest. You’ll be working on some of the most exciting big data open source technologies (Airflow, Hadoop, Hive, Spark, Presto, , etc.), at the scale of hundreds of petabytes of data to help Pinners discover and do what they love. 

What you’ll do:

  • Improve and customize the internals of open source big data technologies to meet our challenges at scale
  • Build and scale workflow orchestration platforms using the latest open-source technologies to process petabytes-scale datasets
  • Build and scale systems that orchestrate and execute complex workflows in big-data pipelines
  • Provide reliable, performant  workflow engines as services to enable ETL, big data analytics and actionable insights
  • Contribute to the team’s technical vision and long-term roadmap

What we’re looking for:

  • Experience with large scale distributed framework 
  • Experience with big data workflow orchestration engines for ETL jobs, such as Airflow, Oozie, Azkaban, etc.
  • Knowledge of container and orchestration frameworks such as Docker and Kubernetes
  • Experiences in one or more open-source Big Data technologies ( Airflow, Hadoop, YARN, Spark, Hive, Presto, SQL, Kafka, Impala,  Parquet, HDFS, etc.)
  • Proficiency in one or more programming languages (Python, Java, Scala)

#LI-KL1

Apply Now

Not You?

Thank you

Recommended Jobs

Software Engineer, Ads Infrastructure

Palo Alto California United States Palo Alto, California, United States Engineering Engineering > Eng
Pinterest is one of the fastest growing online advertising platforms and our continued success depends on the reliability, performance, and scalability of the Ads infrastructure.  You’ll help us build the next-generation Ads infrastructure that wi...

Software Engineer, Big Data Query Platform

San Francisco California United States San Francisco, California, United States Engineering Engineering > Eng
We’re looking for software engineers to help us build a high-performance and efficient big data query platform and infrastructure at Pinterest. You’ll be working on some of the most exciting big data open source technologies such as Spark, Presto,...

Software Engineer, Analytics Platform

San Francisco California United States San Francisco, California, United States Engineering Engineering > Eng
You’ll design, build, support, and own Pinterest’s critical internal data tools end-to-end. Examples of these are Pinterest’s in-house ad-hoc querying IDE, A/B experiments platform, internal analytics tracking app, visual data workflow composer, J...

Software Engineer, Real-Time Analytics

San Francisco California United States San Francisco, California, United States Engineering Engineering > Eng
Pinterest brings millions of Pinners the inspiration to create a life they love for everything; whether that be tonight’s dinner, next summer’s vacation, or a dream house down the road. Our Real Time Analytics team owns, develops and maintains two...