Staff Software Engineer, ML Training Platform
Job Summary
The role involves architecting and maintaining large-scale machine learning training infrastructure, focusing on GPU optimization and system performance. The candidate will lead the development of tools and systems that support ML workflows, enhance model deployment, and improve efficiency. Required experience includes distributed training, container orchestration, and MLOps practices. The position offers a remote-friendly environment with comprehensive benefits and emphasizes mentorship and infrastructure optimization within Reddit's ML platform team.
Required Skills
Benefits
Job Description
Location:
This role is completely remote-friendly. If you happen to live close to one of our physical office locations, our doors are open for you to come into the office as often as you'd like.
Who We Are:
The Machine Learning Platform team at Reddit is a high-impact team that owns the infrastructure that powers recommendations, content discovery, user and content quantification, while directly impacting other teams such as Growth, Ads, Feeds, and Core Machine Learning teams.
What You’ll Do:
As a Staff Software Engineer, ML Training Platform, this person will be instrumental in architecting, implementing, and maintaining foundational Machine Learning Training infrastructure that powers Feeds Ranking, Content Understanding, Recommendations and much more to fulfill Reddit’s mission of bringing community and belonging to everyone in the world. You will oversee GPU optimization in AI/ML batch workloads and debug performance bottlenecks in GPU workloads. You will build and own systems and tools that enable MLEs and data scientists, and continuously improve the ML software development lifecycle.
- Optimize model training on GPUs
- Lead the building, testing, and maintenance of ML infrastructure at Reddit
- Propose, design, and implement high-performance ML platform solutions that significantly advance the deployment of models that serve millions of redditors a seamless experience for MLEs
- Play a pivotal role in designing, building, and optimizing the infrastructure and tooling required to support large-scale machine learning workflows
- Analyze bottlenecks in distributed systems and optimize for performance and cost-efficiency
- Work with management on team goal setting, planning, and de-risk project execution
- Mentor other team members in adopting a rigorous DevOps approach to maintain and/or improve ML platform components and services health and quality
Who You Might Be:
- 8+ years of work experience in a production software development environment or building data systems
- Experience with XLA for Tensorflow or torch.inductor for pytorch for kernel fusion during training
- Experience with optimization of data workloads using collosal.AI or Deepspeed
- Experience with distributed Training optimization using deepspeed, horovod or collosalAI
- Experience with design and architecture of large scale ML Systems
- Experience with training workflows, hyperparameter tuning, and resource optimization on CPU and GPU
- Experience with MLOps practices and tools such as Ray and MLFlow
- Hands-on experience with Kubernetes, Docker, or other container orchestration systems
Benefits:
- Comprehensive Healthcare Benefits and Income Replacement Programs
- 401k Match
- Family Planning Support
- Gender-Affirming Care
- Mental Health & Coaching Benefits
- Flexible Vacation & Reddit Global Days off
- Generous paid Parental Leave
- Paid Volunteer time off
#LI-DB1 #LI-Remote
Pay Transparency:
This job posting may span more than one career level.
In addition to base salary, this job is eligible to receive equity in the form of restricted stock units, and depending on the position offered, it may also be eligible to receive a commission. Additionally, Reddit offers a wide range of benefits to U.S.-based employees, including medical, dental, and vision insurance, 401(k) program with employer match, generous time off for vacation, and parental leave. To learn more, please visit https://www.redditinc.com/careers/.
To provide greater transparency to candidates, we share base pay ranges for all US-based job postings regardless of state. We set standard base pay ranges for all roles based on function, level, and country location, benchmarked against similar stage growth companies. Final offer amounts are determined by multiple factors including, skills, depth of work experience and relevant licenses/credentials, and may vary from the amounts listed below.
Reddit is proud to be an equal opportunity employer, and is committed to building a workforce representative of the diverse communities we serve. Reddit is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If, due to a disability, you need an accommodation during the interview process, please let your recruiter know.
A social media platform where users create and participate in communities (subreddits) to discuss a wide range of topics, share content, and connect.
See more jobsSafe Remote Job Search Tips
Verify Employer Thoroughly
Research the company's identity thoroughly before applying. Check for a professional website with contacts, active social media, and LinkedIn profiles. Verify details across platforms and look for reviews on Glassdoor or Trustpilot to confirm legitimacy.
Never Pay to Get a Job
Legitimate employers never require payment for applications, training, background checks, or equipment. Always reject upfront payment requests or demands for bank details, even if they claim it's for purchasing necessary work gear on your behalf.
Safeguard Your Personal Information
Protect sensitive data like SSN, bank details, or ID copies. Share this only after accepting a formal, written job offer. Ensure it's submitted via a secure company system or portal, never through insecure channels like standard email attachments.
Scrutinize Communication & Interviews
Watch for communication red flags: poor grammar, generic emails (@gmail), vague details, or undue pressure. Be highly suspicious of interviews held only via text or chat apps; legitimate companies typically use video or phone calls.
Beware of Unrealistic Offers
If an offer's salary or benefits seem unrealistically high for the work involved, be cautious. Research standard pay for similar roles. Offers that appear 'too good to be true' are often scams designed to lure you into providing information or payment.
Insist on a Formal Contract
Always secure and review a formal, written job offer or employment contract before starting work or sharing final personal details. Ensure it clearly defines your role, compensation, key terms, and conditions to avoid misunderstandings or scams.