GitLab
Intermediate Machine Learning Engineer, AI Powered: Custom Models
Job Summary
The role involves developing and evaluating AI and machine learning models, particularly Large Language Models (LLMs), to ensure quality and performance. Candidates should have experience in ML model deployment, evaluation metrics, and working with cloud platforms like AWS and GCP. The position requires collaboration with cross-functional teams and a strong background in Python and ML infrastructure. The team emphasizes working remotely with benefits supporting health, growth, and work-life balance.
Required Skills
Benefits
Job Description
GitLab is an open core software company that develops the most comprehensive AI-powered DevSecOps Platform, used by more than 100,000 organizations. Our mission is to enable everyone to contribute to and co-create the software that powers our world. When everyone can contribute, consumers become contributors, significantly accelerating the rate of human progress. This mission is integral to our culture, influencing how we hire, build products, and lead our industry. We make this possible at GitLab by running our operations on our product and staying aligned with our values. Learn more about Life at GitLab.
Thanks to products like Duo Enterprise, and Duo Workflow, customers get the benefit of AI at every stage of the SDLC. The same principles built into our products are reflected in how our team works: we embrace AI as a core productivity multiplier. All team members are encouraged and expected to incorporate AI into their daily workflows to drive efficiency, innovation, and impact across our global organisation.
An overview of this role
The Custom Models team is responsible for Duo Self-Hosted, a key component in GitLab AI that allows customers to run GitLab Duo features on completely private environments, connecting GitLab to their own AI models. They work collaboratively with numerous teams to ensure a complete lifecycle of assessing models, evaluating features, implementing controls for customization and guaranteeing a smooth experience for our largest customers.
Why us? This isn't just a job; it's your chance to shape the future of AI at GitLab. Your expertise in backend development will be critical to your success. Ready to dive into the future of AI at GitLab? Apply now! We're excited to meet potential candidates like you and welcome a new star to our team. Let's shape the future together!
What you’ll do
- Develop evaluation techniques to assist feature teams on guaranteeing the quality of their features on new models.
- Evolve the Evaluation Runner, our internal tool for scaling AI Feature evaluation
- Keep up with the industry to explore brand new models.
- Deploy and manage LLMs internally for evaluation and development.
- Collaborate with product managers, engineers, and other stakeholders as a machine learning specialist, guiding them on effective implementation of AI features.
- Advocate for improvements to product quality, security, and performance.
- Solve technical problems of moderate scope and complexity.
- Craft code that meets our internal standards for style, maintainability, and best practices for a high-scale machine-learning environment. Maintain and advocate for these standards through code review.
- Confidently ship small features and improvements with minimal guidance and support from other team members. Collaborate with the team on larger projects.
- Participate as a reviewer or project maintainer in one or more engineering projects.
What you’ll bring
- A relevant Master’s degree and 2 or more years of experience in ML or PhD degree with a focus on Machine Learning or Data Science.
- Professional experience with Python.
- Experience with performance and optimization problems and a demonstrated ability to both diagnose and prevent these problems
- Comfort working in a highly agile, intensely iterative software development process
- Demonstrated ability to onboard and integrate with an organization long-term
- Positive and solution-oriented mindset
- Effective communication skills: Regularly achieve consensus with peers, and clear status updates
- An inclination towards communication, inclusion, and visibility
- Experience owning a project from concept to production, including proposal, discussion, and execution.
- Self-motivated and self-managing, with strong organizational skills.
- Demonstrated ability to work closely with other parts of the organization
- Share our Values, and work in accordance with those values
- Ability to thrive in a fully remote organization
Two of more of
- Professional experience with prompt engineering and Retrieval Augmented Generation (RAG)
- Experience with evaluation of Machine Learning models or AI features.
- Experience with model deployments on cloud platforms like AWS and GCP.
- Proven experience designing and implementing LLM evaluation systems.
- Strong understanding of ML model architectures.
- Expertise in ML evaluation metrics and dataset management.
- Demonstrated ability to build production-grade ML infrastructure.
- Practical experience with Python-based ML frameworks and evaluation tools (e.g., Langsmith, DSpy).
Bonus qualifications
- Have contributed a Merge Request to GitLab
- Have contributed to ML open source projects
About the team
The Custom Models team is a team formed of GitLab team members from around the globe. Engineers are primarily located across various European countries, with some distributions in Australia, New Zealand, America, and Canada.
The team works closely with these other teams within the organization:
- AI Framework
- MLOps
- Model Validation
- Duo Chat
How GitLab will support you
- Benefits to support your health, finances, and well-being
- All remote, asynchronous work environment
- Flexible Paid Time Off
- Team Member Resource Groups
- Equity Compensation & Employee Stock Purchase Plan
- Growth and development budget
- Parental leave
- Home office support
Please note that we welcome interest from candidates with varying levels of experience; many successful candidates do not meet every single requirement. Additionally, studies have shown that people from underrepresented groups are less likely to apply to a job unless they meet every single qualification. If you're excited about this role, please apply and allow our recruiters to assess your application.
Country Hiring Guidelines: GitLab hires new team members in countries around the world. All of our roles are remote, however some roles may carry specific location-based eligibility requirements. Our Talent Acquisition team can help answer any questions about location after starting the recruiting process.
Privacy Policy: Please review our Recruitment Privacy Policy. Your privacy is important to us.
GitLab is proud to be an equal opportunity workplace and is an affirmative action employer. GitLab’s policies and practices relating to recruitment, employment, career development and advancement, promotion, and retirement are based solely on merit, regardless of race, color, religion, ancestry, sex (including pregnancy, lactation, sexual orientation, gender identity, or gender expression), national origin, age, citizenship, marital status, mental or physical disability, genetic information (including family medical history), discharge status from the military, protected veteran status (which includes disabled veterans, recently separated veterans, active duty wartime or campaign badge veterans, and Armed Forces service medal veterans), or any other basis protected by law. GitLab will not tolerate discrimination or harassment based on any of these characteristics. See also GitLab’s EEO Policy and EEO is the Law. If you have a disability or special need that requires accommodation, please let us know during the recruiting process.
GitLab
From planning to production, bring teams together in one application. Ship secure code more efficiently to deliver value faster.
See more jobsSafe Remote Job Search Tips
Verify Employer Thoroughly
Research the company's identity thoroughly before applying. Check for a professional website with contacts, active social media, and LinkedIn profiles. Verify details across platforms and look for reviews on Glassdoor or Trustpilot to confirm legitimacy.
Never Pay to Get a Job
Legitimate employers never require payment for applications, training, background checks, or equipment. Always reject upfront payment requests or demands for bank details, even if they claim it's for purchasing necessary work gear on your behalf.
Safeguard Your Personal Information
Protect sensitive data like SSN, bank details, or ID copies. Share this only after accepting a formal, written job offer. Ensure it's submitted via a secure company system or portal, never through insecure channels like standard email attachments.
Scrutinize Communication & Interviews
Watch for communication red flags: poor grammar, generic emails (@gmail), vague details, or undue pressure. Be highly suspicious of interviews held only via text or chat apps; legitimate companies typically use video or phone calls.
Beware of Unrealistic Offers
If an offer's salary or benefits seem unrealistically high for the work involved, be cautious. Research standard pay for similar roles. Offers that appear 'too good to be true' are often scams designed to lure you into providing information or payment.
Insist on a Formal Contract
Always secure and review a formal, written job offer or employment contract before starting work or sharing final personal details. Ensure it clearly defines your role, compensation, key terms, and conditions to avoid misunderstandings or scams.