Grafana Labs
Senior Backend Engineer, Observability: Ingest (Remote, USA)
Job Summary
This role involves developing and maintaining backend systems for observability platforms, with a focus on metrics, logs, traces, and profiles. The engineer will collaborate on cloud-based projects, contribute to open source initiatives like OpenTelemetry and Grafana Alloy, and participate in system design, deployment, and operations. Candidates should have experience with programming languages like Go and familiarity with Kubernetes, cloud systems, and distributed architecture. Strong communication skills and the ability to work independently in a remote setup are essential.
Required Skills
Benefits
Job Description
Senior Backend Engineer, Observability: Ingest (Remote, NASA)
This is a remote position. We are looking for candidates in NASA.
What we do
We are the creators and maintainers of Fleet Management, the Kubernetes Monitoring Helm Chart, the cloud-based part of the Frontend Observability ingestion pipeline and other key building blocks to allow our customer base to deploy, configure, administrate and monitor hundreds of thousands of Observability Collector instances worldwide. Our mission is to streamline the transport of Metrics, Logs, Traces and Profiles to Grafana Cloud (and our OSS counterparts) and ensure that required infrastructure can be operated with ease. We do so by working closely with many other teams at Grafana, providing them with a platform to configure the Observability signal ingestion pipeline to their needs as well as close collaboration with our clients and the wider OSS community to ensure our deliverables satisfy their needs.
Some of the projects we have undertaken include:
- The Kubernetes Monitoring helm chart to easily deploy required Observability infrastructure in kubernetes clusters
- Fleet Management, a cloud based offering to enable our customers to manage remotely thousands of Alloy instances through a central UI (Alloy is our OpenTelemetry Collector distribution),
- The ingestion pipeline for frontend observability traffic so that no additional infrastructure is required by our customers
- Contribution of various components to the OpenTelemetry Collector project (e.g. Faro Receiver and Exporter) and Alloy (e.g. remote configuration)
What will you be doing?
- Collaborate with your team to deliver new features, analyze outcomes, and make improvements
- Lead projects from concept to implementation, including ongoing customer support
- Become an active contributor of open source projects like Grafana Alloy and the OpenTelemetry Collector
- Design, build, operate, and maintain essential systems, ensuring reliability, performance, and availability
- Take an active role in influencing our roadmap and your own career objectives
- Participate in on-call rotations and take responsibility for the services you oversee
- Support and mentor team members, engage in design conversations, and work closely with colleagues
- Expand your skill set by deepening your knowledge of our cloud products, understanding our customers, and learning about our codebase
As we have embraced a remote-first approach and our engineering team is primarily remote, strong communication skills and the ability to work independently are essential. We provide support and hold regular meetings through video calls to ensure effective collaboration and alignment.
What are we looking for in you?
- You are a motivated self-starter with a bias toward action
- You have a passion for creating intuitive products that fit customers’ needs.
- While the vast majority of your work will be focused on the backend of 1 product, you don’t shy away from making bugfixes and other small changes to other projects (incl. frontend)
- Pragmatism: You are able to take on complex challenges and break them down to achieve short feedback loops: to analyze, design, and build modular solutions, deliver MVPs, gather data and feedback and then progress iteratively
- Collaboration and communication: The smallest unit we have is a team. You’ll be working with your teammates in a fully remote setup. Good communication skills are a must
Requirements:
- Solid experience with at least one programming language. We use Go, but if you have familiarity with Python, C, C++, Rust or similar then that translates well
- Experience with delivering projects, from gathering requirements, brain-storming ideas all the way to shipping a product to the customer’s hands in a self-driven way
- Experience with developing software that runs in the Cloud or some experience with systems engineering
- Experience with being on-call and performing operations/SRE tasks or with the concept of infrastructure as code
Nice to haves:
- Experience working with Kubernetes
- Been a user of Grafana and Prometheus in operational roles (including on-call for your team at a previous employer or just using these tools on hobby/homelab projects)
- Exposure to microservices architecture and distributed systems
In the US, the Base compensation range for this role is $148,505 - $178,206. Actual compensation may vary based on level, experience, and skillset as assessed in the interview process. Benefits include equity, bonus (if applicable) and other benefits listed here.
*Compensation ranges are country-specific. If you are applying for this role from a different location than listed above, your recruiter will discuss your specific market’s defined pay range & benefits at the beginning of the process.
About Grafana Labs: There are more than 20M users of Grafana, the open source visualization tool, around the globe, monitoring everything from beehives to climate change in the Alps. The instantly recognizable dashboards have been spotted everywhere from a NASA launch and Minecraft HQ to Wimbledon and the Tour de France. Grafana Labs also helps more than 3,000 companies -- including Bloomberg, JPMorgan Chase, and eBay -- manage their observability strategies with the Grafana LGTM Stack, which can be run fully managed with Grafana Cloud or self-managed with the Grafana Enterprise Stack, both featuring scalable metrics (Grafana Mimir), logs (Grafana Loki), and traces (Grafana Tempo).
Grafana Labs
Grafana is the open source analytics & monitoring solution for every database.
See more jobsSafe Remote Job Search Tips
Verify Employer Thoroughly
Research the company's identity thoroughly before applying. Check for a professional website with contacts, active social media, and LinkedIn profiles. Verify details across platforms and look for reviews on Glassdoor or Trustpilot to confirm legitimacy.
Never Pay to Get a Job
Legitimate employers never require payment for applications, training, background checks, or equipment. Always reject upfront payment requests or demands for bank details, even if they claim it's for purchasing necessary work gear on your behalf.
Safeguard Your Personal Information
Protect sensitive data like SSN, bank details, or ID copies. Share this only after accepting a formal, written job offer. Ensure it's submitted via a secure company system or portal, never through insecure channels like standard email attachments.
Scrutinize Communication & Interviews
Watch for communication red flags: poor grammar, generic emails (@gmail), vague details, or undue pressure. Be highly suspicious of interviews held only via text or chat apps; legitimate companies typically use video or phone calls.
Beware of Unrealistic Offers
If an offer's salary or benefits seem unrealistically high for the work involved, be cautious. Research standard pay for similar roles. Offers that appear 'too good to be true' are often scams designed to lure you into providing information or payment.
Insist on a Formal Contract
Always secure and review a formal, written job offer or employment contract before starting work or sharing final personal details. Ensure it clearly defines your role, compensation, key terms, and conditions to avoid misunderstandings or scams.