Customer Reliability Engineer

Global Remote / San Francisco, CA / San Francisco, California, United States·Full-time·Added 10 months ago

Andromeda Cluster

9 open roles

Apply

What they offer

Full-time hours
Per the ad.
Fully remote
Per the ad.

What they ask for

Have the right to work in Spain
Andromeda Cluster doesn't mention sponsorship, but this role may fit the Digital nomad visa.
Work in English
English is required, per the ad.

This job was published 10 months ago

The role

Location: Remote/SF-Hybrid · Full-Time

Our customers run large AI training and inference workloads on GPU clusters we source from providers worldwide. When a node goes dark or a job dies eight hours into a run, the Customer Reliability Engineer is who they hear from, and who gets it sorted.

The job has three parts. You triage incoming issues and debug them at the Linux and Kubernetes layer. You work provider-side to figure out whose fault something actually is and push external providers to fix it. And you build the monitoring and scripts that catch problems before a customer has to tell us.

You need to be comfortable in a Linux shell and know how Kubernetes works. You don't need GPU or HPC experience. Most people pick that up here.

What you'll do

Own issues start to finish: reproduce, diagnose, fix or escalate, close the loop
Debug at the Linux layer: processes, networking, storage, kernel logs, resource contention, systemd, journald
Dig into Kubernetes problems like pods stuck pending or crash-looping, node conditions, scheduling failures, resource limits
Work GPU failures: driver and device-plugin issues, XID errors, thermal throttling, nodes that need cordoning or draining, jobs failing across multiple nodes
Escalate when you're past your depth, with the evidence already gathered
Take part in a 24/7 on-call rotation
First response on alerts and customer-reported outages: assess impact, set severity, pull in the right people
Keep customers updated during incidents. Clear status, honest unknowns, no silence
Write up what happened, then turn it into a runbook, an alert, or a fix so it costs less next time
Work out whether a fault is provider-side, ours, or the customer's before it gets handed anywhere
Open tickets with compute providers and chase them down rather than waiting
Track recurring provider failures and flag the patterns to the people making sourcing decisions
Write Python or Bash to automate the checks you'd otherwise run by hand
Build and improve monitoring: cluster and node health checks, GPU telemetry, dashboards, alerts that fire on real problems
Keep runbooks and customer docs current as you go

Requirements

What we're looking for

Real Linux troubleshooting ability from the command line. You can work a problem through logs, processes, networking, and disk without a script to follow
Working knowledge of Kubernetes: pods, nodes, deployments, services, scheduling, and how to investigate when one of those breaks
Can write a script in Python or Bash to automate something repetitive
Strong writing. You can explain a technical problem to a frustrated customer clearly and without condescension
Good judgment under pressure. You know what to check first, when to escalate, and how to keep people informed while you're still working it out
Willing to join a 24/7 on-call rotation

Nice to have

Hands-on time with NVIDIA GPUs in production: drivers, CUDA, DCGM, the Kubernetes device plugin
Experience with high-performance networking (InfiniBand, RoCE) or NCCL
Experience with HPC or batch schedulers like Slurm
A previous customer-facing technical role: support engineering, TAM, solutions, professional services
Knowledge of Prometheus, Grafana, Datadog, or similar
IAC: Terraform, Ansible, or Helm
Genuine interest in AI infrastructure and how big training jobs behave

Pay & benefits

What you'll get

High-growth environment: Get in early at a company at the center of the AI infrastructure boom
Competitive compensation: + meaningful equity
Comprehensive benefits: for you and your dependents, including healthcare, dental, and vision coverage, 401(k), and unlimited PTO

About Andromeda Cluster

Andromeda Cluster is a technology company focused on AI infrastructure and compute markets, providing services and solutions related to high-performance computing and AI workloads. The company operates in the technology sector, with a focus on compute trading, reliability engineering, and strategic partnerships, indicating a strong emphasis on technical and operational excellence.

In Spain, Andromeda Cluster is establishing a presence, with open roles in areas such as compute trading, site reliability engineering, and technical program management. The company's work culture appears to be highly technical and fast-paced, making it an attractive option for international professionals with expertise in AI infrastructure, cloud computing, and large-scale systems. The roles are based in Spain, offering opportunities for global talent to contribute to cutting-edge compute and AI projects.

Good to know if you are moving

The company is actively hiring for technical roles in Spain, indicating a growing presence in the region.
Roles include positions in AI infrastructure and compute markets, which are high-demand fields globally.
The company uses Ashby for its careers page, suggesting a modern and structured hiring process.
International professionals with experience in AI infrastructure and reliability engineering may find relevant opportunities.

In their own words

About the company & team

Andromeda is a market and infrastructure platform to buy, sell, and operate compute.

We believe demand for compute will grow exponentially. So fast that a handful of vertically integrated providers won't be able to scale across operations, capital, supply chains, and politics to serve it. The result is a massive wave of fragmentation, with AI factories of every shape and size coming to market to fill this demand. Our job is to enable all of that fragmented compute to flow through one platform, delivering reliable capacity to model builders, research labs, and inference providers when they need it. We believe every spare electron should be made productive for AI and we're building the platform that makes that possible.

We sit at the center of three forces:

Companies that need reliable, high-performance compute fast
A fragmented global supply of GPUs across hyperscalers, neoclouds, and independent data centers
Capital, risk, and operational complexity that most teams are not equipped to manage

When we succeed, trillions of dollars of compute will flow through Andromeda. Builders get capacity when they need it. Providers get a reliable way to monetize, operate, and finance infrastructure at scale. Capital gets an easy way to deploy, hedge, and underwrite.

In five years, Andromeda won't just participate in the AI infrastructure market. We will shape it.

Additional information

Andromeda Cluster is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.

All open roles at Andromeda Cluster