Can work independently with minimal hand-holding

Support existing product / speed up development

About The Role:

As a Senior Network Operations Engineer, you are our front-line responder for break/fix incidents—owning alert triage, collaborating with SRE and MLOps teams, and driving rapid resolution to keep our global network and platform running smoothly.

You combine strong operational discipline with hands-on troubleshooting and a bias for automation. Beyond traditional networking, you’ll work hands-on with Kubernetes and Slurm to diagnose issues that span infrastructure, container networking, and HPC job fabrics.

You’re fluent in routing/switching and network security fundamentals, comfortable on Linux, and thrive in fast-moving environments where clear communication and crisp execution matter. You’ll improve monitoring, runbooks, and recovery playbooks to reduce MTTA/MTTR and prevent repeat incidents. Outstanding problem-solving abilities and a solid understanding of fundamental network theory are also critical to your success.

Responsibilities:

• Serve as first responder for network alerts and incidents: assess impact, prioritize, mitigate, and escalate as needed to SRE/MLOps/Network Engineering.
• Own end-to-end incident lifecycle: detection, triage, containment, remediation, comms, and post-incident reviews with clear timelines and action items.
• Monitor network health and capacity across routing/switching, firewalls, and data center fabrics; tune alert thresholds and dashboards to reduce noise.
• Troubleshoot L2–L4 issues (ARP, VLAN/VXLAN/EVPN, routing protocols, ACLs/NAT, DNS, TLS termination, QoS) using packet capture and flow/telemetry tools.
• Execute standard changes (MOPs) and emergency changes with rigorous change control and validation; document outcomes and update runbooks.
• Operate multi-cluster add-ons (e.g., MetalLB/Traefik/NGINX), observe health via Prometheus/Grafana/Loki, and tune alerts to reduce noise.
• Debug CNI/data plane (e.g., VXLAN/EVPN, iptables/nftables, network policies), kube-proxy/iptables mode, CoreDNS, Services (ClusterIP/NodePort/LoadBalancer), and Ingress/EGRESS.
• Maintain accurate network documentation: diagrams, inventories, IPAM, device configs, and topology state.
• Improve operational excellence: automate repetitive tasks, enhance self-service tooling, and contribute to SLOs, error budgets, and reliability roadmaps.
• Participate in a shared on-call rotation providing 24×7 coverage for critical services (once a month, 12-hour shifts).

Requirements

Must have:

• Strong network troubleshooting skills (Kubernetes, data-center, BGP troubleshooting experience preferred)
• Experience operating cloud-scale networks (ideally at Apple, Google, Microsoft, or OCI; experience at scale is needed, but smaller scale is fine if the work was in AI environments)
• Experience supporting Cisco, Arista, or Juniper network equipment
• Strong Linux skills and hands-on troubleshooting: diagnostics and log analysis.
• Foundational/theoretical knowledge of InfiniBand and RoCEv2 (must understand what they are and how they work)
• Clear, concise communicator—able to write incident timelines, RCAs, and user-facing updates under time pressure

Ideal profile:

• 3+ years in a NOC/Network Operations or Network Support role for large-scale data center or service provider-style environments (hybrid/on-prem + cloud)
• Solid understanding of TCP/IP and core protocols: BGP, OSPF/IS-IS, VLAN, VXLAN, EVPN, ACLs/NAT, DHCP, DNS, and QoS
• Proficiency with troubleshooting tools: Wireshark/tcpdump, mtr/traceroute, nmap, curl, iperf
• Familiarity with AWS/GCP/Azure networking concepts (VPC/VNet, IGW/NATGW, peering, PrivateLink, routing, security groups)

Nice to have:

• Hands-on Kubernetes troubleshooting experience: CNI fundamentals (policies, encapsulation), Services/Ingress, DNS (CoreDNS), kube-proxy, and container runtime basics (huge plus)
• Understanding of AI training workloads and the demands they put on networks
• Scripting (Bash/Python) and Git-based workflows for config versioning and change reviews
• Automation experience

About us:

A research-driven artificial intelligence company. We believe open and transparent AI systems will drive innovation and create the best outcomes for society, and together we are on a mission to significantly lower the cost of modern AI systems by co-designing software, hardware, algorithms, and models. We have contributed to leading open-source research, models, and datasets to advance the frontier of AI, and our team has been behind technological advancement such as FlashAttention, Hyena, FlexGen, and RedPajama. We invite you to join a passionate group of researchers and engineers in our journey in building the next generation AI infrastructure

The core profile is a network operations engineer with strong production troubleshooting and incident ownership. Kubernetes and AI/HPC networking would make the candidate stronger.

1–2 interviews, troubleshooting

Slack, video calls, direct collaboration with principal engineers and team leads

6 network engineers; the team is responsible for everything from doing network design, network provisioning, operations, both on the network and also supporting full stack.