We are seeking a Senior Engineer to design, automate, and operate our next-generation object storage platform built on Ceph, Rook, and Kubernetes. This platform provides S3-compatible storage for AI training, inference, and enterprise workloads at multi-petabyte scale.
You will work closely with infrastructure, networking, platform engineering, and site reliability teams to build a highly available, secure, and automated storage service supporting thousands of GPUs and exabytes of future growth.
Responsibilities:
- Design, deploy, and operate large-scale Ceph clusters using Rook on Kubernetes.
- Automate infrastructure provisioning, upgrades, and lifecycle management using Infrastructure-as-Code tools.
- Build and maintain CI/CD pipelines for storage platform deployments.
- Monitor cluster health, capacity, performance, and reliability using Prometheus, Grafana, and related observability tools.
- Optimize Ceph performance, including OSD placement, erasure coding, CRUSH maps, networking, and recovery behavior.
- Manage Ceph services including MON, MGR, RGW, OSD, CephFS (if applicable), and supporting infrastructure.
- Implement disaster recovery, backup, replication, and operational runbooks.
- Collaborate with networking teams on high-speed Ethernet or InfiniBand fabrics and RDMA-enabled storage architectures.
- Develop automation for cluster expansion, hardware replacement, drive failures, and capacity management.
- Participate in on-call rotation and incident response for production storage services.
- Troubleshoot distributed systems issues across compute, networking, Kubernetes, and storage layers.
Required Qualifications
- 5+ years of experience operating Linux production infrastructure.
- 3+ years managing Kubernetes clusters in production.
- Hands-on experience deploying and operating Ceph in production environments.
- Built file-system code and run production clusters: you have committed code to Ceph itself, not only operated or managed it.
- Experience with Rook Ceph Operator or similar Kubernetes storage operators.
- Strong understanding of distributed systems, high availability, and fault tolerance.
- Experience with Infrastructure-as-Code tools such as Terraform, Ansible, or Pulumi.
- Experience building automation using Python, Go, or Bash.
- Strong networking knowledge including TCP/IP, VLANs, BGP, LACP, MTU tuning, and high-speed Ethernet (100/200/400 GbE).
- Experience with Prometheus, Grafana, Alertmanager, and centralized logging platforms.
- Excellent troubleshooting and root cause analysis skills.
Preferred Qualifications
- Experience operating storage platforms at petabyte scale.
- Experience with AI/ML infrastructure or GPU clusters.
- Familiarity with S3-compatible object storage and Ceph RGW.
- Experience with erasure coding, CRUSH map tuning, and storage performance optimization.
- Experience with RDMA, RoCE, or InfiniBand networking.
- Experience with multi-site replication and disaster recovery.
- Familiarity with security best practices including TLS, KMS integration, RBAC, and secrets management.
- Experience operating cloud-native infrastructure across multiple data centers.
Nice to Have
- Experience with Alluxio, MinIO, or other distributed storage systems.
- Experience with GitOps platforms such as Argo CD or Flux.
- Experience with NVIDIA GPU infrastructure and AI training workloads.
- Knowledge of storage benchmarking tools (FIO, COSBench, S3Bench).
- Red Hat Ceph Storage, Kubernetes, or CNCF certifications.
What Success Looks Like
- Operate highly available Ceph clusters with 99.95%+ service availability.
- Automate routine operational tasks to minimize manual intervention.
- Continuously improve cluster performance, scalability, and operational efficiency.
- Reduce deployment time through automation and standardized operational practices.
- Enable reliable, scalable object storage supporting enterprise AI workloads at multi-petabyte scale.
About us:
A research-driven artificial intelligence company. We believe open and transparent AI systems will drive innovation and create the best outcomes for society, and together we are on a mission to significantly lower the cost of modern AI systems by co-designing software, hardware, algorithms, and models. We have contributed to leading open-source research, models, and datasets to advance the frontier of AI, and our team has been behind technological advancement such as FlashAttention, Hyena, FlexGen, and RedPajama. We invite you to join a passionate group of researchers and engineers in our journey in building the next generation AI infrastructure