Lemon.io

Storage Infrastructure Engineer – Ceph Platform | AI Systems

#65816025162

DevOps, Senior · Platform Engineer

Basic info

Involvement40h/week
Project typeFull-time
Durationongoing
Start datewithin 2-3 weeks
Client time zone(GMT -7:00) Mountain Time (US & Canada)

Skills

Must-have:

Kubernetes
Linux
Ceph
Python
Golang
Networking
Pulumi
Terraform
Ansible

Nice to have:

Prometheus
Grafana
GPU

Nice to have experience with/in:

AI
cloud computing
data analytics
machine learning
B2B
AI software
platforms
SaaS
web development
software development

Required soft skills:

Delivery, impact, and autonomy

Note from a customer

Independent, deep expertise, able to ramp up and own niche technology.

Project description

Client's goal:

Launch MVP

We are seeking a Senior Engineer to design, automate, and operate our next-generation object storage platform built on Ceph, Rook, and Kubernetes. This platform provides S3-compatible storage for AI training, inference, and enterprise workloads at multi-petabyte scale.

You will work closely with infrastructure, networking, platform engineering, and site reliability teams to build a highly available, secure, and automated storage service supporting thousands of GPUs and exabytes of future growth.

Responsibilities:

  • Design, deploy, and operate large-scale Ceph clusters using Rook on Kubernetes.
  • Automate infrastructure provisioning, upgrades, and lifecycle management using Infrastructure-as-Code tools.
  • Build and maintain CI/CD pipelines for storage platform deployments.
  • Monitor cluster health, capacity, performance, and reliability using Prometheus, Grafana, and related observability tools.
  • Optimize Ceph performance, including OSD placement, erasure coding, CRUSH maps, networking, and recovery behavior.
  • Manage Ceph services including MON, MGR, RGW, OSD, CephFS (if applicable), and supporting infrastructure.
  • Implement disaster recovery, backup, replication, and operational runbooks.
  • Collaborate with networking teams on high-speed Ethernet or InfiniBand fabrics and RDMA-enabled storage architectures.
  • Develop automation for cluster expansion, hardware replacement, drive failures, and capacity management.
  • Participate in on-call rotation and incident response for production storage services.
  • Troubleshoot distributed systems issues across compute, networking, Kubernetes, and storage layers.

Required Qualifications

  • 5+ years of experience operating Linux production infrastructure.
  • 3+ years managing Kubernetes clusters in production.
  • Hands-on experience deploying and operating Ceph in production environments.
  • Built file-system code and run production clusters: you have committed code to Ceph itself, not only operated or managed it.
  • Experience with Rook Ceph Operator or similar Kubernetes storage operators.
  • Strong understanding of distributed systems, high availability, and fault tolerance.
  • Experience with Infrastructure-as-Code tools such as Terraform, Ansible, or Pulumi.
  • Experience building automation using Python, Go, or Bash.
  • Strong networking knowledge including TCP/IP, VLANs, BGP, LACP, MTU tuning, and high-speed Ethernet (100/200/400 GbE).
  • Experience with Prometheus, Grafana, Alertmanager, and centralized logging platforms.
  • Excellent troubleshooting and root cause analysis skills.

Preferred Qualifications

  • Experience operating storage platforms at petabyte scale.
  • Experience with AI/ML infrastructure or GPU clusters.
  • Familiarity with S3-compatible object storage and Ceph RGW.
  • Experience with erasure coding, CRUSH map tuning, and storage performance optimization.
  • Experience with RDMA, RoCE, or InfiniBand networking.
  • Experience with multi-site replication and disaster recovery.
  • Familiarity with security best practices including TLS, KMS integration, RBAC, and secrets management.
  • Experience operating cloud-native infrastructure across multiple data centers.

Nice to Have

  • Experience with Alluxio, MinIO, or other distributed storage systems.
  • Experience with GitOps platforms such as Argo CD or Flux.
  • Experience with NVIDIA GPU infrastructure and AI training workloads.
  • Knowledge of storage benchmarking tools (FIO, COSBench, S3Bench).
  • Red Hat Ceph Storage, Kubernetes, or CNCF certifications.

What Success Looks Like

  • Operate highly available Ceph clusters with 99.95%+ service availability.
  • Automate routine operational tasks to minimize manual intervention.
  • Continuously improve cluster performance, scalability, and operational efficiency.
  • Reduce deployment time through automation and standardized operational practices.
  • Enable reliable, scalable object storage supporting enterprise AI workloads at multi-petabyte scale.

About us: A research-driven artificial intelligence company. We believe open and transparent AI systems will drive innovation and create the best outcomes for society, and together we are on a mission to significantly lower the cost of modern AI systems by co-designing software, hardware, algorithms, and models. We have contributed to leading open-source research, models, and datasets to advance the frontier of AI, and our team has been behind technological advancement such as FlashAttention, Hyena, FlexGen, and RedPajama. We invite you to join a passionate group of researchers and engineers in our journey in building the next generation AI infrastructure

Project tech stack

Ceph, Rook, Kubernetes, Linux, Docker/Containerd, Terraform, Ansible, Python, Go, Bash, Prometheus, Grafana, Argo CD, GitHub Actions or Jenkins, S3 Object Storage, NVIDIA Networking (Ethernet or InfiniBand), Red Hat Enterprise, Linux or Ubuntu

Client selection process

  • hands-on Ceph/Rook technical round, about 1 hour: practical troubleshooting
  • experience and technical-depth interview, about 30 minutes: senior-level operational depth, architecture judgment, and breadth of Ceph experience

Team and communication

Slack, video calls, direct collaboration with principal engineers and team leads; one-stage technical interview with principal engineer and Amsterdam team member.

Team size

10+

Selection process

  1. Send your application and availability.
  2. Complete your profile and verify your DevOps role. Existing verification is reused.
  3. We review whether you’re a fit for this project and decide on the next step.
  4. If you’re selected, we send your profile to the client for review.

Your progress stays with you, even if this project closes.

Looks like a fit?

Hit apply and we’ll guide you through the next steps.

Already have an account?