December 2025 Update: OpenAI using Ray to coordinate ChatGPT training. Ray joining PyTorch Foundation validating enterprise adoption. Framework scaling from laptop to thousands of GPUs. Achieving millions of tasks per second with sub-millisecond latency—order of magnitude faster than Spark for AI patterns. Native heterogeneous compute supporting CPU/GPU workload mixing.
OpenAI uses Ray to coordinate the training of ChatGPT and other models.¹ The framework scales from a laptop to clusters of thousands of GPUs, handling the distributed computing complexity that would otherwise require custom infrastructure for every project. Ray's adoption exploded through 2025, validated by the framework joining the PyTorch Foundation and Anyscale raising capital to support enterprise deployments.² Understanding Ray's architecture and deployment patterns helps organizations build distributed AI infrastructure that scales from experimentation to production.
Ray provides a unified framework for distributed AI workloads—training, tuning, inference, and data processing—that abstracts the complexity of cluster management while maintaining fine-grained control over resource allocation. For organizations moving beyond single-GPU experimentation into multi-node production systems, Ray offers the most direct path to scalable AI infrastructure.
Why Ray for AI infrastructure
Ray emerged from UC Berkeley's RISELab to address the specific challenges of distributed AI workloads that traditional frameworks like Apache Spark handle poorly:
AI workload requirements
Heterogeneous compute: AI pipelines mix CPU-intensive data processing with GPU-accelerated training and inference. Ray natively supports heterogeneous resource allocation, scheduling tasks across CPUs and GPUs as workloads demand.³
Fine-grained parallelism: Deep learning training requires coordinating gradients across workers, managing model state, and handling failures gracefully. Ray's task and actor abstractions provide the primitives needed for these patterns.
Stateful computation: Unlike MapReduce-style frameworks, AI workloads often maintain state across iterations. Ray actors persist state between invocations, supporting patterns like parameter servers and reinforcement learning agents.
Low latency: Real-time inference and hyperparameter search require microsecond task scheduling. Ray achieves millions of tasks per second with sub-millisecond latency—an order of magnitude faster than Spark for these patterns.⁴
Comparison with alternatives
Apache Spark: Optimized for batch data processing with SQL and DataFrames. Excels at ETL, feature engineering, and analytics on structured data. Less suited for GPU workloads, stateful computation, and low-latency requirements.⁵
Dask: Python-native distributed computing with DataFrame and array APIs. Lighter weight than Spark but lacks Ray's actor model and ML-specific libraries.
Horovod: Focused specifically on distributed deep learning training. Less flexible than Ray for diverse AI workloads but simpler for pure training scenarios.
Ray's advantage: A single framework handles data processing, training, hyperparameter tuning, and inference serving. Teams avoid managing multiple systems and the integration complexity between them.
Production adoption
Major organizations run Ray in production:⁶
- OpenAI: ChatGPT training coordination
- Uber: Distributed ML platform
- Instacart: ML infrastructure backbone
- Shopify: Production ML workloads
- Ant Group: Large-scale financial ML
Ray clusters officially support up to 2,000 nodes, enabling the scale required for frontier model training and high-volume inference.
Ray architecture
Core abstractions
Ray provides two fundamental abstractions for distributed computation:
Tasks: Stateless functions executed remotely. Ray automatically schedules tasks across available workers, handles failures, and returns results.
import ray
ray.init()
@ray.remote
def process_batch(data):
# Runs on any available worker
return transform(data)
# Execute in parallel
futures = [process_batch.remote(batch) for batch in batches]
results = ray.get(futures)
Actors: Stateful objects distributed across the cluster. Each actor maintains state between method calls, enabling patterns like model serving, parameter servers, and game environments.
@ray.remote
class ModelServer:
def __init__(self, model_path):
self.model = load_model(model_path)
def predict(self, input_data):
return self.model(input_data)
# Create actor instance
server = ModelServer.remote("model.pt")
# Call methods remotely
prediction = ray.get(server.predict.remote(data))
Cluster architecture
Ray clusters consist of a head node and worker nodes:
Head node: Runs the Global Control Store (GCS) managing cluster state, object directory tracking data locations, and scheduler coordinating task placement.
Worker nodes: Execute tasks and actors. Each worker runs a Raylet daemon handling local scheduling and object management.
Object store: Distributed shared memory enabling zero-copy data passing between tasks on the same node and efficient serialization across nodes.
Resource management
Ray manages heterogeneous resources explicitly:
@ray.remote(num_cpus=4, num_gpus=1)
def train_step(model, data):
# Guaranteed 4 CPUs and 1 GPU
return model.train(data)
Custom resources enable fine-grained scheduling:
# Request specific hardware
@ray.remote(resources={"TPU": 1})
def tpu_inference(data):
return run_on_tpu(data)
Ray AI libraries
Ray includes purpose-built libraries for common AI workloads:
Ray Train
Distributed training abstraction supporting PyTorch, TensorFlow, and other frameworks:
from ray.train.torch import TorchTrainer
from ray.train import ScalingConfig
def train_func():
# Standard PyTorch training code
model = MyModel()
for epoch in range(10):
train_epoch(model)
trainer = TorchTrainer(
train_loop_per_worker=train_func,
scaling_config=ScalingConfig(
num_workers=8,
use_gpu=True
),
)
result = trainer.fit()
Key capabilities:⁷ - Scale from single GPU to multi-node clusters with 2 lines of code change - Automatic distributed data loading and gradient synchronization - Checkpoint management and fault recovery - Integration with PyTorch Lightning, Hugging Face Transformers, and DeepSpeed
Ray Tune
Distributed hyperparameter optimization:
from ray import tune
from ray.tune.schedulers import ASHAScheduler
def training_function(config):
model = build_model(config["learning_rate"], config["batch_size"])
for epoch in range(100):
loss = train_epoch(model)
tune.report(loss=loss)
analysis = tune.run(
training_function,
config={
"learning_rate": tune.loguniform(1e-4, 1e-1),
"batch_size": tune.choice([32, 64, 128])
},
scheduler=ASHAScheduler(metric="loss", mode="min"),
num_samples=100,
)
Ray Tune provides: - Parallel trial execution across cluster - Early stopping with ASHA, Population-Based Training, and other schedulers - Integration with Optuna, HyperOpt, and other optimization libraries - Checkpointing and resumption
Ray Serve
Production model serving with autoscaling:
from ray import serve
@serve.deployment(num_replicas=2, ray_actor_options={"num_gpus": 1})
class LLMDeployment:
def __init__(self):
self.model = load_llm()
async def __call__(self, request):
prompt = await request.json()
return self.model.generate(prompt["text"])
serve.run(LLMDeployment.bind())
Ray Serve capabilities:⁸ - Autoscaling based on request rate - Zero-downtime deployments - Multi-model composition and request routing - OpenAI-compatible API for LLM serving - 60% reduction in time-to-first-token with prefix-aware routing⁹
Ray Data
Distributed data loading and preprocessing:
import ray
ds = ray.data.read_parquet("s3://bucket/data/")
ds = ds.map(preprocess)
ds = ds.filter(lambda x: x["label"] > 0)
# Convert to PyTorch DataLoader
train_loader = ds.iter_torch_batches(batch_size=32)
Ray Data provides: - Streaming execution for large datasets - GPU-accelerated preprocessing - Integration with Ray Train for distributed training - Native support for image, text, and tabular data
Kubernetes deployment with KubeRay
Production Ray deployments typically run on Kubernetes using KubeRay, the official Kubernetes operator:¹⁰
KubeRay components
RayCluster CRD: Defines cluster configuration including head and worker node specifications, autoscaling policies, and resource requirements.
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: ml-cluster
spec:
headGroupSpec:
rayStartParams:
dashboard-host: '0.0.0.0'
template:
spec:
containers:
- name: ray-head
image: rayproject/ray:2.52.0-py310-gpu
resources:
limits:
nvidia.com/gpu: 1
workerGroupSpecs:
- replicas: 4
minReplicas: 2
maxReplicas: 10
groupName: gpu-workers
rayStartParams: {}
template:
spec:
containers:
- name: ray-worker
image: rayproject/ray:2.52.0-py310-gpu
resources:
limits:
nvidia.com/gpu: 1
RayJob CRD: Submit jobs to automatically provisioned clusters. KubeRay creates the cluster, runs the job, and optionally deletes the cluster on completion.
RayService CRD: Managed Ray Serve deployments with zero-downtime upgrades and health checking.
Production best practices
Container images: Bake dependencies into published Docker images rather than installing at runtime. This ensures reproducibility and faster startup.¹¹
Autoscaling: Enable both Ray autoscaling (scaling workers within a cluster) and Kubernetes autoscaling (scaling cluster nodes):
spec:
enableInTreeAutoscaling: true
autoscalerOptions:
upscalingMode: Default
idleTimeoutSeconds: 60
Storage: Use persistent volumes for checkpoints and shared storage for large datasets:
volumes:
- name: shared-storage
persistentVolumeClaim:
claimName: ml-data-pvc
Monitoring: Integrate with Prometheus and Grafana for observability:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
Cloud-specific deployments
GKE: Ray on GKE add-on provides managed Ray clusters with automatic provisioning and integration with Google Cloud services.¹²
EKS: Deploy KubeRay with Cluster Autoscaler or Karpenter for node scaling. Integration with FSx for Lustre provides high-performance shared storage.
AKS: Microsoft and Anyscale offer Anyscale on Azure as a first-party service accessible from Azure Portal.¹³
Anyscale managed Ray
Anyscale, the company founded by Ray's creators, offers managed Ray deployments:
Anyscale Platform
Managed clusters: Production-grade Ray clusters with automatic scaling, fault tolerance, and monitoring without infrastructure management.
RayTurbo runtime: Proprietary performance improvements delivering higher resilience, faster performance, and lower cost versus open-source Ray.¹⁴
Enterprise features: Role-based access control, audit logging, VPC peering, and compliance certifications.
Cloud partnerships
CoreWeave: Anyscale BYOC (Bring Your Own Cloud) deploys directly in CoreWeave customer accounts via CoreWeave Kubernetes Service.¹⁵
Azure: First-party Anyscale service available in Azure Portal with native integration.
AWS: Anyscale operates on AWS with integration into existing AWS services.
When to use Anyscale
Consider Anyscale for: - Teams without dedicated platform engineering capacity - Production deployments requiring enterprise SLAs - Organizations wanting RayTurbo performance improvements - Quick time-to-production without infrastructure expertise
Consider self-managed KubeRay for: - Existing Kubernetes expertise and platform team - Cost optimization at scale - Custom integration requirements - Multi-cloud or hybrid deployments
Deployment patterns
Training pipelines
Typical distributed training architecture:
Data Lake → Ray Data (preprocessing) → Ray Train (distributed training)
↓
Model Registry
Data preprocessing: Ray Data handles distributed loading, transformation, and augmentation with streaming execution for datasets exceeding memory.
Distributed training: Ray Train coordinates gradient synchronization across workers with automatic fault recovery.
Checkpoint management: Regular checkpoints to cloud storage enable resumption after failures.
Inference serving
Production LLM serving with Ray Serve:
Load Balancer → Ray Serve (autoscaling) → vLLM (inference engine)
↓
Prefix Cache Routing
Ray Serve LLM provides:¹⁶ - First-class vLLM integration - OpenAI-compatible API endpoints - Hybrid reasoning model support (toggle thinking on/off) - Custom request routing for prefix cache locality
Compound AI applications
Multi-step AI pipelines combining multiple models:
@serve.deployment
class RAGPipeline:
def __init__(self):
self.embedder = EmbeddingModel()
self.retriever = VectorDB()
self.generator = LLM()
async def __call__(self, query):
embedding = self.embedder.encode(query)
context = self.retriever.search(embedding)
return self.generator.generate(query, context)
Ray Serve handles orchestration, allowing each component to scale independently based on its resource requirements and load.
Performance optimization
Memory management
Object store tuning: Configure object store memory based on workload:
ray.init(object_store_memory=10 * 1024**3) # 10GB
Spilling configuration: Enable disk spilling for workloads exceeding memory:
ray.init(_system_config={"object_spilling_config": json.dumps({
"type": "filesystem",
"params": {"directory_path": "/tmp/spill"}
})})
GPU utilization
Fractional GPUs: Share GPUs across multiple tasks:
@ray.remote(num_gpus=0.5)
def inference(data):
# Uses half a GPU
return model(data)
Multi-GPU training: Configure data parallelism across GPUs:
ScalingConfig(
num_workers=8,
use_gpu=True,
resources_per_worker={"GPU": 1}
)
Network optimization
Placement groups: Colocate related tasks to minimize network overhead:
from ray.util.placement_group import placement_group
pg = placement_group([{"GPU": 1}] * 8, strategy="PACK")
ray.get(pg.ready())
Object locality: Ray's scheduler considers data location when placing tasks, minimizing cross-node transfers.
Organizations building distributed AI infrastructure can leverage Introl's global deployment expertise for Ray cluster planning and GPU procurement across 257 locations worldwide.
The distributed computing foundation
Ray provides the distributed computing foundation that AI workloads demand—heterogeneous resource management, stateful computation, and sub-millisecond task scheduling. The framework's adoption by OpenAI, major enterprises, and its acceptance into the PyTorch Foundation validates its position as the standard for distributed AI infrastructure.
For organizations scaling AI beyond single-node experiments, Ray offers the most direct path to production. The framework handles distributed complexity while maintaining flexibility for diverse workloads. Whether training models across thousands of GPUs, serving inference at scale, or building compound AI applications, Ray provides unified infrastructure that simplifies operations and accelerates development.
The choice between self-managed KubeRay and Anyscale's managed service depends on organizational capacity. Teams with platform engineering expertise can optimize costs through self-management; those prioritizing time-to-production benefit from Anyscale's operational support. Either path delivers production-grade distributed AI infrastructure built on the framework that powers the industry's most demanding workloads.
References
-
Anyscale. "How Ray Solves Generative AI & LLM Infrastructure Challenges." 2025. https://www.anyscale.com/blog/ray-common-production-challenges-for-generative-ai-infrastructure
-
Anyscale. "Ray is Joining The PyTorch Foundation." 2025. https://www.anyscale.com/blog/ray-by-anyscale-joins-pytorch-foundation
-
Ray Documentation. "Getting Started." 2025. https://docs.ray.io/en/latest/ray-overview/getting-started.html
-
Medium. "Ray vs Spark — The Future of Distributed Computing." By Philippe Dagher. 2024. https://medium.com/@nasdag/ray-vs-spark-the-future-of-distributed-computing-b10b9caa5b82
-
Onehouse. "Ray vs Dask vs Apache Spark — Comparing Data Science & Machine Learning Engines." 2025. https://www.onehouse.ai/blog/apache-spark-vs-ray-vs-dask-comparing-data-science-machine-learning-engines
-
Ray. "Scale Machine Learning & AI Computing." 2025. https://www.ray.io/
-
Ray Documentation. "Ray Train: Scalable Model Training." 2025. https://docs.ray.io/en/latest/train/train.html
-
Ray Documentation. "Ray Serve: Scalable and Programmable Serving." 2025. https://docs.ray.io/en/latest/serve/index.html
-
Anyscale. "Ray Serve: Reduce LLM Inference Latency by 60% with Custom Request Routing." 2025. https://www.anyscale.com/blog/ray-serve-faster-first-token-custom-routing
-
GitHub. "ray-project/kuberay: A toolkit to run Ray applications on Kubernetes." 2025. https://github.com/ray-project/kuberay
-
Ray Documentation. "Best Practices for Storage and Dependencies." 2025. https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/storage.html
-
Google Cloud Blog. "Use Ray on Kubernetes with kuberay." 2025. https://cloud.google.com/blog/products/containers-kubernetes/use-ray-on-kubernetes-with-kuberay
-
Microsoft DevBlogs. "Powering Distributed AI/ML at Scale with Azure and Anyscale." 2025. https://devblogs.microsoft.com/all-things-azure/powering-distributed-aiml-at-scale-with-azure-and-anyscale/
-
Anyscale. "Ray Summit 2025: Anyscale Platform Updates." 2025. https://www.anyscale.com/blog/ray-summit-2025-anyscale-product-updates
-
CoreWeave. "Announcing distributed AI on CoreWeave with fully managed Ray on Anyscale." 2025. https://www.coreweave.com/blog/distributed-ai-on-coreweave-with-ray-anyscale
-
Ray Documentation. "Serving LLMs." 2025. https://docs.ray.io/en/latest/serve/llm/index.html
Key takeaways
For strategic planners: - OpenAI uses Ray for ChatGPT training coordination; Ray joined PyTorch Foundation validating industry standard status - Production adopters: OpenAI, Uber, Instacart, Shopify, Ant Group; clusters support up to 2,000 nodes officially - Framework handles training, tuning, inference, and data processing under single abstraction—eliminates multi-system integration
For ML engineers: - Ray achieves millions of tasks per second with sub-millisecond latency; order of magnitude faster than Spark for AI workloads - Ray Train: 2-line code change scales single GPU to multi-node; integrates PyTorch Lightning, Hugging Face, DeepSpeed - Ray Serve LLM: 60% time-to-first-token reduction with prefix-aware routing; OpenAI-compatible API; vLLM integration
For platform teams: - KubeRay: RayCluster CRD for cluster config, RayJob CRD for auto-provisioned jobs, RayService CRD for zero-downtime deployments - Cloud options: GKE Ray add-on (managed), EKS with Karpenter, AKS with first-party Anyscale service via Azure Portal - Anyscale BYOC deploys directly in CoreWeave customer accounts via CKS; RayTurbo runtime delivers proprietary performance gains
For operations teams: - Fractional GPU support enables multiple tasks per GPU; placement groups colocate related tasks minimizing network overhead - Object store memory configurable; disk spilling for workloads exceeding memory; automatic fault recovery and checkpointing - Container images should bake dependencies rather than install at runtime; integrate Prometheus/Grafana for observability
