GPU Sharing in Kubernetes

GPU costs dominate AI infrastructure budgets. Teams provision expensive GPU nodes, allocate entire devices to single pods, and watch utilization hover around 30-50% while paying for 100%. Kubernetes treats each GPU as an indivisible resource by default — meaning if your inference workload needs 20% of an H100, you still reserve the full device and block other workloads from using the remaining 80%.

GPU sharing addresses this waste directly. By enabling multiple pods to use fractional GPU capacity, teams increase utilization, reduce idle time, and cut infrastructure costs by 50% or more without sacrificing performance. But GPU sharing in Kubernetes is not a single switch — it involves choosing between time-slicing, MIG (Multi-Instance GPU), and MPS (Multi-Process Service), each with distinct trade-offs in isolation, flexibility, and operational complexity.

This guide walks through why GPU sharing in Kubernetes clusters matters, how each sharing method works, when to use which approach, and why static sharing alone still leaves money on the table.

1. Why GPU Sharing Matters in Kubernetes

GPU underutilization and rigid allocation drive two costly problems in Kubernetes clusters running AI workloads.

Kubernetes allocates GPUs whole — the nvidia.com/gpu: 1 problem

By default, Kubernetes schedulers assign entire GPUs to individual pods. When a pod requests nvidia.com/gpu: 1, the scheduler reserves one physical GPU for that pod's exclusive use. Even if the pod's workload uses only 15% of GPU compute and 30% of GPU memory, the remaining capacity sits idle — unavailable to other workloads.

According to ScaleOps' 2026 analysis, this creates a GPU waste tax: "ten services that each use 15% of a GPU still need ten physical GPUs with full-GPU allocation." The real demand is closer to 1.5 GPUs, but Kubernetes reserves ten devices unless sharing is enabled.

This rigidity forces teams into two bad choices:

• Overprovision: Request more GPU nodes than needed because workloads cannot pack efficiently
• Underutilize: Pay for full GPU capacity while using a fraction of available compute and memory

The cost of idle GPUs (utilization vs. what you're billed for)

Even when GPUs are "in use," actual utilization often tells a different story. vCluster reports that GPU utilization averages just 30-50% in production Kubernetes environments. Lightweight inference models, dev/test notebooks, and intermittent training jobs all contribute to this pattern.

A T4 GPU on AWS costs $0.526/hour. If your model serving API uses 25% of that GPU's capacity on average, you're paying $0.526/hour for $0.13/hour worth of actual compute. Scale that across dozens of GPU nodes, and the waste compounds quickly.

Kubernetes GPU utilization becomes the critical metric — not whether a GPU is "allocated," but whether allocated capacity is actually being used. Without sharing, utilization remains low, costs remain high, and teams struggle to justify AI infrastructure spend.

2. Time-Slicing: The Easiest Way to Start

GPU time-slicing is Kubernetes' simplest GPU sharing method. It allows multiple pods to share a single GPU by time-multiplexing — Pod A runs for a few milliseconds, the scheduler switches to Pod B, then back to Pod A. From each pod's perspective, it appears to have dedicated GPU access.

How it works and how to enable it (device plugin replicas, nvidia.com/gpu.shared)

Time-slicing requires editing the NVIDIA device plugin ConfigMap to define the number of "replicas" (virtual GPU slices) per physical GPU. For example, setting replicas: 4 on a node with one GPU makes Kubernetes see four logical GPUs available for scheduling.

Each pod requests nvidia.com/gpu: 1, but the scheduler can now place four pods on the same physical device. The NVIDIA driver handles context switching between processes.

Where it fits — notebooks, dev/test, light inference

Time-slicing works best for workloads with:

• Intermittent GPU usage: Jupyter notebooks, exploratory data analysis, dev/test environments
• Low memory footprint: Inference workloads that allocate small amounts of GPU memory and don't saturate compute
• Tolerant to variability: Users who can accept occasional slowdowns when multiple pods actively use the GPU simultaneously

Real-world example: A team of eight data scientists each running notebooks on dedicated T4 GPUs costs $4.21/hour ($0.526 × 8). If the notebooks have intermittent usage and fit within the available GPU memory, time-slicing with four replicas per GPU could allow the team to consolidate onto two T4 nodes at $1.05/hour — a potential 75% infrastructure cost reduction.

Trade-offs: no memory or fault isolation

Time-slicing provides no hardware isolation. Key limitations:

• No memory isolation: If Pod A allocates 10 GB of GPU memory, it remains allocated even when Pod A is idle. Pod B cannot reclaim that memory.
• No compute isolation: A runaway process in Pod A can starve Pod B of compute cycles. There is no guaranteed minimum compute allocation per pod.
• Shared fault domain: A GPU error (e.g., CUDA out-of-memory) can crash all pods sharing that device.

These trade-offs are acceptable for dev/test and non-critical inference. They are not acceptable for production multi-tenant environments or regulated workloads where performance isolation is mandatory.

3. MIG: Hardware-Isolated GPU Partitions

MIG (Multi-Instance GPU) is NVIDIA's hardware-level partitioning technology available on A100, H100, and newer datacenter GPUs. MIG can divide a GPU into multiple isolated instances—up to seven on A100 and H100 GPUs (the maximum varies across newer supported models).

Profiles, supported GPUs and the 7-instance limit

MIG instances come in fixed profiles defined by GPU memory and compute partitions:

Hardware support: MIG is only available on A100 (80GB and 40GB variants), H100, and select newer GPUs. Older V100s and T4s do not support MIG.

7-instance limit: An A100 can be partitioned into a maximum of seven smallest-profile instances—for example, seven 1g.5gb instances on an A100 40 GB.

MIG configuration requires the NVIDIA MIG Manager to create, modify, and delete MIG instances on GPU nodes. Changes require draining the node, applying the new profile, and restarting the device plugin.

Where it fits — production multi-tenant and regulated workloads

MIG delivers hardware-level isolation, making it ideal for:

• Multi-tenant SaaS platforms: Customer A's inference workload cannot interfere with Customer B's workload. Each tenant gets a dedicated MIG instance.
• Production inference serving: Predictable performance and quality of service. No unexpected slowdowns from noisy neighbors.
• Regulated environments: Compliance or security requirements mandate workload isolation. MIG provides a hardware boundary between workloads.
• Stable memory requirements: Workloads that fit well into MIG's fixed profiles (e.g., multiple 10 GB models on 2g.10gb instances).

When NOT to use MIG:

• Workloads change size frequently (MIG profiles are static until reconfigured)
• Running on V100 or T4 GPUs (no MIG support)
• Memory requirements fall between MIG profile sizes (e.g., 12 GB model forced into 20 GB instance wastes 8 GB)

Trade-offs: static profiles, reconfiguration is an operational event

MIG's rigidity is both a strength and a limitation:

• Static profiles: Once a GPU is partitioned into MIG instances, changing profiles requires draining the node, stopping workloads, reconfiguring MIG, and restarting the device plugin. This is a manual operational event, not a dynamic reallocation.
• Wasted capacity: If your workload needs 12 GB but the smallest available MIG instance is 20 GB, you waste 8 GB per instance.
• Limited flexibility: Cannot dynamically adjust MIG profiles based on workload demand without node disruption.

MIG trades flexibility for isolation. Teams willing to plan capacity around fixed profiles gain predictable performance. Teams needing dynamic allocation should consider time-slicing or MPS.

4. MPS: Concurrency for Trusted CUDA Workloads

MPS (Multi-Process Service) is NVIDIA's CUDA-level sharing mechanism. MPS allows multiple CUDA processes to run concurrently on a single GPU with better context-switching efficiency than time-slicing.

How MPS differs from time-slicing

Time-slicing switches between processes at the OS level. Each context switch incurs overhead as the GPU saves and restores process state.

MPS operates at the CUDA level. The MPS server manages a shared CUDA context, allowing multiple processes to submit work to the GPU simultaneously. This reduces context-switching overhead and improves throughput for compute-bound workloads.

Key difference: MPS provides concurrency (multiple processes can use the GPU at the same time), while time-slicing provides serialization (processes take turns).

Trade-offs: shared fault domain, experimental device plugin support

MPS has significant limitations in Kubernetes:

• Shared fault domain: MPS clients share GPU scheduling resources. If one process crashes or triggers a fatal GPU error, it can affect other workloads sharing that GPU.
• No memory isolation: Like time-slicing, MPS does not provide memory boundaries between processes. A memory leak in one process potentially affects all others.
• Limited Kubernetes support: MPS device plugin support is experimental and not widely deployed in production Kubernetes clusters. Most teams default to time-slicing or MIG.
• Trusted workloads only: MPS is suitable only for environments where all workloads are trusted and unlikely to crash or misbehave.

MPS is a niche solution. It fits training jobs from a single team where all workloads are known and trusted, but it is not recommended for multi-tenant production inference or untrusted workloads.

5. Choosing the Right Method

Different workloads tolerate different levels of isolation and sharing flexibility. Matching the right method to each workload maximizes cost savings without compromising performance.

Comparison table: isolation, memory, fault domain, hardware requirement, best-fit workload

Combining MIG with time-slicing

Mig and time slicing are not mutually exclusive. Teams can partition a GPU with MIG, then enable time-slicing within each MIG instance.

Example: Partition an A100 into two 3g.20gb MIG instances. Enable time-slicing with 2 replicas per MIG instance. Kubernetes now sees four logical GPUs (two MIG instances × 2 time-slicing replicas each).

This hybrid approach provides:

• Hardware isolation between tenants (via MIG)
• Higher utilization within each tenant (via time-slicing)

The trade-off: added configuration complexity. Most teams start with time-slicing alone, then layer MIG on top once multi-tenancy or isolation requirements justify the operational overhead.

Where DRA is taking GPU allocation next

DRA (Dynamic Resource Allocation) is Kubernetes' next-generation resource scheduling framework (graduated to GA in Kubernetes 1.34). DRA allows custom schedulers and device plugins to implement fine-grained GPU allocation beyond the default "one GPU per pod" model.

DRA enables:

• Attribute-based device selection: Workloads can request devices based on characteristics exposed by the driver, such as GPU model, memory capacity, compute capability, or other vendor-specific attributes.
• ResourceClaims: Instead of relying only on extended-resource requests such as nvidia.com/gpu: 1, workloads can request devices through ResourceClaim and ResourceClaimTemplate objects.
• Driver-defined sharing: DRA can support shared or more granular device allocation when the underlying GPU driver implements those capabilities.
• More flexible scheduling: Kubernetes can match workloads with suitable devices based on requested capabilities rather than treating every GPU as an interchangeable integer resource.

DRA is the future of GPU scheduling in Kubernetes, but adoption is still early. Most production clusters today rely on time-slicing, MIG, MPS, or vendor-specific GPU sharing implementations.

6. Why Static Sharing Still Leaves Money on the Table

Enabling time-slicing or MIG is a one-time improvement. But static configurations cannot adapt to changing workloads, leading to persistent waste.

Idle slices and oversized MIG profiles

Even with sharing enabled, utilization gaps remain:

• Idle time-slicing replicas: If you configure 4 replicas per GPU but only 2 pods are running, the remaining 2 slices sit idle. You're paying for the full GPU but using half.
• Oversized MIG instances: A model needs 12 GB of memory but the smallest available MIG profile is 20 GB. You waste 8 GB per instance.

ScaleOps' 2026 report notes that static MIG profiles often result in 20-30% wasted capacity because workload sizes rarely align perfectly with fixed profile boundaries.

GPU memory reservations blocking co-location (vLLM and KV cache)

Inference frameworks like vLLM allocate large KV caches up front to maximize throughput. Even if the model itself uses 10 GB, vLLM may reserve 30 GB of GPU memory at startup for future requests.

This memory reservation blocks other workloads from co-locating on the same GPU, even if actual memory usage is low. Time-slicing and MIG both suffer from this issue — neither can reclaim memory reserved but not actively used.

Result: GPUs appear "full" but actual utilization remains below 50%. Teams pay for capacity they cannot access.

Manual retuning doesn't keep up with changing workloads

Workloads change over time:

• Training jobs finish and new ones start
• Inference traffic spikes during business hours and drops overnight
• Model sizes grow as teams fine-tune larger foundation models

Manual reconfiguration of MIG profiles or time-slicing replicas cannot keep pace. By the time a team reconfigures, workload patterns have already shifted. Static configurations lock teams into yesterday's usage patterns while paying today's rates. Manual reconfiguration can't keep pace with workload changes. This is where automation becomes essential — platforms like CAST AI or Kubecost can monitor utilization in real time and rightsize or consolidate workloads automatically, without waiting for a team to notice the waste.

This is where automation becomes essential.

7. Attribute and Right-Size GPU Spend

Cost optimization without visibility is guesswork. Accurate GPU cost attribution enables teams to identify which models, namespaces, and teams drive the most spend.

Tagging and cost allocation by team, model and namespace

Kubernetes labels and cloud provider tags enable cost breakdowns by dimension.

Recommended labels:

• team: Data Science, ML Research, Engineering
• model: bert-base, llama-70b, stable-diffusion-v3
• environment: dev, staging, production
• workload: training, inference, notebook

Implementation: Add labels to pod specs. Kubernetes cost allocation tools (Kubecost, OpenCost) and cloud billing systems (AWS Cost Explorer, GCP BigQuery) aggregate spend by label.

Example query: "Show me GPU spend by team for the last 30 days." Reveals which teams consume the most GPU hours and enables chargeback or showback.

Unit economics for AI (cost per inference, cost per training run)

Aggregated monthly GPU spend tells you the total bill. Unit economics tells you whether each workload is cost-effective.

Key metrics:

• Cost per training run: Total GPU hours × hourly rate for one model training job. Tracks training efficiency over time.
• Cost per 1K inferences: GPU cost allocated to inference pods ÷ number of API requests served. Identifies expensive models for optimization.
• GPU utilization per namespace: Actual GPU utilization (from DCGM) vs. requested GPU allocation. Highlights overprovisioned workloads.

Example: An inference service reserves an entire GPU but averages 30% utilization. If three compatible inference workloads can safely share the device through time-slicing, the team may be able to consolidate three GPU nodes onto one. If that consolidation allows two nodes to be removed, the underlying infrastructure cost could fall by roughly 66%.

How nOps Optimizes GPU Costs

nOps was designed to help teams put the strategies described in this article into practice. Our mission is to make AI optimization easy for engineers, so they can focus on building and innovating.

nOps Commitment Management handles the compute and infrastructure costs underneath your AI workloads—including the GPU instances, EKS clusters, and supporting services powering model training and inference. We continuously adjust Savings Plans and Reserved Instances in small increments based on your actual usage. That typically translates to 40–60% savings with zero manual effort.

Our AI Cost & Attribution solution ties GPU spend and supporting infrastructure costs together, giving you visibility into which teams, products, and features are driving AI spend — and where to optimize AI costs.

Curious how optimized you are on AI? A 30-minute free savings analysis shows you your current Effective Savings Rate and where the opportunities are. Setup is 5 minutes with no agents or infra changes needed.

nOps manages $4 billion in cloud spend for its customers and is rated 5 stars on G2.

FAQ