A somewhat technical blog on how GPU choice shapes what you can actually learn about CUDA, distributed training, and training optimization and the realities of running a personal ML workstation.
Choosing the Right GPU Setup for AI Engineering (and Learning)Link to this section
This machine is not for winning benchmarks. It is a lab for learning CUDA, optimization and distributed systems through direct experience. That changes the selection criteria: peak FLOPs matter less than how many phenomena you can observe before hardware constraints distort the lesson. Under that lens, two RTX 5070 Ti cards beat a single flagship.
TL;DRLink to this section
- Want to learn CUDA + DDP/FSDP and observe real distributed behavior? Prefer two 16 GB GPUs (e.g., dual 5070 Ti). You trade single-GPU comfort for visibility into collectives, overlap and sharding.
- Want faster single-GPU iteration and fewer VRAM constraints? A 4090/5090 is the comfort path.
- Numbers below reflect public sources and documentation, not original hardware measurements.
Who is this forLink to this section
- Developers who want to study kernels, memory hierarchy, and distributed training in a controlled workstation.
- Practitioners optimizing for learning breadth per watt and per hour, not for leaderboard results.
Decision matrixLink to this section
| Goal | Constraints | Recommended setup | Why |
|---|---|---|---|
| Learn distributed training (DDP/FSDP), profile collectives | Reasonable cost, compact case | Dual 16 GB (5070 Ti or 5080 if price aligns) | Real process groups, measurable overlap and bucket behavior |
| Max single-GPU headroom | Need larger context/batch, minimal friction | 4090 or 5090 | More VRAM and throughput, simpler thermals |
| Some collectives + comfort | Prefer big single GPU but want real NCCL semantics | 4090 + helper CUDA card (e.g., A2000/3060) | Educational collectives with modest scaling; heterogeneous limits apply |
What learning requires from hardwareLink to this section
A workstation for learning must be capable of three modes. It must let you inspect kernels and memory hierarchy without constant resource starvation. It must expose real distributed behavior so you can reason about synchronization, sharding and communication overlap with evidence rather than intuition. It must also be versatile enough to serve as a daily machine; reliability, thermals and power delivery are part of the curriculum because instability invalidates results.
Sixteen gigabytes of VRAM on a modern NVIDIA GPU is sufficient for CUDA fundamentals, Nsight workflows, Triton and CUTLASS experiments, and meaningful training loops with medium models. Twenty-four gigabytes increases comfort substantially: larger batch sizes, longer contexts and fewer memory workarounds. Two GPUs do something different. They replace comfort with visibility. All-reduce timing, bucket sizing, overlap with backward passes, sharded checkpoints, elastic restarts and failure handling stop being theory and become measurements you can log and explain.
VRAM and model sizing (quick guide)Link to this section
- ~16 GB: comfortable for kernels, Nsight, Triton/CUTLASS and ~0.7-1.0B param models in BF16 with small batches (context length matters).
- ~24 GB: larger batches/contexts; fewer memory workarounds; ~1.0-1.3B practical without offload.
- 32 GB+: more headroom; still, distributed algorithms are a different axis than sheer VRAM.
Architectures, precision, and featuresLink to this section
RTX 5070 Ti and RTX 5080 are Blackwell-generation parts with fifth-generation Tensor Cores and support for FP4 and FP6 in addition to FP8/16/32/64. RTX 4090 is Ada with fourth-generation Tensor Cores, which means no FP4/6 but excellent FP8/16 throughput and a generous 24 GB of VRAM. RTX 5090 extends raw compute and VRAM further, but at a size and price that conflict with the goal of a compact, dual-GPU lab. Precision features are interesting when you want to study numerical stability and efficiency; VRAM is decisive when you want to observe memory-compute-communication trade-offs without artificial constraints.
Notes on precision and interconnect
- FP4/FP6 appear in Blackwell hardware, but kernel/optimizer support in frameworks is still maturing. Expect staggered enablement.
- Consumer RTX 40/50 series typically lack NVLink; multi-GPU collectives run on PCIe. P2P can vary by platform/BIOS. Use
nvidia-smi topo -mto inspect topology when you do have access to hardware.
Performance relationships that matter for learningLink to this section
Exact framerates are irrelevant here. Relative scaling is enough to structure experiments and price sanity checks.
Methods and assumptions (no local hardware)
- Source: consolidated public data (reviews, vendor docs, credible community benchmarks). No original measurements.
- Workload lens: BF16 transformer training throughput; kernel maturity (FlashAttention, fused optimizers), drivers and memory pressure shift numbers.
- Interpretation: treat values as guides for experiment design, not purchasing promises.
Across public sources, a practical working model is: 5080 is roughly 13-15% faster than 5070 Ti on training-relevant workloads, 4090 is about twice the 5070 Ti on BF16/FP32 math and roughly 50% ahead of 5080, and 5090 lifts another 50-70% over 5080 while adding more VRAM.
| GPU | VRAM | Tensor gen | Notable precision | Working relative ML throughput |
|---|---|---|---|---|
| RTX 5070 Ti | 16 GB | 5th (Blackwell) | FP4/FP6/FP8/FP16 | 1.00 × |
| RTX 5080 | 16 GB | 5th (Blackwell) | FP4/FP6/FP8/FP16 | ~1.13-1.15 × vs 5070 Ti |
| RTX 4090 | 24 GB | 4th (Ada) | FP8/FP16 (no FP4/6) | ~2.0 × vs 5070 Ti; ~1.5 × vs 5080 |
| RTX 5090 | 32 GB | Blackwell | FP4/FP6/FP8/FP16 | ~1.5-1.7 × vs 5080 |
Table: Relative throughput for training-oriented workloads. Variability is expected across frameworks and kernels; values reflect public internet sources as of 2025-10.
The 4090's additional 8 GB often matters more than its FLOPs. The ability to run 1-1.3B-parameter transformers or longer contexts without offload is a qualitative change in the kinds of questions you can ask. Two 16 GB cards do not raise per-rank capacity, but they let you study distributed algorithms on real hardware rather than by proxy.
Local distributed training is the first laboratory; the cloud is the secondLink to this section
Dual-GPU on a single node gives you DDP and FSDP in conditions you control. Communication happens over PCIe. Latency is low, bandwidth is limited and the step time decomposes cleanly into forward, backward, optimizer and communication slices you can profile with Nsight Systems and the PyTorch profiler. You will see the effect of gradient bucket sizes, parameter sharding, prefetch and reshard policies, and whether compute and communication overlap as intended.
This knowledge transfers. When you move to a cloud node with NVLink or a cluster with NVSwitch or EFA/RDMA, the code does not change. The transport does. Hierarchical collectives become more attractive, bandwidth improves by an order of magnitude and failure modes multiply. That is the correct time to measure scaling curves and sensitivity to topology. Without the local lab, the cloud only tells you that it is faster or slower; it does not tell you why.
Form factor, thermals, and why a 3-slot flagship complicates learningLink to this section
A single 4090 is simple if you never intend to run a second GPU. Its cooler is large, its transient power is high and it occupies the physical space a second card would need for airflow. You can simulate multi-process behavior with CUDA MPS and you can practice orchestration with Kubernetes time-slicing, but you cannot form a meaningful multi-GPU communicator without a second physical or MIG device. In contrast, most 5070 Ti boards are two-slot designs that leave room for a second card and adequate intake. If the goal is to study synchronization rather than merely accelerate a single stream, that mechanical fact matters.
Power delivery and stability considerationsLink to this section
Two 5070 Ti cards and a modern high-end CPU draw roughly 770-860 W sustained with 1.0-1.1 kW transient envelope. A 1000 W ATX 3.1 unit is the minimum that works; a 1200 W unit is the correct choice for quieter operation and margin. Each GPU should be on its own native 12V-2×6 lead. Stability is not a vanity metric. It is the precondition for reproducible measurements and for debugging logic rather than chasing electrical noise.
Minimal dual-GPU lab BOM (example)Link to this section
- CPU with adequate lanes (e.g., 7950X/14900K class)
- Motherboard with two x16 mechanical slots and good spacing
- 64-128 GB RAM
- 2 TB+ Gen4/5 NVMe SSD (useful for offload/FSDP checkpointing)
- ATX 3.1 PSU 1000-1200 W, two native 12V-2×6, separate leads per GPU
- Case with front intake; two 2-slot GPUs fit cleanly
BIOS checklist: Above 4G Decoding, Resizable BAR, enable PCIe P2P if available.
Price sanity through performance parity and elasticityLink to this section
Even when economics is not the driving factor, price should not violate basic proportionality. A useful parity check sets the 5070 Ti price as baseline and scales other targets by relative throughput. With a 5070 Ti reference near R$ 6,500, a strict proportional parity lands the 5080 around R$ 7,400 and the 5090 around R$ 11,300. Elasticity then discounts higher tiers to reflect diminishing educational returns and savings penalties, producing adjusted targets near R$ 7,000 for the 5080 and R$ 11,500 for the 5090, with buy-now and no-go bands around those anchors. These are not commandments but it is wise to try to avoid overpaying.
| GPU | Pure parity vs 5070 Ti | Elasticity-adjusted optimal | Buy-now band | No-go band |
|---|---|---|---|---|
| RTX 5070 Ti | R$ 6,500 | R$ 6,500 | ≤ R$ 5,850 | ≥ R$ 7,150 |
| RTX 5080 | ~R$ 7,400 | ~R$ 7,000 | ≤ ~R$ 6,300 | ≥ ~R$ 7,700 |
| RTX 5090 | ~R$ 11,300 | ~R$ 11,500 | ≤ ~R$ 10,350 | ≥ ~R$ 12,650 |
Table: Pricing sanity relative to a 5070 Ti baseline. Brazil retail street prices as of 2025-10; adjust to your market.
Ceilings serve a different role. They prevent weaker products from drifting into stronger price regions when a superior variant exists. Capping the entire 5080 family at R$ 8,500 and the 5090 at R$ 12,000 preserves the ordering of value even as street prices move.
The asymmetric helper card compromiseLink to this section
If a large 24 GB card is attractive for comfort but you still want genuine NCCL semantics locally, an asymmetrical arrangement is viable. A 4090 can be paired with a short, cool, low-power CUDA card such as an RTX A2000 or a compact 3060/4060. The smaller device becomes the bottleneck and scaling will be modest, but process-group creation, all-reduce behavior, sharded state and failure handling will all be real. When experiments depend on interconnect behavior or tensor parallelism, a cloud session is still required.
Kubernetes, time-slicing, and why sharding a 4090 is not MIGLink to this section
Consumer cards do not expose MIG. Kubernetes with the NVIDIA device plugin can time-slice a single GPU across pods, but this does not create multiple CUDA devices with isolated VRAM. It is useful for rehearsal of manifests, logging and artifact management. It is not a substitute for multi-GPU collectives. For actual distributed algorithms, you need at least two physical GPUs or a datacenter GPU with MIG.
The conclusion and its logicLink to this section
Two RTX 5070 Ti cards are the better laboratory. They maximize learning breadth per watt and per hour. They let you study CUDA deeply and then climb the stack into distributed training without leaving the workstation. They preserve enough mechanical space and electrical headroom to remain reliable. They carry Blackwell precision features for experiments in low-bit training. They trade some single-GPU comfort for access to the part of the system that is hardest to learn from a single device: how multiple processes coordinate work they cannot see. When the time comes to study fabrics and multi-node scaling, the same code runs on a cloud node with different transport and the differences are measurable.
A single large GPU is still a legitimate choice for a different goal: fast iteration and fewer memory constraints. It is simply a different class of instrument. In a laboratory built to understand how the pieces fit rather than merely to accelerate them, dual mid-range GPUs are the more instructive tool.