RTX 3090 vs A100 for a 7B model on vLLM: the cheaper GPU depends on your answer time

Automated excerpt

Two rented cards: an RTX 3090 at $0.259/hour and an A100 80GB PCIe at $1.017/hour. 1 at once: 3090 $1.81, A100 $3.16 per 1M output tokens 4 at once: 3090 $0.49, A100 $0.80 16 at once: 3090 $0.16, A100 $0.24 64 at once: 3090 $0.08, A100 $0.11 At every load we ran on both, the 3090 cost less per token. A100 at 64 at once (p50 6.0 s, p95 7.1 s) costs $0.106. About 18 s: 3090 at 64 at once and A100 at 256 at once both cost $0.082.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.