How do you calculate how much VRAM an LLM needs?
Add weights (params x bytes) to the KV cache (2 x layers x KV heads x head size x bytes x tokens), then divide by 0.92. Worked examples and GPU sizes.
llm
10 posts with the tag “llm”
How does knowledge distillation work for LLMs?
How LLM distillation works (soft labels, teacher outputs, on-policy, hidden states), what each method costs, and a rule for picking one.
What is vLLM, and why is it faster than plain Transformers?
vLLM is an open-source LLM serving engine. PagedAttention and continuous batching let it fit far more requests per GPU than a plain generate() loop.
How do you train an AI model on your own data in the cloud?
Shape your data into prompt and answer pairs, hold out a test set, upload it, then fine-tune an open model with a cost cap. Steps, code and costs.
Can an AI agent fine-tune a model for you, end to end?
A coding agent can run a whole LoRA fine-tune on Nodus if a platform-enforced dollar cap bounds it and you keep the data, eval and shipping decisions.
vLLM vs SGLang: which LLM inference engine is faster?
Neither engine is faster everywhere. SGLang leads on shared-prefix and structured workloads, vLLM on breadth. A decision rule and cost math.
Can you train your own LLM, and what does it actually take?
Yes. Most teams should fine-tune an open model for a few dollars. Continued pretraining runs about $1,200 and scratch about $19,000 for an 8B model.
RAG vs fine-tuning: which should your LLM app use?
Use RAG for facts that change or need citing, fine-tuning for format and behavior. Token math, a $1.54 fine-tune and when a dedicated GPU pays off.
How long does it take to train an AI model on cloud GPUs?
Minutes for a small fine-tune, days for a 7B model from scratch, months for a frontier run. The formula, real published runs and wall-clock tables.
How much does it cost to train an LLM from scratch?
From about $300 for a 1B model to tens of millions for a frontier run. The GPU-hour formula, real published runs and worked costs at H100 list rates.