LLM Fine-Tuning Guide: LoRA, QLoRA, and RLHF for Enterprise AI

Fine-tuning large language models on proprietary data is one of the most powerful ways to build competitive AI products. But it's also one of the most misunderstood — many teams fine-tune when they shouldn't, or use the wrong technique for their use case.

This guide covers when fine-tuning makes sense, which techniques to use, and how to go from training to production.

When to Fine-Tune vs. When to Use RAG

This is the most important decision and most teams get it wrong:

Use RAG when:

• You need the model to access specific, frequently changing knowledge
• Your use case is primarily question-answering over documents
• You need source attribution and citations
• You want to avoid the cost and complexity of training

Use Fine-Tuning when:

• You need the model to adopt a specific behavior, style, or format
• You're building a domain expert (medical, legal, financial)
• Prompt engineering has hit its limits and you need consistent structured output
• You need to significantly improve performance on a specific task
• Latency is critical and you want to avoid the retrieval overhead of RAG

Use Both when:

• You need domain-specific behavior AND access to specific knowledge
• Example: A fine-tuned medical model that also retrieves from patient records

Fine-Tuning Techniques Compared

Full Fine-Tuning

Trains all model parameters. Maximum quality but requires significant GPU resources:

• GPU requirements: 4-8x A100/H100 80GB for a 7B model, much more for larger
• Training time: Hours to days depending on dataset size
• Best for: When you have abundant compute and need maximum performance
• Risk: Catastrophic forgetting if not carefully managed

LoRA (Low-Rank Adaptation)

Trains small adapter matrices instead of the full model. The breakthrough technique that made fine-tuning accessible:

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,                       # Rank of the update matrices
    lora_alpha=32,              # Scaling factor
    target_modules=[            # Which layers to adapt
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(base_model, lora_config)
# Trainable params: ~0.5% of total

• GPU requirements: 1x A100 80GB for a 7B model
• Training time: 1-4 hours for most datasets
• Quality: 90-98% of full fine-tuning quality
• Best for: Most production use cases

QLoRA (Quantized LoRA)

Combines 4-bit quantization with LoRA for even lower memory requirements:

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto"
)

• GPU requirements: 1x A100 40GB (or even 24GB consumer GPU) for a 7B model
• Training time: 2-6 hours
• Quality: 85-95% of full fine-tuning quality
• Best for: Budget-constrained training, experimentation, smaller teams

RLHF (Reinforcement Learning from Human Feedback)

Aligns model behavior with human preferences through a reward model:

1. Supervised Fine-Tuning (SFT): Train on high-quality examples
2. Reward Model Training: Train a model to predict human preferences
3. PPO Training: Use the reward model to optimize the base model

RLHF is complex and expensive but essential for:

• Safety-critical applications
• Customer-facing chatbots
• Applications where "helpfulness" is subjective

DPO (Direct Preference Optimization)

A simpler alternative to RLHF that skips the reward model:

from trl import DPOTrainer, DPOConfig

training_args = DPOConfig(
    output_dir="./dpo_output",
    per_device_train_batch_size=4,
    learning_rate=5e-7,
    num_train_epochs=3,
    beta=0.1,  # KL divergence penalty
)

trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    args=training_args,
    train_dataset=preference_dataset,  # (prompt, chosen, rejected) triples
    tokenizer=tokenizer,
)

DPO achieves comparable results to RLHF with much simpler implementation. It's our recommended approach for preference alignment in production.

Data Preparation

Data quality is the single biggest factor in fine-tuning success. "Garbage in, garbage out" is especially true for LLMs.

Dataset Requirements

Data Quality Checklist

1. Consistency: All examples should follow the same format and quality standard
2. Diversity: Cover the full range of expected inputs and edge cases
3. Deduplication: Remove exact and near-duplicate examples
4. Balance: Avoid over-representing any single category or pattern
5. Quality review: Have domain experts validate a random sample
6. Negative examples: Include examples of what NOT to do for safety-critical applications

Data Format

{
  "messages": [
    {"role": "system", "content": "You are a financial analyst..."},
    {"role": "user", "content": "Analyze Q3 revenue trends for..."},
    {"role": "assistant", "content": "Based on the quarterly report..."}
  ]
}

Training Best Practices

Hyperparameter Selection

training_args = TrainingArguments(
    output_dir="./output",
    num_train_epochs=3,              # 2-5 epochs usually sufficient
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,    # Effective batch size = 16
    learning_rate=2e-4,              # LoRA sweet spot
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=50,
    bf16=True,                       # Use bfloat16 on A100/H100
)

Avoiding Catastrophic Forgetting

• Keep learning rates low (1e-5 to 5e-4 for LoRA)
• Use a validation set from the original domain to monitor general capability
• Train for fewer epochs rather than more
• Use LoRA instead of full fine-tuning when possible

Evaluation During Training

Track these metrics throughout training:

• Training loss: Should decrease steadily
• Validation loss: Should decrease then plateau (stop before it increases)
• Task-specific metrics: Accuracy, F1, BLEU, or custom metrics for your use case
• General capability: Run a benchmark suite to ensure the model hasn't degraded

Production Optimization

Quantization for Inference

After fine-tuning, quantize the model for faster and cheaper inference:

# GPTQ quantization for production
from auto_gptq import AutoGPTQForCausalLM

quantized_model = AutoGPTQForCausalLM.from_pretrained(
    model_path,
    quantize_config=QuantizeConfig(bits=4, group_size=128)
)

Serving with vLLM

from vllm import LLM, SamplingParams

llm = LLM(
    model="./fine-tuned-model",
    tensor_parallel_size=2,          # Multi-GPU serving
    max_model_len=4096,
    gpu_memory_utilization=0.9,
)

Merging LoRA Adapters

For production deployment, merge LoRA weights back into the base model to eliminate adapter overhead:

merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged-model")

Cost Comparison

Getting Started

Fine-tuning is a powerful tool, but it's not always the right one. Start with prompt engineering, try RAG if you need knowledge grounding, and fine-tune when you've identified a clear performance gap that can only be closed with training.

Book a consultation with Cloudpush to discuss your fine-tuning strategy and get expert guidance on model selection, data preparation, and production optimization.