nithinai517/tinyllama-support-dora

TinyLlama Customer Support (DoRA)

TinyLlama-1.1B-Chat fine-tuned with DoRA on the Bitext customer support dataset.

• Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Method: DoRA (r=16, alpha=32, target_modules=q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj)
• Data: bitext/Bitext-customer-support-llm-chatbot-training-dataset (5,000 example subsample)
• Final val loss: ~0.638

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0", torch_dtype=torch.float16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("nithinai517/tinyllama-support-dora")
model = PeftModel.from_pretrained(base_model, "nithinai517/tinyllama-support-dora")