keshavsharma/tinyllama-1.1b-adcopy-generation-lora

TinyLlama-1.1B Ad Copy Generation — LoRA

This repository contains a LoRA adapter fine-tuned on TinyLlama-1.1B-Chat-v1.0 for generating short text advertisements from a product name and product description.

The model is trained using Supervised Fine-Tuning (SFT) with PEFT/LoRA.

Model Details

• Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Fine-tuning method: LoRA
• Task: Text ad copy generation
• Framework: PyTorch + Transformers + PEFT
• Training dataset: smangrul/ad-copy-generation
• Maximum sequence length: 512 tokens

The model is designed to take a product and its description and generate advertising copy.

Intended Input / Output

Input

The model expects:

Product: Fitness Magazine
Description: Fitness magazine for staying active and achieving your fitness goals.

Output

For example:

Stay active with a Fitness Magazine! 💪📖 Experience fitness tips and motivating stories. Perfect for fitness enthusiasts and reaching your health and wellness goals.

The training prompt instructs the model to:

Create a text ad given the following product and description. Output only the Ad Copy and nothing else.

Training

The model was fine-tuned using LoRA with the following configuration:

Parameter	Value
LoRA rank (r)	32
LoRA alpha	64
LoRA dropout	0.1
Learning rate	2e-4
Epochs	3
LR scheduler	Cosine
Warmup ratio	0.03
Weight decay	0.01
Train batch size	24
Evaluation batch size	24
Maximum sequence length	512
Optimizer	AdamW Torch Fused
Precision	BF16

LoRA Target Modules

LoRA adapters were applied to:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

The prompt portion of each training example was masked from the loss, so training focuses on generating the ad copy rather than reproducing the instruction.

Dataset

Training was performed using:

smangrul/ad-copy-generation

The dataset examples were transformed into a chat-style format containing:

System:
Create a text ad given the following product and description.
Output only the Ad Copy and nothing else.

User:
Product: <product>
Description: <description>

Adcopy:

The expected advertisement was used as the target completion.

Usage

Because this repository contains a LoRA adapter, load it together with the original TinyLlama base model.

Installation

%uv pip install torch transformers peft accelerate

Example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE_MODEL = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
ADAPTER = "keshavsharma/tinyllama-1.1b-adcopy-generation-lora"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER
)

model.eval()

Generate an Ad

product = "Fitness Magazine"
description = "Fitness magazine for staying active and achieving your fitness goals."

messages = [
    {
        "role": "system",
        "content": (
            "Create a text ad given the following product and description. "
            "Output only the Ad Copy and nothing else."
        ),
    },
    {
        "role": "user",
        "content": (
            f"Product: {product}\n"
            f"Description: {description}\n\n"
            "Adcopy: "
        ),
    },
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        temperature=0.8,
        top_p=0.9,
        do_sample=True,
    )

generated = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True
)

print(generated)

Training Objective

The objective of the fine-tuning process is to adapt TinyLlama to the following mapping:

Product + Description
        ↓
   Ad Copy

For example:

Product: Running Shoes
Description: Lightweight running shoes designed for everyday training.
        ↓
Run farther and feel lighter with our performance running shoes...

Limitations

This model is a relatively small 1.1B parameter language model and is specifically fine-tuned for advertising copy generation.

It may:

• Produce repetitive advertisements
• Generate inaccurate product claims
• Add marketing language that is not explicitly present in the description
• Produce inconsistent formatting for unusual inputs
• Perform poorly on tasks outside ad-copy generation

Generated advertising content should therefore be reviewed before being used commercially.

Disclaimer

This model is intended for research and experimentation with small language-model fine-tuning and automated advertising copy generation. Users are responsible for reviewing generated content and ensuring that advertisements comply with applicable advertising, consumer-protection, and platform policies.

License

The base model is:

TinyLlama/TinyLlama-1.1B-Chat-v1.0

Please refer to the base model repository for its licensing terms and conditions.

The LoRA adapter in this repository is provided for research and experimentation.