ZyroGod/distilgpt2-news-topic-classification

DistilGPT2 News Topic Classifier

This repository provides a fine-tuned distilgpt2 (82M parameter causal language model) adapted for 4-class sequence classification on the AG News dataset (fancyzhx/ag_news).

The model categorizes news headlines and article snippets into four distinct domains:

• World (Class 0)
• Sports (Class 1)
• Business (Class 2)
• Sci/Tech (Class 3)

---

Benchmark Performance (Test Set: 1,000 Samples)

The model was evaluated on a held-out test split of 1,000 balanced samples:

Metric	Score	Percentage
Test Accuracy	0.9130	91.30%
Weighted F1	0.9131	91.31%
Macro F1	0.9134	91.34%
Weighted Precision	0.9142	91.42%
Weighted Recall	0.9130	91.30%

Per-Class Performance Breakdown

Class	Precision	Recall	F1-Score	Accuracy	Test Support
Sports	97.58%	96.80%	97.19%	96.80%	250
World	92.98%	91.84%	92.40%	91.84%	245
Sci/Tech	85.02%	91.53%	88.16%	91.53%	248
Business	90.12%	85.21%	87.60%	85.21%	257

---

Evaluation Visuals

Confusion Matrix

Per-Class Accuracy

Training & Validation Loss/F1 Curves

---

Architectural & Causal LM Details

• Base Model Architecture: distilgpt2 (6 decoder layers, 12 attention heads, 768 embedding dimensions, ~82M parameters).
• Classification Head: Linear projection from the final non-padded token embedding to 4 output logits (summary_type = "cls_index").
• Padding Configuration:
  • Left-Padding (padding_side = "left"): Crucial for decoder-only causal models so that the final position in the sequence contains the target text token rather than padding tokens.
  • Pad Token (pad_token = eos_token): DistilGPT2 has no default pad token, so eos_token (<|endoftext|>, ID 50256) is reused for padding.

---

Hyperparameters & Training Setup

• Dataset: fancyzhx/ag_news (2,400 balanced train, 400 val, 1,000 test)
• Epochs: 10
• Micro-Batch Size: 8 (per-device)
• Gradient Accumulation Steps: 2 (effective batch size: 16)
• Learning Rate: 2e-5 (linear warmup for 5% of steps, cosine decay)
• Optimizer: adamw_torch_fused (weight decay: 0.01)
• Precision: Mixed Precision FP16 (AMP)
• Sequence Length: 128 tokens
• Target Compute: NVIDIA GeForce RTX 3050 Laptop GPU (4GB VRAM)

---

Quickstart & Usage

1. High-Level Transformers Pipeline

from transformers import pipeline

# Load classifier pipeline directly from Hugging Face Hub
classifier = pipeline("text-classification", model="ZyroGod/distilgpt2-news-topic-classification")

# Predict on news headlines
headline = "NASA's James Webb Space Telescope discovers oldest galaxy cluster ever observed."
result = classifier(headline)

print(result)
# Output: [{'label': 'Sci/Tech', 'score': 0.9852}]

2. PyTorch Native AutoModel & AutoTokenizer

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "ZyroGod/distilgpt2-news-topic-classification"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

# Ensure proper left padding for causal sequence classification
tokenizer.padding_side = "left"
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

headline = "Wall Street rallies as Federal Reserve signals potential interest rate cuts."
inputs = tokenizer(
    headline,
    return_tensors="pt",
    truncation=True,
    max_length=128,
    padding=True
)

with torch.no_grad():
    outputs = model(**inputs)
    probabilities = torch.softmax(outputs.logits, dim=-1)[0]

id2label = model.config.id2label
for idx, prob in enumerate(probabilities):
    label = id2label.get(str(idx), id2label.get(idx, f"Class {idx}"))
    print(f"{label:<10s}: {prob.item():.2%}")

---

Label Mapping

Label ID	Category Name	Description
0	World	International news, geopolitics, conflicts, foreign diplomacy
1	Sports	Competitions, tournament coverage, team news, athlete statistics
2	Business	Financial markets, stocks, macroeconomic policy, corporations
3	Sci/Tech	Science, space exploration, software, hardware, AI breakthroughs

---

Intended Use & Limitations

• Intended Use: High-throughput categorization of English news headlines, RSS feeds, financial/tech wire services, and article abstracts.
• Limitations: Optimized for English language content. Non-English text or text significantly exceeding 128 tokens may experience lower accuracy.

---

License

This model is licensed under the Apache 2.0 License.