magic-bert-50m-mlm / README.md
mjbommar's picture
Upload magic-bert-50m-mlm model files
826210c verified
|
Raw History Blame Contribute Delete
11.2 kB
---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- binary-analysis
- file-type-detection
- byte-level
- fill-mask
- mlm
- magic-bytes
- security
pipeline_tag: fill-mask
model-index:
- name: magic-bert-50m-mlm
results:
- task:
type: fill-mask
name: Masked Language Modeling
metrics:
- name: Perplexity
type: perplexity
value: 1.05
- name: Fill-mask Top-1 Accuracy
type: accuracy
value: 58.9
- name: Fill-mask Top-5 Accuracy
type: accuracy
value: 73.5
- name: Probing Classification Accuracy
type: accuracy
value: 87.0
---
# Magic-BERT 50M MLM
A BERT-style transformer model trained for binary file understanding using masked language modeling (MLM). This model learns byte-level patterns in binary files, including magic bytes, headers, and structural patterns across 106 file types.
## Why Not Just Use libmagic?
For intact files starting at byte 0, libmagic works well. But libmagic matches *signatures at fixed offsets*. Magic-BERT learns *structural patterns* throughout the file, enabling use cases where you don't have clean file boundaries:
- **Network streams**: Classifying packet payloads mid-connection, before headers arrive
- **Disk forensics**: Identifying file types during carving, when scanning raw disk images without filesystem metadata
- **Fragment analysis**: Working with partial files, slack space, or corrupted data
- **Adversarial contexts**: Detecting file types when magic bytes are stripped, spoofed, or deliberately misleading
## Model Description
Magic-BERT uses a custom BERT architecture with absolute position embeddings, trained on binary file data using a byte-level BPE tokenizer. The MLM objective teaches the model to predict masked bytes given surrounding context, which implicitly learns file format structure.
| Property | Value |
|----------|-------|
| Parameters | 59M |
| Hidden Size | 512 |
| Layers | 8 |
| Attention Heads | 8 |
| Max Sequence Length | 512 tokens |
| Vocabulary Size | 32,768 (byte-level BPE) |
| Position Encoding | Absolute (learned embeddings) |
### Tokenizer
The tokenizer uses the Binary BPE methodology introduced in [Bommarito (2025)](https://arxiv.org/abs/2511.17573). The original Binary BPE tokenizers (available at [mjbommar/binary-tokenizer-001-64k](https://huggingface.co/mjbommar/binary-tokenizer-001-64k)) were trained exclusively on executable binaries (ELF, PE, Mach-O). This tokenizer uses the same BPE training approach but was trained on a diverse corpus spanning 106 file types including documents, images, audio/video, archives, and source code.
## Intended Uses
**Primary use cases:**
- Fill-mask: Predicting missing bytes in binary files
- Magic byte and file signature recognition
- Feature extraction for downstream classification
- Research on binary file structure
**Example tasks:**
- Completing partial file headers
- Identifying file type from structure
- Anomaly detection in binary data
## Detailed Use Cases
### Network Traffic Analysis
When inspecting packet payloads, you often see file data mid-stream—TCP reassembly may give you bytes 1500-3000 of a PDF before you ever see byte 0. Traditional signature matching fails here. Magic-BERT's structural understanding can identify file types from interior content.
### Disk Forensics & File Carving
During disk image analysis, you scan raw bytes looking for file boundaries. Tools like Scalpel rely on header/footer signatures, but many files lack clear footers. Magic-BERT can score byte ranges for file type probability, helping identify carved fragments or validate carving results.
### Incident Response
Malware often strips or modifies magic bytes to evade detection. Polyglot files (valid as multiple types) exploit signature-based tools. Learning structural patterns provides a second opinion that doesn't rely solely on the first few bytes.
### Embedded Content Detection
Files within files (email attachments, archive contents, OLE streams) may appear at arbitrary offsets. Embeddings from Magic-BERT enable similarity search: "find all chunks that look structurally like JPEG data" regardless of where they appear.
## Training
### Data
Trained on a diverse corpus of binary files spanning 106 MIME types, including:
- Documents (PDF, Office formats, OpenDocument)
- Images (PNG, JPEG, GIF, WebP, TIFF)
- Audio/Video (MP3, MP4, WebM, FLAC)
- Archives (ZIP, GZIP, 7z, TAR)
- Executables (ELF, PE, Mach-O)
- And 90+ additional formats
### Procedure
| Phase | Steps | Learning Rate | Batch Size | Objective |
|-------|-------|---------------|------------|-----------|
| MLM Pre-training | 100,000 | 1e-4 | 240 | Masked LM (15% masking) |
**Data augmentation:** 50% of samples use random byte offset to reduce position bias.
## Evaluation Results
### Perplexity by Region
| Region | Perplexity |
|--------|------------|
| Magic Bytes (0-9) | 1.07 |
| Header (10-49) | 1.06 |
| Body (50+) | 1.05 |
| **Overall** | **1.05** |
### Fill-Mask Accuracy
| Metric | Value |
|--------|-------|
| Top-1 Accuracy | 58.9% |
| Top-5 Accuracy | 73.5% |
| Mean Reciprocal Rank | 0.67 |
### Representation Quality
| Metric | Value |
|--------|-------|
| Linear Probe Accuracy | 87.0% |
| Silhouette Score | 0.39 |
| Separation Ratio | 2.78 |
## Architecture: Absolute vs Rotary Position Embeddings
This model uses **absolute position embeddings**, where each position (0-511) has a learned embedding vector added to the token embedding. This is the original BERT approach.
An alternative is **Rotary Position Embeddings (RoPE)**, used by the RoFormer variant. RoPE encodes relative position through rotation matrices applied to query and key vectors in attention, rather than learning absolute position vectors.
**Key finding from our experiments:** Both approaches show similar position bias (~47-48% accuracy drop at offset 1000). Position bias is primarily a data distribution issue (files naturally start at offset 0) rather than an architecture limitation.
| Aspect | Magic-BERT (this) | RoFormer |
|--------|-------------------|----------|
| Position Encoding | Absolute (learned) | RoPE (rotary) |
| Parameters | 59M | 42.3M |
| Perplexity | **1.05** | 1.13 |
| Fill-mask Top-1 | 58.9% | **61.8%** |
| Probing Accuracy | **87.0%** | 85.0% |
Magic-BERT achieves slightly better perplexity and probing accuracy, while RoFormer achieves better fill-mask accuracy with fewer parameters.
## MLM vs Classification: Two-Phase Training
This is the **Phase 1 (MLM)** model. The training pipeline has two phases:
| Phase | Model | Task | Purpose |
|-------|-------|------|---------|
| **Phase 1** | **This model** | Masked Language Modeling | Learn byte-level patterns and file structure |
| Phase 2 | magic-bert-50m-classification | Contrastive Learning | Optimize embeddings for file type discrimination |
**When to use each:**
- Use **this model (MLM)** for: fill-mask tasks, research, or as a base for custom fine-tuning
- Use **classification model** for: file type detection, similarity search, production classification
## How to Use
```python
from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch
model = AutoModelForMaskedLM.from_pretrained("mjbommar/magic-bert-50m-mlm", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("mjbommar/magic-bert-50m-mlm")
model.eval()
# Read a binary file
with open("example.pdf", "rb") as f:
data = f.read(512)
# Decode bytes to string using latin-1 (preserves all byte values 0-255)
text = data.decode("latin-1")
# Tokenize and mask a position
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
mask_pos = 1 # Mask second token (first is [CLS])
inputs["input_ids"][0, mask_pos] = tokenizer.mask_token_id
# Predict masked token
with torch.no_grad():
outputs = model(**inputs)
predictions = outputs.logits[0, mask_pos].topk(5)
print("Top-5 predictions:", tokenizer.convert_ids_to_tokens(predictions.indices))
```
### Getting Embeddings
```python
from transformers import AutoModel
# Load base model for embeddings
model = AutoModel.from_pretrained(
"mjbommar/magic-bert-50m-mlm",
trust_remote_code=True
)
with torch.no_grad():
outputs = model(**inputs)
# CLS token embedding
cls_embedding = outputs.last_hidden_state[:, 0, :] # [batch_size, 512]
# Mean pooling
mean_embedding = outputs.last_hidden_state.mean(dim=1) # [batch_size, 512]
```
## Limitations
1. **Position bias:** The model performs best when file content starts at position 0. Accuracy drops ~48% when content starts at offset 1000. This reflects training data distribution, not architectural limitations.
2. **Sequence length:** Limited to 512 tokens. Longer files require truncation or chunking.
3. **Text files:** Lower performance on high-entropy or highly variable content (e.g., encrypted data, random bytes).
4. **Domain specificity:** Trained on common file formats; may not generalize to rare or proprietary formats.
## Model Selection Guide
| Use Case | Recommended Model | Reason |
|----------|-------------------|--------|
| Fill-mask / byte prediction | **This model** | Best perplexity (1.05) |
| Research baseline | **This model** | Established BERT architecture |
| Classification + fill-mask | magic-bert-50m-classification | Retains 41.8% fill-mask capability |
| **Production classification** | **magic-bert-50m-roformer-classification** | Highest accuracy (93.7%), efficient (42M params) |
## Related Models
- **[magic-bert-50m-classification](https://huggingface.co/mjbommar/magic-bert-50m-classification)**: Same architecture fine-tuned for classification (89.7% accuracy)
- **[magic-bert-50m-roformer-mlm](https://huggingface.co/mjbommar/magic-bert-50m-roformer-mlm)**: RoFormer variant with rotary position embeddings
- **[magic-bert-50m-roformer-classification](https://huggingface.co/mjbommar/magic-bert-50m-roformer-classification)**: RoFormer variant fine-tuned for classification (93.7% accuracy, recommended for production)
## Related Work
This model builds on the Binary BPE tokenization approach:
- **Binary BPE Paper**: [Bommarito (2025)](https://arxiv.org/abs/2511.17573) introduced byte-level BPE tokenization for binary analysis, demonstrating 2-3x compression over raw bytes for executable content.
- **Binary BPE Tokenizers**: Pre-trained tokenizers for executables are available at [mjbommar/binary-tokenizer-001-64k](https://huggingface.co/mjbommar/binary-tokenizer-001-64k).
**Key difference**: The original Binary BPE work focused on executable binaries (ELF, PE, Mach-O). Magic-BERT extends this to general file type understanding across 106 diverse formats, using a tokenizer trained on the broader dataset.
## Citation
A paper describing Magic-BERT, the training methodology, and the dataset is forthcoming.
```bibtex
@article{bommarito2025binarybpe,
title={Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis},
author={Bommarito, Michael J., II},
journal={arXiv preprint arXiv:2511.17573},
year={2025}
}
```