---
language:
  - en
license: apache-2.0
library_name: transformers
tags:
  - binary-analysis
  - file-type-detection
  - byte-level
  - fill-mask
  - mlm
  - magic-bytes
  - security
pipeline_tag: fill-mask
model-index:
  - name: magic-bert-50m-mlm
    results:
      - task:
          type: fill-mask
          name: Masked Language Modeling
        metrics:
          - name: Perplexity
            type: perplexity
            value: 1.05
          - name: Fill-mask Top-1 Accuracy
            type: accuracy
            value: 58.9
          - name: Fill-mask Top-5 Accuracy
            type: accuracy
            value: 73.5
          - name: Probing Classification Accuracy
            type: accuracy
            value: 87.0
---

# Magic-BERT 50M MLM

A BERT-style transformer model trained for binary file understanding using masked language modeling (MLM). This model learns byte-level patterns in binary files, including magic bytes, headers, and structural patterns across 106 file types.

## Why Not Just Use libmagic?

For intact files starting at byte 0, libmagic works well. But libmagic matches *signatures at fixed offsets*. Magic-BERT learns *structural patterns* throughout the file, enabling use cases where you don't have clean file boundaries:

- **Network streams**: Classifying packet payloads mid-connection, before headers arrive
- **Disk forensics**: Identifying file types during carving, when scanning raw disk images without filesystem metadata
- **Fragment analysis**: Working with partial files, slack space, or corrupted data
- **Adversarial contexts**: Detecting file types when magic bytes are stripped, spoofed, or deliberately misleading

## Model Description

Magic-BERT uses a custom BERT architecture with absolute position embeddings, trained on binary file data using a byte-level BPE tokenizer. The MLM objective teaches the model to predict masked bytes given surrounding context, which implicitly learns file format structure.

| Property | Value |
|----------|-------|
| Parameters | 59M |
| Hidden Size | 512 |
| Layers | 8 |
| Attention Heads | 8 |
| Max Sequence Length | 512 tokens |
| Vocabulary Size | 32,768 (byte-level BPE) |
| Position Encoding | Absolute (learned embeddings) |

### Tokenizer

The tokenizer uses the Binary BPE methodology introduced in [Bommarito (2025)](https://arxiv.org/abs/2511.17573). The original Binary BPE tokenizers (available at [mjbommar/binary-tokenizer-001-64k](https://huggingface.co/mjbommar/binary-tokenizer-001-64k)) were trained exclusively on executable binaries (ELF, PE, Mach-O). This tokenizer uses the same BPE training approach but was trained on a diverse corpus spanning 106 file types including documents, images, audio/video, archives, and source code.

## Intended Uses

**Primary use cases:**
- Fill-mask: Predicting missing bytes in binary files
- Magic byte and file signature recognition
- Feature extraction for downstream classification
- Research on binary file structure

**Example tasks:**
- Completing partial file headers
- Identifying file type from structure
- Anomaly detection in binary data

## Detailed Use Cases

### Network Traffic Analysis
When inspecting packet payloads, you often see file data mid-stream—TCP reassembly may give you bytes 1500-3000 of a PDF before you ever see byte 0. Traditional signature matching fails here. Magic-BERT's structural understanding can identify file types from interior content.

### Disk Forensics & File Carving
During disk image analysis, you scan raw bytes looking for file boundaries. Tools like Scalpel rely on header/footer signatures, but many files lack clear footers. Magic-BERT can score byte ranges for file type probability, helping identify carved fragments or validate carving results.

### Incident Response
Malware often strips or modifies magic bytes to evade detection. Polyglot files (valid as multiple types) exploit signature-based tools. Learning structural patterns provides a second opinion that doesn't rely solely on the first few bytes.

### Embedded Content Detection
Files within files (email attachments, archive contents, OLE streams) may appear at arbitrary offsets. Embeddings from Magic-BERT enable similarity search: "find all chunks that look structurally like JPEG data" regardless of where they appear.

## Training

### Data
Trained on a diverse corpus of binary files spanning 106 MIME types, including:
- Documents (PDF, Office formats, OpenDocument)
- Images (PNG, JPEG, GIF, WebP, TIFF)
- Audio/Video (MP3, MP4, WebM, FLAC)
- Archives (ZIP, GZIP, 7z, TAR)
- Executables (ELF, PE, Mach-O)
- And 90+ additional formats

### Procedure

| Phase | Steps | Learning Rate | Batch Size | Objective |
|-------|-------|---------------|------------|-----------|
| MLM Pre-training | 100,000 | 1e-4 | 240 | Masked LM (15% masking) |

**Data augmentation:** 50% of samples use random byte offset to reduce position bias.

## Evaluation Results

### Perplexity by Region
| Region | Perplexity |
|--------|------------|
| Magic Bytes (0-9) | 1.07 |
| Header (10-49) | 1.06 |
| Body (50+) | 1.05 |
| **Overall** | **1.05** |

### Fill-Mask Accuracy
| Metric | Value |
|--------|-------|
| Top-1 Accuracy | 58.9% |
| Top-5 Accuracy | 73.5% |
| Mean Reciprocal Rank | 0.67 |

### Representation Quality
| Metric | Value |
|--------|-------|
| Linear Probe Accuracy | 87.0% |
| Silhouette Score | 0.39 |
| Separation Ratio | 2.78 |

## Architecture: Absolute vs Rotary Position Embeddings

This model uses **absolute position embeddings**, where each position (0-511) has a learned embedding vector added to the token embedding. This is the original BERT approach.

An alternative is **Rotary Position Embeddings (RoPE)**, used by the RoFormer variant. RoPE encodes relative position through rotation matrices applied to query and key vectors in attention, rather than learning absolute position vectors.

**Key finding from our experiments:** Both approaches show similar position bias (~47-48% accuracy drop at offset 1000). Position bias is primarily a data distribution issue (files naturally start at offset 0) rather than an architecture limitation.

| Aspect | Magic-BERT (this) | RoFormer |
|--------|-------------------|----------|
| Position Encoding | Absolute (learned) | RoPE (rotary) |
| Parameters | 59M | 42.3M |
| Perplexity | **1.05** | 1.13 |
| Fill-mask Top-1 | 58.9% | **61.8%** |
| Probing Accuracy | **87.0%** | 85.0% |

Magic-BERT achieves slightly better perplexity and probing accuracy, while RoFormer achieves better fill-mask accuracy with fewer parameters.

## MLM vs Classification: Two-Phase Training

This is the **Phase 1 (MLM)** model. The training pipeline has two phases:

| Phase | Model | Task | Purpose |
|-------|-------|------|---------|
| **Phase 1** | **This model** | Masked Language Modeling | Learn byte-level patterns and file structure |
| Phase 2 | magic-bert-50m-classification | Contrastive Learning | Optimize embeddings for file type discrimination |

**When to use each:**
- Use **this model (MLM)** for: fill-mask tasks, research, or as a base for custom fine-tuning
- Use **classification model** for: file type detection, similarity search, production classification

## How to Use

```python
from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

model = AutoModelForMaskedLM.from_pretrained("mjbommar/magic-bert-50m-mlm", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("mjbommar/magic-bert-50m-mlm")

model.eval()

# Read a binary file
with open("example.pdf", "rb") as f:
    data = f.read(512)

# Decode bytes to string using latin-1 (preserves all byte values 0-255)
text = data.decode("latin-1")

# Tokenize and mask a position
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
mask_pos = 1  # Mask second token (first is [CLS])
inputs["input_ids"][0, mask_pos] = tokenizer.mask_token_id

# Predict masked token
with torch.no_grad():
    outputs = model(**inputs)
    predictions = outputs.logits[0, mask_pos].topk(5)

print("Top-5 predictions:", tokenizer.convert_ids_to_tokens(predictions.indices))
```

### Getting Embeddings

```python
from transformers import AutoModel

# Load base model for embeddings
model = AutoModel.from_pretrained(
    "mjbommar/magic-bert-50m-mlm",
    trust_remote_code=True
)

with torch.no_grad():
    outputs = model(**inputs)
    # CLS token embedding
    cls_embedding = outputs.last_hidden_state[:, 0, :]  # [batch_size, 512]
    # Mean pooling
    mean_embedding = outputs.last_hidden_state.mean(dim=1)  # [batch_size, 512]
```

## Limitations

1. **Position bias:** The model performs best when file content starts at position 0. Accuracy drops ~48% when content starts at offset 1000. This reflects training data distribution, not architectural limitations.

2. **Sequence length:** Limited to 512 tokens. Longer files require truncation or chunking.

3. **Text files:** Lower performance on high-entropy or highly variable content (e.g., encrypted data, random bytes).

4. **Domain specificity:** Trained on common file formats; may not generalize to rare or proprietary formats.

## Model Selection Guide

| Use Case | Recommended Model | Reason |
|----------|-------------------|--------|
| Fill-mask / byte prediction | **This model** | Best perplexity (1.05) |
| Research baseline | **This model** | Established BERT architecture |
| Classification + fill-mask | magic-bert-50m-classification | Retains 41.8% fill-mask capability |
| **Production classification** | **magic-bert-50m-roformer-classification** | Highest accuracy (93.7%), efficient (42M params) |

## Related Models

- **[magic-bert-50m-classification](https://huggingface.co/mjbommar/magic-bert-50m-classification)**: Same architecture fine-tuned for classification (89.7% accuracy)
- **[magic-bert-50m-roformer-mlm](https://huggingface.co/mjbommar/magic-bert-50m-roformer-mlm)**: RoFormer variant with rotary position embeddings
- **[magic-bert-50m-roformer-classification](https://huggingface.co/mjbommar/magic-bert-50m-roformer-classification)**: RoFormer variant fine-tuned for classification (93.7% accuracy, recommended for production)

## Related Work

This model builds on the Binary BPE tokenization approach:

- **Binary BPE Paper**: [Bommarito (2025)](https://arxiv.org/abs/2511.17573) introduced byte-level BPE tokenization for binary analysis, demonstrating 2-3x compression over raw bytes for executable content.
- **Binary BPE Tokenizers**: Pre-trained tokenizers for executables are available at [mjbommar/binary-tokenizer-001-64k](https://huggingface.co/mjbommar/binary-tokenizer-001-64k).

**Key difference**: The original Binary BPE work focused on executable binaries (ELF, PE, Mach-O). Magic-BERT extends this to general file type understanding across 106 diverse formats, using a tokenizer trained on the broader dataset.

## Citation

A paper describing Magic-BERT, the training methodology, and the dataset is forthcoming.

```bibtex
@article{bommarito2025binarybpe,
  title={Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis},
  author={Bommarito, Michael J., II},
  journal={arXiv preprint arXiv:2511.17573},
  year={2025}
}
```