Fill-Mask
Transformers
Safetensors
English
magic-bert
binary-analysis
file-type-detection
byte-level
mlm
magic-bytes
security
custom_code
Eval Results (legacy)
Instructions to use mjbommar/magic-bert-50m-mlm with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mjbommar/magic-bert-50m-mlm with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="mjbommar/magic-bert-50m-mlm", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("mjbommar/magic-bert-50m-mlm", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from mjbommar/magic-bert-50m-mlm: direct link, hf CLI and curl.
- Browser
- Download file 11.2 kB
-
https://huggingface.co/mjbommar/magic-bert-50m-mlm/resolve/main/README.md
- Command line
-
hf download hf://mjbommar/magic-bert-50m-mlm/README.md
-
curl -L -o README.md https://huggingface.co/mjbommar/magic-bert-50m-mlm/resolve/main/README.md
11.2 kB
| language: | |
| - en | |
| license: apache-2.0 | |
| library_name: transformers | |
| tags: | |
| - binary-analysis | |
| - file-type-detection | |
| - byte-level | |
| - fill-mask | |
| - mlm | |
| - magic-bytes | |
| - security | |
| pipeline_tag: fill-mask | |
| model-index: | |
| - name: magic-bert-50m-mlm | |
| results: | |
| - task: | |
| type: fill-mask | |
| name: Masked Language Modeling | |
| metrics: | |
| - name: Perplexity | |
| type: perplexity | |
| value: 1.05 | |
| - name: Fill-mask Top-1 Accuracy | |
| type: accuracy | |
| value: 58.9 | |
| - name: Fill-mask Top-5 Accuracy | |
| type: accuracy | |
| value: 73.5 | |
| - name: Probing Classification Accuracy | |
| type: accuracy | |
| value: 87.0 | |
| # Magic-BERT 50M MLM | |
| A BERT-style transformer model trained for binary file understanding using masked language modeling (MLM). This model learns byte-level patterns in binary files, including magic bytes, headers, and structural patterns across 106 file types. | |
| ## Why Not Just Use libmagic? | |
| For intact files starting at byte 0, libmagic works well. But libmagic matches *signatures at fixed offsets*. Magic-BERT learns *structural patterns* throughout the file, enabling use cases where you don't have clean file boundaries: | |
| - **Network streams**: Classifying packet payloads mid-connection, before headers arrive | |
| - **Disk forensics**: Identifying file types during carving, when scanning raw disk images without filesystem metadata | |
| - **Fragment analysis**: Working with partial files, slack space, or corrupted data | |
| - **Adversarial contexts**: Detecting file types when magic bytes are stripped, spoofed, or deliberately misleading | |
| ## Model Description | |
| Magic-BERT uses a custom BERT architecture with absolute position embeddings, trained on binary file data using a byte-level BPE tokenizer. The MLM objective teaches the model to predict masked bytes given surrounding context, which implicitly learns file format structure. | |
| | Property | Value | | |
| |----------|-------| | |
| | Parameters | 59M | | |
| | Hidden Size | 512 | | |
| | Layers | 8 | | |
| | Attention Heads | 8 | | |
| | Max Sequence Length | 512 tokens | | |
| | Vocabulary Size | 32,768 (byte-level BPE) | | |
| | Position Encoding | Absolute (learned embeddings) | | |
| ### Tokenizer | |
| The tokenizer uses the Binary BPE methodology introduced in [Bommarito (2025)](https://arxiv.org/abs/2511.17573). The original Binary BPE tokenizers (available at [mjbommar/binary-tokenizer-001-64k](https://huggingface.co/mjbommar/binary-tokenizer-001-64k)) were trained exclusively on executable binaries (ELF, PE, Mach-O). This tokenizer uses the same BPE training approach but was trained on a diverse corpus spanning 106 file types including documents, images, audio/video, archives, and source code. | |
| ## Intended Uses | |
| **Primary use cases:** | |
| - Fill-mask: Predicting missing bytes in binary files | |
| - Magic byte and file signature recognition | |
| - Feature extraction for downstream classification | |
| - Research on binary file structure | |
| **Example tasks:** | |
| - Completing partial file headers | |
| - Identifying file type from structure | |
| - Anomaly detection in binary data | |
| ## Detailed Use Cases | |
| ### Network Traffic Analysis | |
| When inspecting packet payloads, you often see file data mid-stream—TCP reassembly may give you bytes 1500-3000 of a PDF before you ever see byte 0. Traditional signature matching fails here. Magic-BERT's structural understanding can identify file types from interior content. | |
| ### Disk Forensics & File Carving | |
| During disk image analysis, you scan raw bytes looking for file boundaries. Tools like Scalpel rely on header/footer signatures, but many files lack clear footers. Magic-BERT can score byte ranges for file type probability, helping identify carved fragments or validate carving results. | |
| ### Incident Response | |
| Malware often strips or modifies magic bytes to evade detection. Polyglot files (valid as multiple types) exploit signature-based tools. Learning structural patterns provides a second opinion that doesn't rely solely on the first few bytes. | |
| ### Embedded Content Detection | |
| Files within files (email attachments, archive contents, OLE streams) may appear at arbitrary offsets. Embeddings from Magic-BERT enable similarity search: "find all chunks that look structurally like JPEG data" regardless of where they appear. | |
| ## Training | |
| ### Data | |
| Trained on a diverse corpus of binary files spanning 106 MIME types, including: | |
| - Documents (PDF, Office formats, OpenDocument) | |
| - Images (PNG, JPEG, GIF, WebP, TIFF) | |
| - Audio/Video (MP3, MP4, WebM, FLAC) | |
| - Archives (ZIP, GZIP, 7z, TAR) | |
| - Executables (ELF, PE, Mach-O) | |
| - And 90+ additional formats | |
| ### Procedure | |
| | Phase | Steps | Learning Rate | Batch Size | Objective | | |
| |-------|-------|---------------|------------|-----------| | |
| | MLM Pre-training | 100,000 | 1e-4 | 240 | Masked LM (15% masking) | | |
| **Data augmentation:** 50% of samples use random byte offset to reduce position bias. | |
| ## Evaluation Results | |
| ### Perplexity by Region | |
| | Region | Perplexity | | |
| |--------|------------| | |
| | Magic Bytes (0-9) | 1.07 | | |
| | Header (10-49) | 1.06 | | |
| | Body (50+) | 1.05 | | |
| | **Overall** | **1.05** | | |
| ### Fill-Mask Accuracy | |
| | Metric | Value | | |
| |--------|-------| | |
| | Top-1 Accuracy | 58.9% | | |
| | Top-5 Accuracy | 73.5% | | |
| | Mean Reciprocal Rank | 0.67 | | |
| ### Representation Quality | |
| | Metric | Value | | |
| |--------|-------| | |
| | Linear Probe Accuracy | 87.0% | | |
| | Silhouette Score | 0.39 | | |
| | Separation Ratio | 2.78 | | |
| ## Architecture: Absolute vs Rotary Position Embeddings | |
| This model uses **absolute position embeddings**, where each position (0-511) has a learned embedding vector added to the token embedding. This is the original BERT approach. | |
| An alternative is **Rotary Position Embeddings (RoPE)**, used by the RoFormer variant. RoPE encodes relative position through rotation matrices applied to query and key vectors in attention, rather than learning absolute position vectors. | |
| **Key finding from our experiments:** Both approaches show similar position bias (~47-48% accuracy drop at offset 1000). Position bias is primarily a data distribution issue (files naturally start at offset 0) rather than an architecture limitation. | |
| | Aspect | Magic-BERT (this) | RoFormer | | |
| |--------|-------------------|----------| | |
| | Position Encoding | Absolute (learned) | RoPE (rotary) | | |
| | Parameters | 59M | 42.3M | | |
| | Perplexity | **1.05** | 1.13 | | |
| | Fill-mask Top-1 | 58.9% | **61.8%** | | |
| | Probing Accuracy | **87.0%** | 85.0% | | |
| Magic-BERT achieves slightly better perplexity and probing accuracy, while RoFormer achieves better fill-mask accuracy with fewer parameters. | |
| ## MLM vs Classification: Two-Phase Training | |
| This is the **Phase 1 (MLM)** model. The training pipeline has two phases: | |
| | Phase | Model | Task | Purpose | | |
| |-------|-------|------|---------| | |
| | **Phase 1** | **This model** | Masked Language Modeling | Learn byte-level patterns and file structure | | |
| | Phase 2 | magic-bert-50m-classification | Contrastive Learning | Optimize embeddings for file type discrimination | | |
| **When to use each:** | |
| - Use **this model (MLM)** for: fill-mask tasks, research, or as a base for custom fine-tuning | |
| - Use **classification model** for: file type detection, similarity search, production classification | |
| ## How to Use | |
| ```python | |
| from transformers import AutoModelForMaskedLM, AutoTokenizer | |
| import torch | |
| model = AutoModelForMaskedLM.from_pretrained("mjbommar/magic-bert-50m-mlm", trust_remote_code=True) | |
| tokenizer = AutoTokenizer.from_pretrained("mjbommar/magic-bert-50m-mlm") | |
| model.eval() | |
| # Read a binary file | |
| with open("example.pdf", "rb") as f: | |
| data = f.read(512) | |
| # Decode bytes to string using latin-1 (preserves all byte values 0-255) | |
| text = data.decode("latin-1") | |
| # Tokenize and mask a position | |
| inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512) | |
| mask_pos = 1 # Mask second token (first is [CLS]) | |
| inputs["input_ids"][0, mask_pos] = tokenizer.mask_token_id | |
| # Predict masked token | |
| with torch.no_grad(): | |
| outputs = model(**inputs) | |
| predictions = outputs.logits[0, mask_pos].topk(5) | |
| print("Top-5 predictions:", tokenizer.convert_ids_to_tokens(predictions.indices)) | |
| ``` | |
| ### Getting Embeddings | |
| ```python | |
| from transformers import AutoModel | |
| # Load base model for embeddings | |
| model = AutoModel.from_pretrained( | |
| "mjbommar/magic-bert-50m-mlm", | |
| trust_remote_code=True | |
| ) | |
| with torch.no_grad(): | |
| outputs = model(**inputs) | |
| # CLS token embedding | |
| cls_embedding = outputs.last_hidden_state[:, 0, :] # [batch_size, 512] | |
| # Mean pooling | |
| mean_embedding = outputs.last_hidden_state.mean(dim=1) # [batch_size, 512] | |
| ``` | |
| ## Limitations | |
| 1. **Position bias:** The model performs best when file content starts at position 0. Accuracy drops ~48% when content starts at offset 1000. This reflects training data distribution, not architectural limitations. | |
| 2. **Sequence length:** Limited to 512 tokens. Longer files require truncation or chunking. | |
| 3. **Text files:** Lower performance on high-entropy or highly variable content (e.g., encrypted data, random bytes). | |
| 4. **Domain specificity:** Trained on common file formats; may not generalize to rare or proprietary formats. | |
| ## Model Selection Guide | |
| | Use Case | Recommended Model | Reason | | |
| |----------|-------------------|--------| | |
| | Fill-mask / byte prediction | **This model** | Best perplexity (1.05) | | |
| | Research baseline | **This model** | Established BERT architecture | | |
| | Classification + fill-mask | magic-bert-50m-classification | Retains 41.8% fill-mask capability | | |
| | **Production classification** | **magic-bert-50m-roformer-classification** | Highest accuracy (93.7%), efficient (42M params) | | |
| ## Related Models | |
| - **[magic-bert-50m-classification](https://huggingface.co/mjbommar/magic-bert-50m-classification)**: Same architecture fine-tuned for classification (89.7% accuracy) | |
| - **[magic-bert-50m-roformer-mlm](https://huggingface.co/mjbommar/magic-bert-50m-roformer-mlm)**: RoFormer variant with rotary position embeddings | |
| - **[magic-bert-50m-roformer-classification](https://huggingface.co/mjbommar/magic-bert-50m-roformer-classification)**: RoFormer variant fine-tuned for classification (93.7% accuracy, recommended for production) | |
| ## Related Work | |
| This model builds on the Binary BPE tokenization approach: | |
| - **Binary BPE Paper**: [Bommarito (2025)](https://arxiv.org/abs/2511.17573) introduced byte-level BPE tokenization for binary analysis, demonstrating 2-3x compression over raw bytes for executable content. | |
| - **Binary BPE Tokenizers**: Pre-trained tokenizers for executables are available at [mjbommar/binary-tokenizer-001-64k](https://huggingface.co/mjbommar/binary-tokenizer-001-64k). | |
| **Key difference**: The original Binary BPE work focused on executable binaries (ELF, PE, Mach-O). Magic-BERT extends this to general file type understanding across 106 diverse formats, using a tokenizer trained on the broader dataset. | |
| ## Citation | |
| A paper describing Magic-BERT, the training methodology, and the dataset is forthcoming. | |
| ```bibtex | |
| @article{bommarito2025binarybpe, | |
| title={Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis}, | |
| author={Bommarito, Michael J., II}, | |
| journal={arXiv preprint arXiv:2511.17573}, | |
| year={2025} | |
| } | |
| ``` | |