Card: training data and safety section, full-benchmark table, specimen stat strip, energy chart
Browse files- README.md +19 -9
- assets/numbers.png +2 -2
- assets/qib.png +2 -2
README.md
CHANGED
|
@@ -139,14 +139,14 @@ way, Agate scores 0.535 and 0.550.
|
|
| 139 |
each image against a per-prompt checklist in five areas. We ran the judge exactly as published: their checklists
|
| 140 |
and aggregation code, greedy decoding, thinking on.
|
| 141 |
|
| 142 |
-
<p align="center"><img src="assets/qib.png" alt="Qwen-Image-Bench scores for Agate
|
| 143 |
|
| 144 |
-
**Full benchmark, all prompts**
|
| 145 |
|
| 146 |
-
| Model | **Total** | Quality | Aesthetics | Alignment | Real-world Fidelity | Creative Generation |
|
| 147 |
-
|---|---|---|---|---|---|---|
|
| 148 |
-
|
|
| 149 |
-
|
|
| 150 |
|
| 151 |
**Every model, 300 evenly spaced prompts:**
|
| 152 |
|
|
@@ -165,7 +165,7 @@ and aggregation code, greedy decoding, thinking on.
|
|
| 165 |
| **Agate** | **28.4** | 26.3 | 34.2 | 28.9 | 36.8 | 15.5 |
|
| 166 |
| Stable Diffusion 1.5 | **28.9** | 37.5 | 28.2 | 25.1 | 39.3 | 14.3 |
|
| 167 |
|
| 168 |
-
On all 1,000 prompts Agate scores **28.2** against SD 1.5's **29.1**, close but behind. It leads on aesthetics (33.8 vs 28.0) and alignment (28.2 vs 25.7). It trails on quality (25.9 vs 36.7) and real-world fidelity (36.6 vs 38.9). The judge sees every image at 512 px, so Agate's 256 px output is upscaled first, which likely weighs on its quality score. Across our lineage the total climbs from 17.3 to 28.4
|
| 169 |
|
| 170 |
Judge outputs that failed to parse: 0. Scores are 0–100.
|
| 171 |
|
|
@@ -259,6 +259,16 @@ short caption, weighted toward *composition* (layout is the weakest GenEval skil
|
|
| 259 |
image's long *detail* caption (median 217 tokens). The model therefore learns both what people type and what
|
| 260 |
carries the concepts, and the jump to 512 tokens stops long captions from being truncated.
|
| 261 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 262 |
## Compute and energy
|
| 263 |
|
| 264 |
<p align="center"><img src="assets/energy.png" alt="Energy of every GPU job in the project, with Agate's three training phases highlighted" width="100%"></p>
|
|
@@ -315,8 +325,8 @@ This card shows the failures as well as the wins:
|
|
| 315 |
- **Text rendering is unreliable**, as are counts above three and negations ("no", "empty", "without").
|
| 316 |
- **Synthetic training data.** Every training image was generated by FLUX.1-dev. Agate inherits that model's
|
| 317 |
look and biases, and it was never trained on a photograph.
|
| 318 |
-
- **No safety
|
| 319 |
-
|
| 320 |
- Intended for research, education, prototyping and small-footprint deployment.
|
| 321 |
|
| 322 |
## Licence
|
|
|
|
| 139 |
each image against a per-prompt checklist in five areas. We ran the judge exactly as published: their checklists
|
| 140 |
and aggregation code, greedy decoding, thinking on.
|
| 141 |
|
| 142 |
+
<p align="center"><img src="assets/qib.png" alt="Qwen-Image-Bench scores for Agate and similar-size open models" width="100%"></p>
|
| 143 |
|
| 144 |
+
**Full benchmark, all 1,000 prompts:** Agate, SD 1.5 and the open text-to-image models closest to Agate in size.
|
| 145 |
|
| 146 |
+
| Model | Size | Native | Licence | **Total** | Quality | Aesthetics | Alignment | Real-world Fidelity | Creative Generation |
|
| 147 |
+
|---|---|---|---|---|---|---|---|---|---|
|
| 148 |
+
| [**Agate**](https://huggingface.co/Logolabs/agate-preview-001) | 191M + 68M text enc. | 256 px | MIT | **28.2** | 25.9 | 33.8 | 28.2 | 36.6 | 14.7 |
|
| 149 |
+
| [Stable Diffusion 1.5](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) | 860M UNet + 123M CLIP | 512 px | CreativeML OpenRAIL-M | **29.1** | 36.7 | 28.0 | 25.7 | 38.9 | 15.5 |
|
| 150 |
|
| 151 |
**Every model, 300 evenly spaced prompts:**
|
| 152 |
|
|
|
|
| 165 |
| **Agate** | **28.4** | 26.3 | 34.2 | 28.9 | 36.8 | 15.5 |
|
| 166 |
| Stable Diffusion 1.5 | **28.9** | 37.5 | 28.2 | 25.1 | 39.3 | 14.3 |
|
| 167 |
|
| 168 |
+
On all 1,000 prompts Agate scores **28.2** against SD 1.5's **29.1**, close but behind. It leads on aesthetics (33.8 vs 28.0) and alignment (28.2 vs 25.7). It trails on quality (25.9 vs 36.7) and real-world fidelity (36.6 vs 38.9). The judge sees every image at 512 px, so Agate's 256 px output is upscaled first, which likely weighs on its quality score. Across our lineage the total climbs from 17.3 to 28.4 on the 300-prompt subset.
|
| 169 |
|
| 170 |
Judge outputs that failed to parse: 0. Scores are 0–100.
|
| 171 |
|
|
|
|
| 259 |
image's long *detail* caption (median 217 tokens). The model therefore learns both what people type and what
|
| 260 |
carries the concepts, and the jump to 512 tokens stops long captions from being truncated.
|
| 261 |
|
| 262 |
+
## Training data and safety
|
| 263 |
+
|
| 264 |
+
**Why FLUX-Reason-6M.** We chose the training set because it is curated and contains no adult images. Agate never
|
| 265 |
+
saw explicit material during training, so it has no learned way to produce it.
|
| 266 |
+
|
| 267 |
+
**Internal safety tests.** We deliberately prompted Agate for nudity and sexual content. We could not get it to
|
| 268 |
+
generate genitals in any attempt. The worst output was a blurry nude female figure with no anatomical detail
|
| 269 |
+
(no areolae). That is a finding from our own red-teaming, not a guarantee. Agate ships without a safety classifier,
|
| 270 |
+
so deployments facing the public should add one.
|
| 271 |
+
|
| 272 |
## Compute and energy
|
| 273 |
|
| 274 |
<p align="center"><img src="assets/energy.png" alt="Energy of every GPU job in the project, with Agate's three training phases highlighted" width="100%"></p>
|
|
|
|
| 325 |
- **Text rendering is unreliable**, as are counts above three and negations ("no", "empty", "without").
|
| 326 |
- **Synthetic training data.** Every training image was generated by FLUX.1-dev. Agate inherits that model's
|
| 327 |
look and biases, and it was never trained on a photograph.
|
| 328 |
+
- **No safety classifier** is bundled with the pipeline. Photorealistic people can be generated. Do not use Agate
|
| 329 |
+
to depict real people or to deceive.
|
| 330 |
- Intended for research, education, prototyping and small-footprint deployment.
|
| 331 |
|
| 332 |
## Licence
|
assets/numbers.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/qib.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|