Incorporo-user commited on
Commit
e8dfd7d
·
verified ·
1 Parent(s): df266b9

Card: training data and safety section, full-benchmark table, specimen stat strip, energy chart

Browse files
Files changed (3) hide show
  1. README.md +19 -9
  2. assets/numbers.png +2 -2
  3. assets/qib.png +2 -2
README.md CHANGED
@@ -139,14 +139,14 @@ way, Agate scores 0.535 and 0.550.
139
  each image against a per-prompt checklist in five areas. We ran the judge exactly as published: their checklists
140
  and aggregation code, greedy decoding, thinking on.
141
 
142
- <p align="center"><img src="assets/qib.png" alt="Qwen-Image-Bench scores for Agate, SD 1.5 and Supra2-IMG" width="100%"></p>
143
 
144
- **Full benchmark, all prompts** (Agate and SD 1.5):
145
 
146
- | Model | **Total** | Quality | Aesthetics | Alignment | Real-world Fidelity | Creative Generation |
147
- |---|---|---|---|---|---|---|
148
- | Stable Diffusion 1.5 | **29.1** | 36.7 | 28.0 | 25.7 | 38.9 | 15.5 |
149
- | **Agate** | **28.2** | 25.9 | 33.8 | 28.2 | 36.6 | 14.7 |
150
 
151
  **Every model, 300 evenly spaced prompts:**
152
 
@@ -165,7 +165,7 @@ and aggregation code, greedy decoding, thinking on.
165
  | **Agate** | **28.4** | 26.3 | 34.2 | 28.9 | 36.8 | 15.5 |
166
  | Stable Diffusion 1.5 | **28.9** | 37.5 | 28.2 | 25.1 | 39.3 | 14.3 |
167
 
168
- On all 1,000 prompts Agate scores **28.2** against SD 1.5's **29.1**, close but behind. It leads on aesthetics (33.8 vs 28.0) and alignment (28.2 vs 25.7). It trails on quality (25.9 vs 36.7) and real-world fidelity (36.6 vs 38.9). The judge sees every image at 512 px, so Agate's 256 px output is upscaled first, which likely weighs on its quality score. Across our lineage the total climbs from 17.3 to 28.4; Supra2-IMG scores 17.7.
169
 
170
  Judge outputs that failed to parse: 0. Scores are 0–100.
171
 
@@ -259,6 +259,16 @@ short caption, weighted toward *composition* (layout is the weakest GenEval skil
259
  image's long *detail* caption (median 217 tokens). The model therefore learns both what people type and what
260
  carries the concepts, and the jump to 512 tokens stops long captions from being truncated.
261
 
 
 
 
 
 
 
 
 
 
 
262
  ## Compute and energy
263
 
264
  <p align="center"><img src="assets/energy.png" alt="Energy of every GPU job in the project, with Agate's three training phases highlighted" width="100%"></p>
@@ -315,8 +325,8 @@ This card shows the failures as well as the wins:
315
  - **Text rendering is unreliable**, as are counts above three and negations ("no", "empty", "without").
316
  - **Synthetic training data.** Every training image was generated by FLUX.1-dev. Agate inherits that model's
317
  look and biases, and it was never trained on a photograph.
318
- - **No safety filter** is included, and the training data was not filtered by us. Photorealistic people can be
319
- generated. Do not use Agate to depict real people or to deceive.
320
  - Intended for research, education, prototyping and small-footprint deployment.
321
 
322
  ## Licence
 
139
  each image against a per-prompt checklist in five areas. We ran the judge exactly as published: their checklists
140
  and aggregation code, greedy decoding, thinking on.
141
 
142
+ <p align="center"><img src="assets/qib.png" alt="Qwen-Image-Bench scores for Agate and similar-size open models" width="100%"></p>
143
 
144
+ **Full benchmark, all 1,000 prompts:** Agate, SD 1.5 and the open text-to-image models closest to Agate in size.
145
 
146
+ | Model | Size | Native | Licence | **Total** | Quality | Aesthetics | Alignment | Real-world Fidelity | Creative Generation |
147
+ |---|---|---|---|---|---|---|---|---|---|
148
+ | [**Agate**](https://huggingface.co/Logolabs/agate-preview-001) | 191M + 68M text enc. | 256 px | MIT | **28.2** | 25.9 | 33.8 | 28.2 | 36.6 | 14.7 |
149
+ | [Stable Diffusion 1.5](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) | 860M UNet + 123M CLIP | 512 px | CreativeML OpenRAIL-M | **29.1** | 36.7 | 28.0 | 25.7 | 38.9 | 15.5 |
150
 
151
  **Every model, 300 evenly spaced prompts:**
152
 
 
165
  | **Agate** | **28.4** | 26.3 | 34.2 | 28.9 | 36.8 | 15.5 |
166
  | Stable Diffusion 1.5 | **28.9** | 37.5 | 28.2 | 25.1 | 39.3 | 14.3 |
167
 
168
+ On all 1,000 prompts Agate scores **28.2** against SD 1.5's **29.1**, close but behind. It leads on aesthetics (33.8 vs 28.0) and alignment (28.2 vs 25.7). It trails on quality (25.9 vs 36.7) and real-world fidelity (36.6 vs 38.9). The judge sees every image at 512 px, so Agate's 256 px output is upscaled first, which likely weighs on its quality score. Across our lineage the total climbs from 17.3 to 28.4 on the 300-prompt subset.
169
 
170
  Judge outputs that failed to parse: 0. Scores are 0–100.
171
 
 
259
  image's long *detail* caption (median 217 tokens). The model therefore learns both what people type and what
260
  carries the concepts, and the jump to 512 tokens stops long captions from being truncated.
261
 
262
+ ## Training data and safety
263
+
264
+ **Why FLUX-Reason-6M.** We chose the training set because it is curated and contains no adult images. Agate never
265
+ saw explicit material during training, so it has no learned way to produce it.
266
+
267
+ **Internal safety tests.** We deliberately prompted Agate for nudity and sexual content. We could not get it to
268
+ generate genitals in any attempt. The worst output was a blurry nude female figure with no anatomical detail
269
+ (no areolae). That is a finding from our own red-teaming, not a guarantee. Agate ships without a safety classifier,
270
+ so deployments facing the public should add one.
271
+
272
  ## Compute and energy
273
 
274
  <p align="center"><img src="assets/energy.png" alt="Energy of every GPU job in the project, with Agate's three training phases highlighted" width="100%"></p>
 
325
  - **Text rendering is unreliable**, as are counts above three and negations ("no", "empty", "without").
326
  - **Synthetic training data.** Every training image was generated by FLUX.1-dev. Agate inherits that model's
327
  look and biases, and it was never trained on a photograph.
328
+ - **No safety classifier** is bundled with the pipeline. Photorealistic people can be generated. Do not use Agate
329
+ to depict real people or to deceive.
330
  - Intended for research, education, prototyping and small-footprint deployment.
331
 
332
  ## Licence
assets/numbers.png CHANGED

Git LFS Details

  • SHA256: 147f4e58a302c4035590192e95f3f740db56110ba9655fd11a7cb701dd22efae
  • Pointer size: 131 Bytes
  • Size of remote file: 104 kB

Git LFS Details

  • SHA256: 767bc0df86c37c93bc65e710bb5cfc5138532f8330add4ffd0f7cb3c537ae4cd
  • Pointer size: 131 Bytes
  • Size of remote file: 114 kB
assets/qib.png CHANGED

Git LFS Details

  • SHA256: 7ac3765a6146d662eb2882fecb78bed14758f3ce37f2b1254698d432436c3793
  • Pointer size: 131 Bytes
  • Size of remote file: 170 kB

Git LFS Details

  • SHA256: fc46d329cb8ed9963adf47969d954be6f71d171c2a6570cd487e86d5646444dd
  • Pointer size: 131 Bytes
  • Size of remote file: 166 kB