Add files using upload-large-folder tool

README.md

@@ -0,0 +1,119 @@
+---
+license: other
+license_name: mixed-source-research-preview
+base_model: yafitzdev/pyrrho-opsis-v1-nano-g1
+library_name: transformers
+pipeline_tag: image-text-to-text
+language:
+  - en
+tags:
+  - pyrrho
+  - opsis
+  - rag
+  - image-to-text
+  - document-ai
+  - table-recognition
+  - onnx
+  - cpu
+---
+
+# Pyrrho Opsis v1 nano g2
+
+Pyrrho Opsis is a small local visual-ingestion co-processor for RAG systems. It converts one image
+into either concise discovery text or a GitHub-flavored Markdown table. Charts and diagrams are
+described with visible labels, values, and relationships. It is designed to let retrieval systems
+index visual PDF regions without a hosted VLM API or inference GPU.
+
+This is a research preview, not an authoritative OCR or financial/scientific extraction system.
+
+## Intended use
+
+- enrich image, chart, diagram, and table crops with searchable RAG text;
+- run locally through ONNX Runtime on CPU;
+- route native PDF text around the model and send only visual regions through Opsis;
+- preserve the source image/page alongside generated text for verification.
+
+Do not treat generated numbers or table cells as ground truth. Do not use it for safety-critical
+decisions without independent source verification.
+
+## Output contract
+
+Input: one raster image.
+
+Output:
+
+- `<description>concise factual text</description>` for ordinary images, charts, and diagrams;
+- `<table>...Markdown table...</table>` for table images.
+
+The packaged parser strips the wrapper tags from its public result.
+
+## Model
+
+- base architecture: SmolVLM-256M-Instruct;
+- immediate parent checkpoint: Pyrrho Opsis v1 nano g1;
+- generation: Pyrrho Opsis v1 nano g2;
+- total parameters: 258,099,264;
+- adapted parameters: 1,614,336 (0.625%);
+- LoRA targets: text attention, vision attention, and the vision-to-text connector;
+- training resolution: 1024-pixel longest edge.
+
+G2 retains the v1 capability and output contract. It is generation 2 because it changes data
+balance and vision adaptation, not the fundamental task or heads.
+
+## CPU usage
+
+With the `rag-image-parser` project installed:
+
+```powershell
+uv run rag-image-parse parse .\image.png `
+  --model yafitzdev/pyrrho-opsis-v1-nano-g2 `
+  --backend onnx `
+  --onnx-variant fp32 `
+  --threads 8 `
+  --image-longest-edge 1024
+```
+
+FP32 ONNX is the recommended and packaged artifact. Mixed INT8 reduced routing and Markdown
+validity in evaluation and is not included in this repository.
+
+## Evaluation
+
+On a 50-case, source-held-out automatic comparison containing ten examples each from tables,
+charts, diagrams, ordinary images, and PDF-domain crops:
+
+| Metric | G1 | G2 |
+|---|---:|---:|
+| Kind accuracy | 100% | 100% |
+| Valid Markdown tables | 100% | 100% |
+| Exact tables | 0% | 7.1% |
+| Correct table shape | 21.4% | 42.9% |
+| Mean table cell F1 | 31.0% | 50.5% |
+| Required-term recall | 53.8% | 57.7% |
+| Directed-relation recall | 34.0% | 48.7% |
+| Median CPU latency | 3.71 s | 3.72 s |
+| p95 CPU latency | 16.74 s | 10.95 s |
+| Peak process RSS | 2,377 MiB | 2,382 MiB |
+
+This benchmark is automatically labeled and too small for a product-quality claim. Exact table
+accuracy remains low.
+
+## Training data
+
+G2 trained on 50,000 rendered files:
+
+- 12,500 clean tables;
+- 6,250 clean charts;
+- 6,250 clean diagrams;
+- 12,500 clean ordinary images;
+- 12,500 PDF-style crops balanced across the four content types.
+
+Sources include PubTabNet, ChartQA, AI2D, and DOCCI. Charts, diagrams, and ordinary images use
+deterministic label-preserving variants because their strict unique-source pools are smaller than
+the rendered file counts.
+
+## License
+
+Mixed-source research preview. The training mixture includes research-restricted AI2D data and
+sources with separate attribution, redistribution, or underlying-image terms. This package must
+not be represented as commercially cleared. Review and satisfy every source obligation before
+making the repository public or using the weights commercially.

SHA256SUMS

@@ -0,0 +1,3 @@
+b8e6497f118a586d02594d0dadf4d7316df3b5a4e35d54df5afab051dee9b30a  onnx/vision_encoder.onnx
+a68ea4058524059ed0e2873fdbc4f33b93e13ae3d3204938d32d0566d390ae00  onnx/embed_tokens.onnx
+322b33decff7518392dc915377f6a11d33a4570e5f7b58c46017b4d892ee9a74  onnx/decoder_model_merged.onnx

chat_template.jinja

@@ -0,0 +1,2 @@
+<|im_start|>{% for message in messages %}{{message['role'] | capitalize}}{% if message['content'][0]['type'] == 'image' %}{{':'}}{% else %}{{': '}}{% endif %}{% for line in message['content'] %}{% if line['type'] == 'text' %}{{line['text']}}{% elif line['type'] == 'image' %}{{ '<image>' }}{% endif %}{% endfor %}<end_of_utterance>
+{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}

config.json

@@ -0,0 +1,194 @@
+{
+  "architectures": [
+    "Idefics3ForConditionalGeneration"
+  ],
+  "dtype": "float32",
+  "image_token_id": 49190,
+  "model_type": "idefics3",
+  "pad_token_id": 128002,
+  "scale_factor": 4,
+  "text_config": {
+    "_attn_implementation_autoset": false,
+    "_flash_attn_2_enabled": true,
+    "_name_or_path": "None",
+    "add_cross_attention": false,
+    "architectures": [
+      "VLlama3ForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 1,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "dtype": "float32",
+    "eos_token_id": 2,
+    "finetuning_task": null,
+    "head_dim": 64,
+    "hidden_act": "silu",
+    "hidden_size": 576,
+    "initializer_range": 0.041666666666666664,
+    "intermediate_size": 1536,
+    "is_decoder": false,
+    "is_llama_config": true,
+    "max_position_embeddings": 8192,
+    "mlp_bias": false,
+    "model_type": "llama",
+    "neftune_noise_alpha": 0.0,
+    "num_attention_heads": 9,
+    "num_hidden_layers": 30,
+    "num_key_value_heads": 3,
+    "pad_token_id": 2,
+    "perceiver_config": {
+      "_attn_implementation_autoset": false,
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "attention_dropout": 0.0,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "cross_attention_hidden_size": null,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": null,
+      "exponential_decay_length_penalty": null,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "hidden_act": "silu",
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "min_length": 0,
+      "model_type": "vllama3",
+      "no_repeat_ngram_size": 0,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_key_value_heads": 1,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": null,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "qk_layer_norms_perceiver": false,
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "resampler_depth": 6,
+      "resampler_head_dim": 96,
+      "resampler_n_heads": 16,
+      "resampler_n_latents": 64,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "sep_token_id": null,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": true,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torch_dtype": null,
+      "torchscript": false,
+      "transformers_version": "4.46.0",
+      "typical_p": 1.0,
+      "use_bfloat16": false
+    },
+    "pixel_shuffle_factor": 4,
+    "prefix": null,
+    "pretraining_tp": 1,
+    "pruned_heads": {},
+    "qk_layer_norms": false,
+    "rms_norm_eps": 1e-05,
+    "rope_interleaved": false,
+    "rope_parameters": {
+      "rope_theta": 100000,
+      "rope_type": "default"
+    },
+    "sep_token_id": null,
+    "task_specific_params": null,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "torchscript": false,
+    "transformers.js_config": {
+      "kv_cache_dtype": {
+        "fp16": "float16",
+        "q4f16": "float16"
+      }
+    },
+    "use_bfloat16": false,
+    "use_cache": true,
+    "use_resampler": false,
+    "vocab_size": 49280
+  },
+  "tie_word_embeddings": false,
+  "transformers.js_config": {
+    "kv_cache_dtype": {
+      "fp16": "float16",
+      "q4f16": "float16"
+    }
+  },
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vision_config": {
+    "_attn_implementation_autoset": false,
+    "add_cross_attention": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": null,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "dtype": "float32",
+    "eos_token_id": null,
+    "finetuning_task": null,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 768,
+    "image_size": 512,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "is_decoder": false,
+    "layer_norm_eps": 1e-06,
+    "max_image_size": {
+      "longest_edge": 512
+    },
+    "model_type": "idefics3_vision",
+    "num_attention_heads": 12,
+    "num_channels": 3,
+    "num_hidden_layers": 12,
+    "pad_token_id": null,
+    "patch_size": 16,
+    "prefix": null,
+    "pruned_heads": {},
+    "sep_token_id": null,
+    "size": {
+      "longest_edge": 2048
+    },
+    "task_specific_params": null,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "torchscript": false,
+    "use_base_siglip": true,
+    "use_bfloat16": false
+  },
+  "vocab_size": 49280
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 49279,
+  "pad_token_id": 2,
+  "transformers_version": "5.15.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3c6a4e43d52279e72e8b652da487b56e1865dcdea013bac3fd645c0d539cf96
+size 1025998224

onnx/decoder_model_merged.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:322b33decff7518392dc915377f6a11d33a4570e5f7b58c46017b4d892ee9a74
+size 540640062

onnx/embed_tokens.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a68ea4058524059ed0e2873fdbc4f33b93e13ae3d3204938d32d0566d390ae00
+size 113541419

onnx/vision_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8e6497f118a586d02594d0dadf4d7316df3b5a4e35d54df5afab051dee9b30a
+size 374316497

processor_config.json

@@ -0,0 +1,31 @@
+{
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_image_splitting": true,
+    "do_normalize": true,
+    "do_pad": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Idefics3ImageProcessor",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_image_size": {
+      "longest_edge": 512
+    },
+    "resample": 1,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 1024
+    }
+  },
+  "image_seq_len": 64,
+  "processor_class": "Idefics3Processor"
+}

rag_image_parser_config.json

@@ -0,0 +1,25 @@
+{
+  "format": "rag-image-parser-split-onnx-v1",
+  "checkpoint": "yafitzdev/pyrrho-opsis-v1-nano-g2",
+  "parent_checkpoint": "yafitzdev/pyrrho-opsis-v1-nano-g1",
+  "base_architecture": "HuggingFaceTB/SmolVLM-256M-Instruct",
+  "prompt_version": "rag-image-v3",
+  "image_longest_edge": 1024,
+  "recommended_backend": "onnxruntime-cpu",
+  "variants": [
+    "fp32"
+  ],
+  "patched_connector_weights": 1,
+  "patched_vision_weights": 48,
+  "patched_decoder_weights": 120,
+  "training": {
+    "rendered_files": 50000,
+    "epochs": 1.0,
+    "learning_rate": 0.0001,
+    "trainable_parameters": 1614336,
+    "total_parameters": 258099264,
+    "lora_rank": 8,
+    "lora_alpha": 16,
+    "vision_lora": true
+  }
+}

tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|im_start|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<end_of_utterance>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<fake_token_around_image>",
+    "<image>",
+    "<end_of_utterance>"
+  ],
+  "is_local": true,
+  "legacy": false,
+  "local_files_only": false,
+  "max_length": null,
+  "model_max_length": 8192,
+  "pad_to_multiple_of": null,
+  "pad_token": "<|im_end|>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "processor_class": "Idefics3Processor",
+  "size": {
+    "longest_edge": 1024
+  },
+  "tokenizer_class": "GPT2Tokenizer",
+  "truncation_side": "left",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}