Instructions to use alansmithee-cpu/experimental_merge_241026_v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use alansmithee-cpu/experimental_merge_241026_v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="alansmithee-cpu/experimental_merge_241026_v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("alansmithee-cpu/experimental_merge_241026_v2")
model = AutoModelForCausalLM.from_pretrained("alansmithee-cpu/experimental_merge_241026_v2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use alansmithee-cpu/experimental_merge_241026_v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "alansmithee-cpu/experimental_merge_241026_v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alansmithee-cpu/experimental_merge_241026_v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/alansmithee-cpu/experimental_merge_241026_v2

SGLang

How to use alansmithee-cpu/experimental_merge_241026_v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "alansmithee-cpu/experimental_merge_241026_v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alansmithee-cpu/experimental_merge_241026_v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "alansmithee-cpu/experimental_merge_241026_v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alansmithee-cpu/experimental_merge_241026_v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use alansmithee-cpu/experimental_merge_241026_v2 with Docker Model Runner:
```
docker model run hf.co/alansmithee-cpu/experimental_merge_241026_v2
```

To be added

This model is another and final variant of the earlier update, where a small (0.2) portion of Behemoth-123B-v2.1 was added to both halves. I decided to add this model after seeing how well it performed on UGI Leaderboard.

If you have any feedback or comments, please don't hesitate to chat in the Community tab. All community feedback is warmly welcomed.

Again, still not yet tested due to the lack of computational resources.

Should be my final attempt before I can test the 3 merges I uploaded here first.

Upcoming Testing Plan

Using the following chat template:

[SYSTEM_PROMPT] <|system|>{{system_message}}[/SYSTEM_PROMPT][INST] <|user|>{{user_message}}[/INST] <|model|>{{assistant_message}}

Template is taken from TheDrummer/Behemoth-123B-v2.1, all credits go to the respective author(s)

Temp: 1.0, min_p: 0.03, top_p: 0.95, max_token: 204860

{{system_message}} structure:

SYSTEM

<insert your system prompt here, each section separated by a *** >

---

License

All components derive from Mistral Large 123B and are released under the Mistral AI Research License. This merge inherits that license: research and non-commercial use only. Please refer to the individual component model cards for any additional terms.