Refract-Labs

/

code-fdsp-v2

Like 0

Follow

Refract AI Labs 7

@Bc-AI here are the issues:

1. The startup gradient check will reject a healthy model. In gradient_health_summary() names are checked against prefixes such as "vault" and "embedding". FSDP adds _fsdp_wrapped_module., so the gradients are incorrectly classified as cortex gradients. all four other groups become zero, causing the startup probe to raise. Also, gradient norms need to be combined across ranks because individual ranks hold only portions of the parameters.

2. Telemetry and ablation probes bypass FSDP. model.run_telemetry_probe() and model.run_ablation_probe() resolve to methods on the underlying model. Their internal self(...) calls bypass the FSDP wrapper, so parameters aren’t gathered before computation. This can produce tensor-shape errors. The probes should perform forward passes through the wrapped model(...).

3. Checkpoint resume is broken in two ways. In restore_checkpoint() Model weights are saved as .safetensors but loaded with torch.load(). Use the already-imported load_file() instead. Only model_shards[0] and optimizer_shards[0] are loaded. The saving function splits larger states into multiple files, so restoration must load and merge every listed shard.

4. Periodic checkpointing can deadlock the eight GPUs. Each rank independently evaluates the checkpoint condition, although saving requires every rank to participate together. Worse, last_save_time and dirty are updated only on rank zero. Other ranks can therefore enter another save while rank zero continues training. Rank zero should decide when to save, broadcast that decision, and update checkpoint bookkeeping on every rank.

5. Data-pipeline failure recovery can also deadlock. If rank zero’s stream.batch() raises inside distributed_batch(), the other ranks are already waiting for its broadcast. Rank zero then attempts a collective recovery save that they never enter. Broadcast a success/failure status before broadcasting the batch.

6. The suggested OOM fix does nothing. The error messages say to set MICRO_BATCH_SIZE=56, but main() overwrites that setting with:
   MICRO_BATCH_SIZE = GLOBAL_MICRO_BATCH_SIZE // WORLD_SIZE
   With the current settings, that always becomes 14 per GPU. To halve the physical batch while preserving tokens per update, change GLOBAL_MICRO_BATCH_SIZE from 112 to 56; accumulation then becomes 2.

I could be wrong on some of those so take my advice with a grain of salt (or 10)

@Bc-AI where the heck did you get 8xH200s????? how can I get access to some?

@Bc-AI where the heck did you get 8xH200s????? how can I get access to some?

prob paid

@GGUFGuy @Hoglet-33 completely free. i am a year 7 student and i got no job so i cant do paid lol. you have to use a like custom email like yourname@yourwebsite.com for sign up i think https://www.supermicro.com/en/jumpstart

you can also acces B300s and blah at least the person said so the email said if i wanted to trial other stuff i email them.

@Bc-AI good thing i have a custom domain email! so how does it work and how long do you get the server for and can i do it multiple times?

@Hoglet-33 yeah you can do multiple times I think I applied for B300x8 after this. So u go to the site find one u like click apply button fill the form our put in our custom email domain and uh my one is for 5 day idk for others

@Bc-AI Nice! gonna check it out

@GGUFGuy @Hoglet-33 completely free. i am a year 7 student and i got no job so i cant do paid lol. you have to use a like custom email like yourname@yourwebsite.com for sign up i think https://www.supermicro.com/en/jumpstart

Wow!

@GGUFGuy you can try too if you have a student email or company one

@GGUFGuy are you also a student? i am @Hoglet-33 is, @ArushCodes is also

@GGUFGuy are you also a student? i am @Hoglet-33 is, @ArushCodes is also

no, just doing that as a placeholder

@GGUFGuy do u work as a AI dev?

@GGUFGuy do u work as a AI dev?

Kinda? I develop/train my own AI models independently, but I’m not employed as an AI dev lol

@GGUFGuy most ppl aren’t employed as a ai dev lol

If my memory is correct @juiceb0xc0de is like a bar tender or smth

@Bc-AI i know thats as soon as im out of high school im gonna try and work at OpenAI (or make basically AI big)

@Hoglet-33 lol i wanna try that too

@Hoglet-33 yeah you can do multiple times I think I applied for B300x8 after this. So u go to the site find one u like click apply button fill the form our put in our custom email domain and uh my one is for 5 day idk for others

can you make me a custom domain email idh one

@hoglet-33 @banaxi-tech @ggufguy btw supermicro takes a loooooooong time to schedule it btw i got mine after like 3 month lol

@Banaxi-Tech
here are the steps (original):

1. go to namecheap.com
2. look for a domain (probably something related to bananamind, and either get a TLD of .com .co .ai .dev or .net those are probably the best options, any will be fine though)
3. buy the domain and in checkout say you want to buy a mailbox or a few
4. go to firebase and add your domain to your website, unless firebase makes you pay to use custom then just migrate the website to netlify.com that what i use
5. go through the setup usually they will tell you what you need to add, if it gets too confusing just ask AI thats what i had to do and it works fine
6. go to your namecheap dashboard and say you want to create one of the mailboxes and name it something like banaxi@domain.com or something thats not too weird (dont use numbers or special characters though)
7. namecheap will tell you some stuff and add it to your domain thing in firebase or netlify
8. wait for certificates to propagate for 24 hours before doing anything
9. test it out, send an email to this new domain email
10. done! hopefully

(AI)
Here’s how to get a custom domain and email:

1. Go to Namecheap and search for a domain you like. I’d recommend something related to BananaMind. For the ending, .com, .co, .ai, .dev, or .net are all good options, but you can use basically any TLD you want.
2. Buy the domain. During checkout, you can also add Namecheap Private Email. Choose however many mailboxes you think you’ll need you can always add more later.
3. Once you own the domain, connect it to your website. If you’re using (I assume you are) Firebase, go into your hosting settings and add the custom domain. Firebase should give you DNS records that need to be added to your domain.
4. If Firebase ends up being annoying for custom domains, you can also move the site to Netlify. That’s what I use, and custom domains are pretty straightforward there.
5. Go to your Namecheap dashboard and open the DNS settings for your domain. Add the DNS records that Firebase or Netlify gives you. They’ll usually tell you exactly what record type, host/name, and value to enter.
6. Next, set up your email. Go to Namecheap Private Email and create your mailbox. Something simple and professional like banaxi@yourdomain.com, hello@yourdomain.com, or contact@yourdomain.com would work well. I’d avoid unnecessary numbers or special characters.
7. Make sure the email DNS records are set up too. Namecheap may configure these automatically depending on how your DNS is set up. Otherwise, it should tell you which MX and other DNS records need to be added.
8. Give DNS and SSL some time to update. Some changes happen within minutes, while others can take several hours or occasionally up to 24–48 hours. You don't necessarily have to wait the full 24 hours before testing it.
9. Test everything. Open the domain and make sure the website loads over HTTPS. Then send an email from another email address to your new domain email and reply from it to make sure sending and receiving both work.
10. Done! You should now have your own domain, website, and custom email.
    If any of the DNS stuff gets confusing, just send an AI a screenshot or copy the DNS records that Firebase/Netlify/Namecheap is asking for. That’s what I did when setting mine up, and it makes the DNS part a lot easier.

@Hoglet-33 why did u have to do a AI before and after lol

@Bc-AI mostly because i want to see what people like more, and maybe one instruction set is better than the other or one has something that the other doesnt and so on

Ok nice! @Hoglet-33 did u get the super micro?

@Hoglet-33 have you done anything cool with omnirush yet? I’m making it turn Raspberry pi OS into IOS 26 style lol

@Hoglet-33 since when did u have 2xRtx 6000?

@Hoglet-33 since when did u have 2xRtx 6000?

@Bc-AI here is how i got the GPUs on my profile
2 T4s from Kaggle, 1 T4 from Colab, 2 rtx 6000s from molab, 1 rtx 3060 just because i rent that one a lot from vast.ai

@Hoglet-33 have you done anything cool with omnirush yet? I’m making it turn Raspberry pi OS into IOS 26 style lol

no, 5M tokens is kinda not that much and i dont really feel like having it only be through either another desktop app or an API. and i go through about 10M tokens a day on average according to my usage reports that ChatGPT gives me, and with however i do stuff i almost never hit max usage, probably because the main model im using is GPT-6.1-Sol on medium and i only use Astra for 3D modeling and now game engine use soo i havnt used it yet

Ok nice! @Hoglet-33 did u get the super micro?

I'm deciding what to get I will probably schedule for 8x L40S because i can get that within six weeks and i dont really train that big of models anyway so for 5 days its probbaly enough, after all i trained pebble 10m, 25m and 50m all in 7 days on 1 rtx 3090 so i think 8x L40S will be enough for all of those sizes of Pebble 1.5 plus maybe a few bigger ones

Ok nice! @Hoglet-33 did u get the super micro?

I'm deciding what to get I will probably schedule for 8x L40S because i can get that within six weeks and i dont really train that big of models anyway so for 5 days its probbaly enough, after all i trained pebble 10m, 25m and 50m all in 7 days on 1 rtx 3090 so i think 8x L40S will be enough for all of those sizes of Pebble 1.5 plus maybe a few bigger ones

@Hoglet-33 i got a 8xB300 after 😂

@Hoglet-33 have you done anything cool with omnirush yet? I’m making it turn Raspberry pi OS into IOS 26 style lol

no, 5M tokens is kinda not that much and i dont really feel like having it only be through either another desktop app or an API. and i go through about 10M tokens a day on average according to my usage reports that ChatGPT gives me, and with however i do stuff i almost never hit max usage, probably because the main model im using is GPT-6.1-Sol on medium and i only use Astra for 3D modeling and now game engine use soo i havnt used it yet

@Hoglet-33 u also make games? I used Claude sonnet 5.5 at medium to make a ICBM simulator for fun

YOOOOO @Bc-AI BananaMind 3 750M A50M IS TRAINING!!!

@Banaxi-Tech noice 750M total where it stops being a toy and starts being gee this is helpful

@Banaxi-Tech u can go supermicro and get free 8xL40S system i checked next available to 19th of oct u can use 4 days

50M Active and DSA so it works

I will make a 2 or 4B version of T2.1

I got 8xH200😂🤣

I requested for 8xB300 but I’ll have to wait a while probably

YESS 8xL40s

WAIT OMG I REALIZED I DONT NEED CUSTOM DOMAIN, firefox have this email forwarding at @mozmail .com and ITS NOT DETECTED!!!

oh nah it needs match email bruh

NVM I set https://mozmail.com IT WORKS 8xL40S

@Banaxi-Tech wdym match email?

I think the only reason I got the 8xH200 was cuz I used my school email and my school is uh like kinda good

And I waited 3 month

I set EMail as a @mozmail .com and IT WORKED

@Banaxi-Tech nice! I think u can also try for 8xH200 lol

@Banaxi-Tech what did sea wolf ai do to u? It seems like u have beef

their models suck and read this https://huggingface.co/FINAL-Bench/Darwin-180B-RSI/discussions/1 https://huggingface.co/FINAL-Bench/Darwin-180B-RSI/discussions/7 and all content is AI generated

YOOOOO @Bc-AI BananaMind 3 750M A50M IS TRAINING!!!

It's first ever MoE >500M params?

@Bc-AI you only need a custom email for when you need to reserve the bigger GPU sets, so all of the ones that say request instead of schedule, i still my biz email though, i get my 8x L40S on the 12th

NVM I set https://mozmail.com IT WORKS 8xL40S

Bro 😎 it's peak

Shhh! 🤫

YOOOOO @Bc-AI BananaMind 3 750M A50M IS TRAINING!!!

@Banaxi-Tech AI SLOP BATTLE

Lol

Internet issue .