Post
1351
I tried to teach a network to write using nothing but reward. No pretraining, no data, random weights. It worked twice.
Both wins were fake.
Test 1, copy a fixed sentence: solved in 1,639 episodes. Then I checked why. The per-position reward splits 12^12 into twelve independent 12-armed bandits. It doesn't guide the search, it deletes it. Transfer to a target sharing no positions was slower than starting over.
Test 2, a grammar judged by a hand-written parser: 99.9% grammatical. Then I forced the antecedent and measured the consequent. P(noun agrees | determiner) = 0.333. Exactly 2 determiners out of 6. It learned no rule at all. It found an all-plural sublanguage where agreement is vacuously true.
Then the one that stung: an untrained network plus a filter beats 20,000 episodes of REINFORCE on both validity and diversity. Nobody runs that baseline because it looks too stupid to bother with.
None of this is new. Reward shaping moving the optimum: Ng, Harada & Russell, 1999. The variance decomposition that retires the word "sparse": Fisher's ANOVA.
The real lesson:
A high score on a verifier you wrote yourself measures your specification, not the agent. It finds the corner of the output space where your constraint is vacuous, and it looks like success from every angle except the one you forgot to check.
Code, 8 figures, and the notebook with every refuted hypothesis 👇
🔗 https://huggingface.co/blog/RDTvlokip/teaching-a-network-to-write-with-reward-only
💻 https://github.com/RDTvlokip/RDTRL
Both wins were fake.
Test 1, copy a fixed sentence: solved in 1,639 episodes. Then I checked why. The per-position reward splits 12^12 into twelve independent 12-armed bandits. It doesn't guide the search, it deletes it. Transfer to a target sharing no positions was slower than starting over.
Test 2, a grammar judged by a hand-written parser: 99.9% grammatical. Then I forced the antecedent and measured the consequent. P(noun agrees | determiner) = 0.333. Exactly 2 determiners out of 6. It learned no rule at all. It found an all-plural sublanguage where agreement is vacuously true.
Then the one that stung: an untrained network plus a filter beats 20,000 episodes of REINFORCE on both validity and diversity. Nobody runs that baseline because it looks too stupid to bother with.
None of this is new. Reward shaping moving the optimum: Ng, Harada & Russell, 1999. The variance decomposition that retires the word "sparse": Fisher's ANOVA.
The real lesson:
A high score on a verifier you wrote yourself measures your specification, not the agent. It finds the corner of the output space where your constraint is vacuous, and it looks like success from every angle except the one you forgot to check.
Code, 8 figures, and the notebook with every refuted hypothesis 👇
🔗 https://huggingface.co/blog/RDTvlokip/teaching-a-network-to-write-with-reward-only
💻 https://github.com/RDTvlokip/RDTRL