TorchSharp NanoGpt

[GeorgeS2019]

https://github.com/biegehydra/NanoGptDotnet/blob/master/src/NanoGpt/NanoGpt.cs

[ejhg]

Thank you @GeorgeS2019 !

I can almost feel the AGI now!

[travisjj]

This is a nicely implemented version of Karpathy's first setup of reproducing GPT, but does not include the updates from GPT-2 such as some of the CUDA performance improvements, the checkmarks, and some minor tweaks to preactivation.

[GeorgeS2019]

@travisjj check these:
https://github.com/biegehydra/NanoGptDotnet
https://github.com/PentoreXannaci/NanoGPTNet

[ejhg]

Along these lines, I have been playing a lot with o1-preview. I copy and paste large chunks of transformer related python code and ask it to translate it to C#. It is mind-blowingly effective. The tasks I give it are particularly challenging because I'm asking it to rewrite llama from scratch without dependencies, or I ask it to unflatten tensor operations a certain way, etc. This approach could be effective in bringing over any of Karpathy's implementations to C#/TorchSharp.

[ejhg]

Example: I couldn't find any C# code to train a Llama model from a quick google search, so I asked it to write it for me from one of Karparthy's repos. (I still have to test it...)

https://chatgpt.com/share/672d06a0-e6fc-8009-b29a-408e0575a4bd

[travisjj]

Hi @GeorgeS2019 , one of those (NanoGPTNet) is a simple nano-gpt.com wrapper, which doesn't really include any torchsharp code unfortunately. Pentore is also from the older version released by Karpathy. A great indicator is the lack of GELU, but there are more subtleties as well such as all of the performance increases he gains in the subsequent version; for example with F.scaled_dot_product_attention inside of the CausalSelfAttention's forward pass.

I do have a working version I could share, perhaps with some polish... I am used to publishing c# enterprise code and this code isn't commented and the call to start the training itself is rather chicken scratch. I haven't played around too much with it, but on an Azure A100 it hit 600k tokens per second, and 200k tokens processed per second on gradient accumulation.