Posted inArtificial Intelligence Python PyTorch Web Dev
Offload LLM using Facebook FairScale
What is Fairscale FairScale is an open-source library from Facebook. It help researchers and engineers…
Posted inArtificial Intelligence Python PyTorch
DeepSpeed API ZeRO Offload
The biggest problem in Deep learning and training large language models (LLMs) is the limit…
Posted inArtificial Intelligence Python PyTorch
Offload LLM custom back propagation
What is Model Offloading? Model offloading, as described in the DeepSpeed ZeRO's paper, refers to…
Posted inArtificial Intelligence Python PyTorch
NanoGPT – My takes
NanoGPT is develop by Andrew Karpathy, a Lead engineer from Tesla. The simplest, fastest repository…
Visual Studio Code debug settings
Include environment { // Use IntelliSense to learn about possible attributes. // Hover to view…
Posted inPython
Useful Anaconda commands
Create environment If you're juggling multiple projects like I am, keeping your packages and dependencies…
Posted inArtificial Intelligence Python
Tensorboard summary writer 101
Tensorboard is a very convenient way to record a neural network's weights and parameters while…
Posted inArtificial Intelligence Python
Create artificial language for LLM incremental training
Brief Explanation on Transformer Model's Query, Key, and Value Concept Very loosely speaking, the Query,…
Posted inArtificial Intelligence Matplotlib Python
How to plot weights animation of a LLM for debugging
So, I want to quickly plot a layer of weights changes overtime during training. If…