Scaling Laws

Quick notes around Chinchilla Scaling Law/ Limits & beyond for DeepLearning and LLMs.

Factors

• Model size (N)
• Dataset size (D)
• Training Cost (aka Compute) (C)
• Test Cross-entropy loss (L)

The intuitive way,

• Larger data will need a larger model, and have higher training cost. In other words, N, D, C all increase together, not necessarily linearly, could be exponential, log-linear, etc.
• Likewise Loss is likely to increase for larger datasets. So an inverse relationship between L & D (& the rest).
• Tying them into equations would be some constants (scaling, exponential, alpha, beta, etc), unknown for now (identified later).
  

Beyond common sense, the theoretical foundations linking the factors aren't available right now. Perhaps the nature of the problem is it's hard (NP).

The next best thing then, is to somehow work out the relationships/ bounds empirically. To work with existing Deep Learning models, LLMs, etc using large data sets spanning TB/ PB of data, Trillions of parameters, etc using large compute budget cumulatively spanning years.

Papers by Hestness & Narang, Kaplan, Chinchilla are all attempts along the empirical route. So are more recent papers like Mosaic, DeepSeek, MoE, Llam3, Microsoft among many others. 

Key take away being,

• The scale & bounds are getting larger over time. 
• Models from a couple of years back, are found to be grossly under-trained in terms of volumes of training data used. They should have been trained on an order of magnitude larger training data for an optimal training, without risk of overfitting.
• Conversely, the previously used data volumes are suited to much smaller models (SLMs), with inference capabilities similar to those older LLMs.

References

• https://en.wikipedia.org/wiki/Neural_scaling_law
• https://lifearchitect.ai/chinchilla/
• https://medium.com/@raniahossam/chinchilla-scaling-laws-for-large-language-models-llms-40c434e4e1c1
• https://bigscience.huggingface.co/blog/what-language-model-to-train-if-you-have-two-million-gpu-hours
• https://medium.com/nlplanet/two-minutes-nlp-scaling-laws-for-neural-language-models-add6061aece7
• https://lifearchitect.ai/the-sky-is-bigger/

Diffusion Models

Diffusion

•     Forward, Backward (Learning), Sampling (Random)    
•     Continous Diffusion
•     VAE, Denoising Autoencoder
•     Markov Chains
•     U-Net
•     DALL-E (OpenAI), Stable Diffusion,
•     Imagen, Muse, VEO (Google)
•     LLaDa, Mercury Coder (Inception)

Non-equilibrium Thermodynamics

•     Langevin dynamics
•     Thermodynamic Equilibrium - Boltzmann Distribution
•     Wiener Process - Multidimensional Brownian Motion
•     Energy Based Models

Gaussian Noise

•     Denoising
•     Noise/ Variance Schedule
•     Derivation by Reparameterization

Variational Inference    

•     Denoising Diffusion Probabilistic Model (DDPM)
•     Noise Prediction Networks    
•     Denoising Diffusion Implicit Model (DDIM)

Loss Functions

•     Variational Lower Bound (VLB)
•     Evidence Lower Bound (ELBO)
•     Kullback-Leibler divergence (KL divergence)
•     Mean Squared Error (MSE)

Score Based Generative Model

•     Annealing
•     Noise conditional score network (NCSN)
•     Equivalence: DDPM and Score BBased Generative Models

Conditional (Guided) Generation

•     Classifier Guidance    
•     Classifier Free Guidance (CFG)

Latent Varible Generative Model

•     Latent Diffusion Model (LDM)
•     Lower Dimension (Latent) Space

References:

• https://en.wikipedia.org/wiki/Diffusion_model
• https://www.assemblyai.com/blog/diffusion-models-for-machine-learning-introduction
• https://www.ibm.com/think/topics/diffusion-models
• https://hackernoon.com/what-is-a-diffusion-llm-and-why-does-it-matter
• Large Language Diffusion Models (LLaDA): https://arxiv.org/abs/2502.09992
  

Mechanistic Interpretability

• Clearer better understanding of Neural Networks working (white box).
• Strong grounds for Superposition: n-dimensions (neurons) represent more than n-features

References

• https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J#z=EuO4CLwSIzX7AEZA1ZOsnwwF
• https://www.neelnanda.io/mechanistic-interpretability/glossary
• https://transformer-circuits.pub/2022/toy_model/index.html
• https://www.anthropic.com/research/superposition-memorization-and-double-descent
• https://transformer-circuits.pub/2023/toy-double-descent/index.html 

State Space Models

• Vector Space of States (of the System)
  
• Alt. to Transformers, reducible to one another 

 

        (Image source: https://en.wikipedia.org/wiki/State-space_representation)

References

• https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state
• https://huggingface.co/blog/lbourdois/ssm-2022
• https://huggingface.co/blog/lbourdois/get-on-the-ssm-train
• https://en.wikipedia.org/wiki/State-space_representation

Learning Deep

Head out straight to KdNugget's Top 20 Deep Learning Papers of 2018. Has a good listing of research publications spanning over the last 4-5 years. You could further go on to read the papers referred to within these papers & then those referred to in the referred papers & so on for some really deep learning!