The Big Book of LLM

A book by Damien Benveniste of AIEdge. Though a work in progress, chapters 2 - 4 available for preview are fantastic. 

Look forward to a paperback edition, which I certainly hope to own...

Mozilla.ai

Mozilla pedigree, AI focus, Open-source, Dev oriented.

Blueprint Hub: Mozilla.ai's Hub of open-source templtaized customizable AI solutions for developers.

Lumigator: Platform for model evaluation and selection. Consists a Python FastAPI backend for AI lifecycle management & capturing workflow data useful for evaluation.

Streamlit

Streamlit is a web wrapper for Data Science projects in pure Python. It's a lightweight, simple, rapid prototyping web app framework for sharing scripts.

• https://streamlit.io/playground
• https://www.restack.io/docs/streamlit-knowledge-streamlit-vs-flask-vs-django
• https://docs.streamlit.io/develop/concepts/architecture/architecture
• https://docs.snowflake.com/en/developer-guide/streamlit/about-streamlit

Scaling Laws

Quick notes around Chinchilla Scaling Law/ Limits & beyond for DeepLearning and LLMs.

Factors

• Model size (N)
• Dataset size (D)
• Training Cost (aka Compute) (C)
• Test Cross-entropy loss (L)

The intuitive way,

• Larger data will need a larger model, and have higher training cost. In other words, N, D, C all increase together, not necessarily linearly, could be exponential, log-linear, etc.
• Likewise Loss is likely to increase for larger datasets. So an inverse relationship between L & D (& the rest).
• Tying them into equations would be some constants (scaling, exponential, alpha, beta, etc), unknown for now (identified later).
  

Beyond common sense, the theoretical foundations linking the factors aren't available right now. Perhaps the nature of the problem is it's hard (NP).

The next best thing then, is to somehow work out the relationships/ bounds empirically. To work with existing Deep Learning models, LLMs, etc using large data sets spanning TB/ PB of data, Trillions of parameters, etc using large compute budget cumulatively spanning years.

Papers by Hestness & Narang, Kaplan, Chinchilla are all attempts along the empirical route. So are more recent papers like Mosaic, DeepSeek, MoE, Llam3, Microsoft among many others. 

Key take away being,

• The scale & bounds are getting larger over time. 
• Models from a couple of years back, are found to be grossly under-trained in terms of volumes of training data used. They should have been trained on an order of magnitude larger training data for an optimal training, without risk of overfitting.
• Conversely, the previously used data volumes are suited to much smaller models (SLMs), with inference capabilities similar to those older LLMs.

References

• https://en.wikipedia.org/wiki/Neural_scaling_law
• https://lifearchitect.ai/chinchilla/
• https://medium.com/@raniahossam/chinchilla-scaling-laws-for-large-language-models-llms-40c434e4e1c1
• https://bigscience.huggingface.co/blog/what-language-model-to-train-if-you-have-two-million-gpu-hours
• https://medium.com/nlplanet/two-minutes-nlp-scaling-laws-for-neural-language-models-add6061aece7
• https://lifearchitect.ai/the-sky-is-bigger/

Diffusion Models

Diffusion

•     Forward, Backward (Learning), Sampling (Random)    
•     Continous Diffusion
•     VAE, Denoising Autoencoder
•     Markov Chains
•     U-Net
•     DALL-E (OpenAI), Stable Diffusion,
•     Imagen, Muse, VEO (Google)
•     LLaDa, Mercury Coder (Inception)

Non-equilibrium Thermodynamics

•     Langevin dynamics
•     Thermodynamic Equilibrium - Boltzmann Distribution
•     Wiener Process - Multidimensional Brownian Motion
•     Energy Based Models

Gaussian Noise

•     Denoising
•     Noise/ Variance Schedule
•     Derivation by Reparameterization

Variational Inference    

•     Denoising Diffusion Probabilistic Model (DDPM)
•     Noise Prediction Networks    
•     Denoising Diffusion Implicit Model (DDIM)

Loss Functions

•     Variational Lower Bound (VLB)
•     Evidence Lower Bound (ELBO)
•     Kullback-Leibler divergence (KL divergence)
•     Mean Squared Error (MSE)

Score Based Generative Model

•     Annealing
•     Noise conditional score network (NCSN)
•     Equivalence: DDPM and Score BBased Generative Models

Conditional (Guided) Generation

•     Classifier Guidance    
•     Classifier Free Guidance (CFG)

Latent Varible Generative Model

•     Latent Diffusion Model (LDM)
•     Lower Dimension (Latent) Space

References:

• https://en.wikipedia.org/wiki/Diffusion_model
• https://www.assemblyai.com/blog/diffusion-models-for-machine-learning-introduction
• https://www.ibm.com/think/topics/diffusion-models
• https://hackernoon.com/what-is-a-diffusion-llm-and-why-does-it-matter
• Large Language Diffusion Models (LLaDA): https://arxiv.org/abs/2502.09992
  

Mechanistic Interpretability

• Clearer better understanding of Neural Networks working (white box).
• Strong grounds for Superposition: n-dimensions (neurons) represent more than n-features

References

• https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J#z=EuO4CLwSIzX7AEZA1ZOsnwwF
• https://www.neelnanda.io/mechanistic-interpretability/glossary
• https://transformer-circuits.pub/2022/toy_model/index.html
• https://www.anthropic.com/research/superposition-memorization-and-double-descent
• https://transformer-circuits.pub/2023/toy-double-descent/index.html 

State Space Models

• Vector Space of States (of the System)
  
• Alt. to Transformers, reducible to one another 

 

        (Image source: https://en.wikipedia.org/wiki/State-space_representation)

References

• https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state
• https://huggingface.co/blog/lbourdois/ssm-2022
• https://huggingface.co/blog/lbourdois/get-on-the-ssm-train
• https://en.wikipedia.org/wiki/State-space_representation

OCR

Tesseract that "quirky command-line tool that does an outstanding job" (credit A. Kay) truly works well.  Give it a shot whenever you get the opportunity.

Sample commands for ref:

•    tesseract IMG-1.jpg IMG-1 --psm 4

•    tesseract -l eng IMG-2.jpg b1