Showing posts with label Lucence. Show all posts
Showing posts with label Lucence. Show all posts

Thursday, April 17, 2025

On Quantization

  • Speed vs Accuracy trade off.
  • Reduce costs on storage, compute, operations .
  • Speed up output generation, inference, etc.
  • Work with lower precision data.
  • Cast/ map data from Int32, Float32, etc 32-bit or higher precision to lower precision data types such as 16-bit Brain Float (BFloat16) or 4-bit (NFloat)/ int4 or int8, etc.
    • East mapping Float32 (1-bit Sign, 7-bit Exponent, 23-bit Mantissa) => BFloat16 (1-bit Sign, 7-bit Exponent, 7-bit Mantissa). Just discard the higher 16-bits of mantissa. No overflow!
    • Straightforward mapping work out max, min, data distribution, mean, variance, etc & then sub-divide into equally sized buckets based on bit size of the lower precision data type. E.g int4 (4-bit) => 2^4 = 16 buckets. 
    • Handle outliers, data skew which can mess up the mapping, yet lead to loss of useful info if discarded randomly.
    • Work out Bounds wrt Loss of Accuracy.

LLMs, AI/ ML side:

  • https://newsletter.theaiedge.io/p/reduce-ai-model-operational-costs

Lucene, Search side:

  • https://www.elastic.co/search-labs/blog/scalar-quantization-101
  • https://www.elastic.co/search-labs/blog/scalar-quantization-in-lucene