llm-quantization-complete-guide.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[],"dockerImageVersionId":30673,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Quantization\nQuantization is a technique to reduce the computational and memory costs of running inference by representing the weights and activations with low-precision data types like 8-bit integer (int8) instead of the usual 32-bit floating point (float32).\n\nReducing the number of bits means the resulting model requires less memory storage, consumes less energy (in theory), and operations like matrix multiplication can be performed much faster with integer arithmetic. It also allows to run models on embedded devices, which sometimes only support integer data types.\n\n![](https://miro.medium.com/v2/resize:fit:1400/1*hWIaIAQ7GWbrjfbaoUoYxw.jpeg)\n\n**The two most common quantization cases are float32 -> float16 and float32 -> int8**\n\n## Quantization to float16\nPerforming quantization to go from float32 to float16 is quite straightforward since both data types follow the same representation scheme. The questions to ask yourself when quantizing an operation to float16 are:\n\n- Does my operation have a float16 implementation?\n- Does my hardware suport float16?\n- Is my operation sensitive to lower precision? For instance the value of epsilon in LayerNorm is usually very small (~ 1e-12), but the smallest representable value in float16 is ~ 6e-5, this can cause NaN issues. The same applies for big values.\n\n## Quantization to int8\nPerforming quantization to go from float32 to int8 is more tricky. Only 256 values can be represented in int8, while float32 can represent a very wide range of values. The idea is to find the best way to project our range [a, b] of float32 values to the int8 space.\n\nLet’s consider a float x in [a, b], then we can write the following quantization scheme, also called the affine quantization scheme:\n\n`x_q = round(x/S + Z)`\n\n- x_q is the quantized int8 value associated to x\n- S is the scale, and is a positive float32\n- Z is called the zero-point, it is the int8 value corresponding to the value 0 in the float32 realm. This is important to be able to represent exactly the value 0 because it is used everywhere throughout machine learning models.\n\nAnd float32 values outside of the [a, b] range are clipped to the closest representable value, so for any floating-point number x:\n\n`x_q = clip(round(x/S + Z), round(a/S + Z), round(b/S + Z))`\n\nSo, Quantization in Machine Learning (ML) is the process of converting data in FP32 (floating point 32 bits) to a smaller precision like INT8 (Integer 8 bit) and perform all critical operations like Convolution in INT8 and at the end, convert the lower precision output to higher precision in FP32.\n\n![](https://miro.medium.com/v2/resize:fit:1024/1*UC2UOr0_TYFKJegNPtSEcQ.png)","metadata":{}},{"cell_type":"markdown","source":"## Questions are:\n- How accuracy is recovered in Quantization?\n- How doing calculation in INT8 or FP32 gives the same result?","metadata":{}},{"cell_type":"markdown","source":"## Problem with Low Precision Formats?\n\nAs per the current state of research, we are struggling to maintain accuracy with INT4 and INT1 and the performance improvement with INT32 oe FP16 is not significant.\n\n- INT1 is an integer data type with only 1 bit, meaning it can represent values of 0 and 1 only.\n- INT4 is an integer data type with only 1 bit, meaning it can represent values of 0 and 15 only.\n- INT8 is an integer data type with only 1 bit, meaning it can represent values of -127 and 128 only.\n\nINT1 and INT4 loose information while quantization and also it does not capture direction because of positive values only and not negative values, while doing clipping more information got lost as well, thats why INT8 is preferred.\n\n\n### **The most popular choice is: INT8**\n\nWhen we are doing calculations in a particular datatype (say INT8), we need another structure with a datatype which can hold the result such that it handles overflow. This is known as accumulation data type. For, FP32, accumulation is FP32 but for INT8, accumulation is INT32.\n\nThis table gives you the idea of the reduction in data size and increase of mathematical power depending on the data type:\n\n| Data Type | Accumulation | Math Power | Data Size Reduced |\n|-----------|--------------|------------|-------------------|\n| FP32      | FP32         | 1X         | 1X                |\n| FP16      | FP16         | 8X         | 2X                |\n| INT4      | INT32        | 16X        | 4X                |\n| INT4      | INT32        | 32X        | 8X                |\n| INT1      | INT32        | 128X       | 32X               |\n\n\n### There are two basic operations in Quantization:\n\n- #### **Quantize**: Convert data to lower precision like INT8\n- #### **Dequantize**: Convert data to higher precision like FP32","metadata":{}},{"cell_type":"markdown","source":"## Quantize\n\nWe have to convert a data of range [A1, A2] to the range of the B bit Integer (INT8 in our case).\n\nHence, the problem is to map all elements in the range [A1, A2] to the range [-(2^B), (2^B-1)]. Elements outside the range of [A1, A2] will be clipped to the nearest bound.\n\nThere are two main types of Range mapping in Quantization:\n\n- Affine quantization\n- Scale quantization\n\n![](https://miro.medium.com/v2/resize:fit:1400/1*c18I2HGMvv6ijsE_dwGkDg.png)\n\n### Affine Quantization\nIn Affine Quantization, the parameters s and z are as follows:\n\n`s = (2^B + 1)/(A1-A2)`\n\n`z = -(ROUND(A2 * s)) - 2^(B-1)`\n\nFor INT8, s and z are as follows:\n\n`s = (255)/(A1-A2)`\n\n`z = -(ROUND(A2 * s)) - 128`\n\nOnce you convert all the input data using the above equation, we will get a quantized data. In this data, some values may be out of range. To bring it into range, we need another operation \"Clip\" to map all data outside the range to come within the range.\n\nThe Clip operation is as follows:\n\n`clip(x, l, u) = x   ... if x is within [l, u]`\n\n`clip(x, l, u) = l   ... if x < l`\n\n`clip(x, l, u) = u   ... if x > u`\n\nIn the above equation, l is the lower limit in the quantization range while u is the upper limit in the quantization range.\n\nSo, the overall equation for Quantization in Affine Quantization is:\n\n\n`x_quantize = quantize(x, b, s, z)`\n\n`x_quantize = clip(round(s * x + z), −2^(B−1), 2^(B−1) − 1)`\n\nFor dequantization, the equation in Affine Quantization is:\n\n`x_dequantize = dequantize(x_quantize, s, z) = (x_quantize − z) / s`\n\n![](https://apple.github.io/coremltools/docs-guides/_images/quantization-technique.png)","metadata":{}},{"cell_type":"markdown","source":"### Scale quantization\nThe difference in Scale Quantization (in comparison to Affine Quantization) is that in this case, the zero point (z) is set to 0 and does not play a role in the equations. We use the scale factor (s) in the calculations of Scale Quantization.\n\nWe use the following equation:\n\n`F(x) = s.x`\n\nThere are many variants of Scale Quantization and the simpliest is Symmetric Quantization. In this, the resultant range is symmetric. For INT8, the range will be [-127, 127]. Note that we are not considering -128 in the calculations.\n\nHence, in this, we will quantize a data from range [-A1, A1] to [-(2^(B-1), 2^(B-1)]. The equations for Quantization will be:\n\n`s = (2^(B - 1) − 1) / A1`\n\nNote, s is the scale factor.\n\nThe overall equation is:\n\n`x_quantize = quantize(x, B, s)` \n\n`x_quantize = clip(round(s * x), −2^(B - 1) + 1, 2^(B - 1) − 1)`\n\nThe equation for dequantization will be:\n\n`x_dequantize = dequantize(x_quantize, s) = x_quantize / s`\n\nThe input data is multi-dimensional and to quantize it, we can use the same scale value and zero point for the entire data or these parameters can be different for each 1D data in each dimension or different for each dimension.\n\n![](https://developer-blogs.nvidia.com/wp-content/uploads/2021/07/8-bit-signed-integer-quantization.png)\n\nThis process of grouping the data for the quantization and dequantization process is known as Quantization Granularity.\n\nThe common choice of Quantization Granularity are:\n\n- Per channel for 3D input\n- Per row or Per column for 2D input","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}