Hamming code
41226
225493091
2008-07-14T00:00:51Z
168.18.163.141
/* Hamming codes with additional parity (SECDED) */ then
In [[telecommunication]], a '''Hamming code''' is a [[linear code|linear]] [[error-correcting code]] named after its inventor, [[Richard Hamming]]. Hamming codes can detect and correct single-bit errors. In other words, the [[Hamming distance]] between the transmitted and received code-words must be zero or one for reliable communication. Alternatively, it can detect (but not correct) up to two simultaneous bit errors.
In contrast, the simple [[parity]] code cannot correct errors, nor can it be used to detect more than one error (such as where two bits are transposed).
In [[mathematics|mathematical]] terms, Hamming codes are a class of binary linear codes. For each integer <math>m>1</math> there is a code with parameters: <math>[2^m-1, 2^m-m-1, 3]</math>. The [[parity-check matrix]] of a Hamming code is constructed by listing all columns of length <math>m</math> that are pair-wise independent.
Because of the simplicity of Hamming codes, they are widely used in computer memory ([[RAM]]). In particular, a single-error-correcting ''and'' double-error-detecting variant commonly referred to as '''SECDED'''.
== History ==
Hamming worked at [[Bell Labs]] in the 1940s on the [[Bell Model V]] computer, an [[electromechanical]] relay-based machine with cycle times in seconds. Input was fed in on [[punch card]]s, which would invariably have read errors. During weekdays, special code would find errors and flash lights so the operators could correct the problem. During after-hours periods and on weekends, when there were no operators, the machine simply moved on to the next job.
Hamming worked on weekends, and grew increasingly frustrated with having to restart his programs from scratch due to the unreliability of the card reader. Over the next few years he worked on the problem of error-correction, developing an increasingly powerful array of algorithms. In 1950 he published what is now known as Hamming Code, which remains in use in some applications today.
=== Codes predating Hamming ===
A number of simple error-detecting codes were used before Hamming codes, but none were as effective as Hamming codes in the same overhead of space.
==== Parity ====
{{main|Parity bit}}
Parity adds a single [[binary digit|bit]] that indicates whether the number of [[1 (number)|1]] bits in the preceding data was [[even number|even]] or [[odd number|odd]]. If a single bit is changed in transmission, the message will change parity and the error can be detected at this point. (Note that the bit that changed may have been the parity bit itself!) The most common convention is that a parity value of '''1''' indicates that there is an '''odd''' number of ones in the data, and a parity value of '''0''' indicates that there is an '''even''' number of ones in the data. In other words: The data and the parity bit '''together''' should contain an even number of 1s.
Parity checking is not very robust, since if the number of bits changed is even, the check bit will be valid and the error will not be detected. Moreover, parity does not indicate which bit contained the error, even when it can detect it. The data must be discarded entirely and re-transmitted from scratch. On a noisy transmission medium, a successful transmission could take a long time or may never occur. However, while the quality of parity checking is poor, since it uses only a single bit, this method results in the least overhead. Furthermore, parity checking does allow for the restoration of a missing bit when the missing bit is known.
====Two-out-of-five code====
{{main|Two-out-of-five code}}
In the 1940s Bell used a slightly more sophisticated [[M of n codes|m of n]] code known as the two-out-of-five code. This code ensured that every block of five bits (known as a ''5-block'') had exactly two 1s. The computer could tell there was an error if, in its input, there were not exactly two 1s in each block. Two-of-five was still only able to detect single bit errors; if one bit flipped to a 1 and another to a 0 in the same block, the two-of-five rule remained true and the error would go undiscovered.
====Repetition====
{{main|triple modular redundancy}}
Another code in use at the time repeated every data bit several times in order to ensure that it got through. For instance, if the data bit to be sent was a 1, an n=3 ''repetition code'' would send "111". If the three bits received were not identical, an error occurred. If the channel is clean enough, most of the time only one bit will change in each triple. Therefore, 001, 010, and 100 each correspond to a 0 bit, while 110, 101, and 011 correspond to a 1 bit, as though the bits counted as "votes" towards what the original bit was. A code with this ability to reconstruct the original message in the presence of errors is known as an ''error-correcting'' code.
Such codes cannot correctly repair all errors, however. In our example, if the channel flipped two bits and the receiver got "001", the system would detect the error, but conclude that the original bit was 0, which is incorrect. If we increase the number of times we duplicate each bit to four, we can detect all two-bit errors but can't correct them (the votes "tie"); at five, we can correct all two-bit errors, but not all three-bit errors.
Moreover, the repetition code is extremely inefficient, reducing throughput by three times in our original case, and the efficiency drops drastically as we increase the number of times each bit is duplicated in order to detect and correct more errors.
==Hamming codes==
If more error-correcting bits are included with a message, and if those bits can be arranged such that different incorrect bits produce different error results, then bad bits could be identified. In a 7-bit message, there are seven possible single bit errors, so three error control bits could potentially specify not only that an error occurred but also which bit caused the error.
Hamming studied the existing coding schemes, including two-of-five, and generalized their concepts. To start with he developed a [[wikt:nomenclature|nomenclature]] to describe the system, including the number of data bits and error-correction bits in a block. For instance, parity includes a single bit for any data word, so assuming [[ASCII]] words with 7-bits, Hamming described this as an ''(8,7)'' code, with eight bits in total, of which 7 are data. The repetition example would be ''(3,1)'', following the same logic. The ''information rate'' is the second number divided by the first, for our repetition example, 1/3.
Hamming also noticed the problems with flipping two or more bits, and described this as the "distance" (it is now called the ''[[Hamming distance]]'', after him). Parity has a distance of 2, as any two bit flips will be invisible. The (3,1) repetition has a distance of 3, as three bits need to be flipped in the same triple to obtain another code word with no visible errors. A (4,1) repetition (each bit is repeated four times) has a distance of 4, so flipping two bits can be detected, but not corrected. When three bits flip in the same group there can be situations where the code corrects towards the wrong code word.
Hamming was interested in two problems at once; increasing the distance as much as possible, while at the same time increasing the information rate as much as possible. During the 1940s he developed several encoding schemes that were dramatic improvements on existing codes. The key to all of his systems was to have the parity bits overlap, such that they managed to check each other as well as the data.
===General algorithm===
Although any number of algorithms can be created, the following general algorithm positions the parity bits at powers of two to ease calculation of which bit was flipped upon detection of incorrect parity.
#All bit positions that are powers of two are used as parity bits. (positions 1, 2, 4, 8, 16, 32, 64, etc.), see {{OEIS2C|id=A000079}} at the [[On-Line Encyclopedia of Integer Sequences]].
#All other bit positions are for the data to be encoded. (positions 3, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 17, etc.), see {{OEIS2C|id=A057716}} at the [[On-Line Encyclopedia of Integer Sequences]].
#Each parity bit calculates the parity for some of the bits in the code word. The position of the parity bit determines the sequence of bits that it alternately checks and skips.
#* Position 1 (n=1): skip 0 bit (0=n−1), check 1 bit (n), skip 1 bit (n), check 1 bit (n), skip 1 bit (n), etc. (1,3,5,7,9,11,13,15,...)
#* Position 2 (n=2): skip 1 bit (1=n−1), check 2 bits (n), skip 2 bits (n), check 2 bits (n), skip 2 bits (n), etc. (2,3,6,7,10,11,14,15,...)
#* Position 4 (n=4): skip 3 bits (3=n−1), check 4 bits (n), skip 4 bits (n), check 4 bits (n), skip 4 bits (n), etc. (4,5,6,7,12,13,14,15,20,21,22,23,...)
#* Position 8 (n=8): skip 7 bits (7=n−1), check 8 bits (n), skip 8 bits (n), check 8 bits (n), skip 8 bits (n), etc. (8-15,24-31,40-47,...)
#* Position 16 (n=16): skip 15 bits (15=n−1), check 16 bits (n), skip 16 bits (n), check 16 bits (n), skip 16 bits (n), etc. (16-31,48-63,80-95,...)
#* Position 32 (n=32): skip 31 bits (31=n−1), check 32 bits (n), skip 32 bits (n), check 32 bits (n), skip 32 bits (n), etc. (32-63,96-127,160-191,...)
#* General rule for position ''n'': skip ''n''−1 bits, check ''n'' bits, skip ''n'' bits, check ''n'' bits...
#* And so on.
In other words, the parity bit at position <math>2^k</math> checks bits in positions having bit k set in their binary representation. Conversely, for instance, bit 13, i.e. 1101<sub>(2)</sub>, is checked by bits 1000<sub>(2)</sub> = 8, 0100<sub>(2)</sub>=4 and 0001<sub>(2)</sub> = 1.
This general rule can be shown visually:
:{| class="wikitable" style="text-align:center;"
|-
!colspan="2"| Bit position
! 1 !! 2 !! 3 !! 4 !! 5 !! 6 !! 7 !! 8 !! 9 !! 10 !! 11 !! 12 !! 13 !! 14 !! 15 !! 16 !! 17 !! 18 !! 19 !! 20
|rowspan="7"| ...
|-
!colspan="2"| Encoded data bits
!style="background-color: #90FF90;"| p1
!style="background-color: #90FF90;"| p2 !! d1
!style="background-color: #90FF90;"| p3 !! d2 !! d3 !! d4
!style="background-color: #90FF90;"| p4 !! d5 !! d6 !! d7 !! d8 !! d9 !! d10 !! d11
!style="background-color: #90FF90;"| p5 !! d12 !! d13 !! d14 !! d15
|-
!rowspan="5"|Parity<br />bit<br />coverage
! p1
| X || || X || || X || || X || || X || || X || || X || || X || || X || || X ||
|-
! p2
| || X || X || || || X || X || || || X || X || || || X || X || || || X || X ||
|-
! p3
| || || || X || X || X || X || || || || || X || X || X || X || || || || || X
|-
! p4
| || || || || || || || X || X || X || X || X || X || X || X || || || || ||
|-
! p5
| || || || || || || || || || || || || || || || X || X || X || X || X
|}
Shown is only 20 encoded bits (5 parity, 15 data) but the pattern continues indefinitely. The key thing about Hamming Codes that can easily be seen from visual inspection is that any given bit has a unique parity bit coverage. For example, the only bit covered by p3 and p4 only is bit 12 (d8). It is this unique bit coverage that lets a Hamming Code correct any single bit error.
It is also how a two-bit error can be detected but not corrected. For example, if bits 1 (p1) & 2 (p2) were flipped then this would be confused with bit 3 (d1) being flipped since the parity bit coverage of bit 3 is the same as bits 1 & 2.
==Hamming codes with additional parity (SECDED)==
Hamming codes have a minimum distance of 3, which means that the code can detect and correct a single error, but a double bit error is indistinguishable from a different code with a single bit error. Thus, they can detect double-bit errors only if correction is not attempted.
By including an extra parity bit, it is possible to increase the minimum distance of the Hamming code to 4. This gives the code the ability to detect and correct a single error and at the same time detect (but not correct) a double error. (It could also be used to detect up to 3 errors but not correct any.)
This code system is popular in computer memory systems, where it is known as SECDED ("single error correction, double error detection"). Particularly popular is the (72,64) code, a truncated (127,120) Hamming code plus an additional parity bit, which has the same space overhead as a (9,8) parity code.
==Hamming(7,4) code==
[[Image:Hamming(7,4).svg|thumb|300px|Graphical depiction of the 4 data bits and 3 parity bits and which parity bits apply to which data bits]]
{{main|Hamming(7,4)}}
In [[1950]], Hamming introduced the (7,4) code.
It encodes 4 data bits into 7 bits by adding three parity bits.
Hamming(7,4) can detect and correct single-bit errors but can only detect double-bit errors.
== Construction of G and H ==
The matrix
<math>\mathbf{G} := \begin{pmatrix}
I_k | -A^T \\
\end{pmatrix}</math> is called a (Canonical) generator matrix of a linear (n,k) code,
and <math>\mathbf{H} := \begin{pmatrix}
A | I_{n-k} \\
\end{pmatrix}</math> is called a [[parity-check matrix]].
This is the construction of '''G''' and '''H''' in standard (or systematic) form. Regardless of form, '''G''' and '''H''' for linear block codes must satisfy
<math>\mathbf{G}\,\mathbf{H}^T = \mathbf{0}</math>, an all-zeros matrix [Moon, p.89].
Since (7,4,3)=(n,k,d)=[2<sup>m</sup> − 1, 2<sup>m</sup>−1-m, 3]. The [[parity-check matrix]] '''H''' of a Hamming code is constructed by listing all columns of length m that are pair-wise independent.
Thus '''H''' is a matrix whose right side is all of the nonzero n-tuples where order of the n-tuples in the columns of matrix does not matter. The left hand side is just the (n-k)-[[identity matrix]].
So '''G''' can be obtained from '''H''' by taking the transpose of the left hand side of H with the identity k-[[identity matrix]] on the right hand side.
The code [[generator matrix]] <math>\mathbf{G}</math> and the [[parity-check matrix]] <math>\mathbf{H}</math> are:
<math>\mathbf{H} := \begin{pmatrix}
1 & 1 & 0 & 1 & 1 & 0 & 0 \\
1 & 0 & 1 & 1 & 0 & 1 & 0 \\
0 & 1 & 1 & 1 & 0 & 0 & 1 \\
\end{pmatrix}_{3,7}</math >
and
<math>\mathbf{G} := \begin{pmatrix}
1 & 0 & 0 & 0 & 1 & 1 & 0 \\
0 & 1 & 0 & 0 & 1 & 0 & 1 \\
0 & 0 & 1 & 0 & 0 & 1 & 1 \\
0 & 0 & 0 & 1 & 1 & 1 & 1 \\
\end{pmatrix}_{4,7}.</math>
Finally, these matrices can be mutated into equivalent non-systematic codes by the following operations [Moon, p. 85]:
* Column permutations (swapping columns)
* Elementary row operations (replacing a row with a linear combination of rows)
== Encoding ==
Example
From the above matrix we have 2<sup>k</sup>=2<sup>4</sup>=16 codewords.
The codewords <math> \overrightarrow{x}</math> of this binary code can be obtained from <math>\overrightarrow{x}=\overrightarrow{a}G </math>. With <math>\overrightarrow{a}=a_1a_2a_3a_4 </math> with <math> a_i </math> exist in <math> F_2 </math> ( A field with two elements namely 0 and 1).
Thus the codewords are all the 4-tuples (k-tuples).
Therefore,
(1,0,1,1) gets encoded as (1,0,1,1,0,1,0).
==Hamming(8,4) code==
[[Image:Hamming(8,4).svg|thumb|300px|The same (7,4) example from above with an extra parity bit]]
The Hamming(7,4) can easily be extended to an (8,4) code by adding an extra parity bit on top of the (7,4) encoded word (''see [[Hamming(7,4)]]'').
This can be summed up with the revised matrices:
:<math>\mathbf{G} := \begin{pmatrix}
1 & 1 & 1 & 0 & 0 & 0 & 0 & 1\\
1 & 0 & 0 & 1 & 1 & 0 & 0 & 1\\
0 & 1 & 0 & 1 & 0 & 1 & 0 & 1\\
1 & 1 & 0 & 1 & 0 & 0 & 1 & 0
\end{pmatrix}_{8,4}</math>
and
:<math>
\mathbf{H} :=
\begin{pmatrix}
1 & 0 & 1 & 0 & 1 & 0 & 1 & 0\\
0 & 1 & 1 & 0 & 0 & 1 & 1 & 0\\
0 & 0 & 0 & 1 & 1 & 1 & 1 & 0\\
1 & 1 & 1 & 1 & 1 & 1 & 1 & 1
\end{pmatrix}_{4,8}
.</math>
<!-- the equations here before had two (2) errors--these satisfy G*H^T=0 -->
Note that H is not in standard form. To obtain G, elementary row operations can be used to obtain an equivalent matrix to H in systematic form:
:<math>
\mathbf{H} =
\left(\left.\begin{array}{cccc}
0 & 1 & 1 & 1\\
1 & 0 & 1 & 1\\
1 & 1 & 0 & 1\\
1 & 1 & 1 & 0\end{array}\right|\begin{array}{cccc}
1 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 1 & 0\\
0 & 0 & 0 & 1\end{array}\right)_{4,8}
.</math>
For example, the first row in this matrix is the sum of the second and third rows of H in non-systematic form. Using the systematic construction for Hamming codes from above, the matrix A is apparent and the systematic form of G is written as
:<math>
\mathbf{G} =
\left(\left.\begin{array}{cccc}
1 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 1 & 0\\
0 & 0 & 0 & 1\end{array}\right|\begin{array}{cccc}
0 & 1 & 1 & 1\\
1 & 0 & 1 & 1\\
1 & 1 & 0 & 1\\
1 & 1 & 1 & 0\end{array}\right)_{4,8}
.</math>
The non-systematic form of G can be row reduced (using elementary row operations) to match this matrix.
The addition of the fourth row effectively computes the sum of all the codeword bits (data and parity) as the fourth parity bit.
For example, <font color="blue">1011</font> is encoded into <font color="red">01</font><font color="blue">1</font><font color="red">0</font><font color="blue">011</font><font color="green">0</font> where blue digits are data; red digits are parity from the Hamming(7,4) code; and the green digit is the parity added by Hamming(8,4).
The green digit makes the parity of the (7,4) code even.
Finally, it can be shown that the minimum distance has increased from 3, as with the (7,4) code, to 4 with the (8,4) code. Therefor, the code can be defined as Hamming(8,4,4). <!-- if we need more proof, or a citation, let me know -->
<br clear="all" />
==Hamming(11,7) code==
{| align="right"
|-
| [[Image:Hamming(11,7).svg|thumb|300px|Graphical depiction of the 7 data bits and 4 parity bits and which parity bits apply to which data bits]]
|-
| [[Image:Hamming(11,7) as bits.svg|thumb|300px|Bit position of the data and parity bits]]
|-
| [[Image:Hamming(11,7) example 0110101.svg|thumb|300px|Mapping in the example data value. The parity of the red, yellow, green, and blue circles are even.]]
|-
| [[Image:Hamming(11,7) example 0110101 bit 11 error.svg|thumb|300px|A bit error on bit 11 causes bad parity in the red, yellow, and green circles]]
|}
Consider the 7-bit data word "0110101". To demonstrate how Hamming codes are calculated and used to detect an error, see the tables below. They use '''d''' to signify data bits and '''p''' to signify [[parity bit]]s.
Firstly the data bits are inserted into their appropriate positions and the parity bits calculated in each case using ''even'' parity.
The diagram to the right shows which of the four parity bits cover which data bits.
:{| class="wikitable"
|+ align="bottom"|Calculation of Hamming code parity bits
|-
! !!p<sub>1</sub>!!p<sub>2</sub>!!d<sub>1</sub>!!p<sub>3</sub>!!d<sub>2</sub>!!d<sub>3</sub>!!d<sub>4</sub>!!p<sub>4</sub>!!d<sub>5</sub>!!d<sub>6</sub>!!d<sub>7</sub>
|- style="background-color: #CCCCCC"
!Data word (without parity): !! !! !!0!! !!1!!1!!0!! !!1!!0!!1
|-
!p<sub>1</sub>
|'''1'''|| ||0|| ||1|| ||0|| ||1|| ||1
|-
!p<sub>2</sub>
| ||'''0'''||0|| || ||1||0|| || ||0||1
|-
!p<sub>3</sub>
| || || ||'''0'''||1||1||0|| || || ||
|-
!p<sub>4</sub>
| || || || || || || ||'''0'''||1||0||1
|- style="background-color: #CCCCCC"
!Data word (with parity):
|'''1'''||'''0'''||0||'''0'''||1||1||0||'''0'''||1||0||1
|}
The new data word (with parity bits) is now "10001100101". We now assume the final bit gets corrupted and turned from 1 to 0. Our new data word is "10001100100"; and this time when we analyze how the Hamming codes were created we flag each parity bit as 1 when the even parity check fails.
:{| class="wikitable"
|+ align="bottom"|Checking of parity bits (switched bit highlighted)
|-
! !!p<sub>1</sub>!!p<sub>2</sub>!!d<sub>1</sub>!!p<sub>3</sub>!!d<sub>2</sub>!!d<sub>3</sub>!!d<sub>4</sub>!!p<sub>4</sub>!!d<sub>5</sub>!!d<sub>6</sub>!!d<sub>7</sub>!!Parity check!!Parity bit
|- style="background-color: #CCCCCC"
!Received data word: !!1!!0!!0!!0!!1!!1!!0!!0!!1!!0!!0!! !!
|-
!p<sub>1</sub>
|'''1'''|| ||0|| ||1|| ||0|| ||1|| || style="background-color: #DDDDDD"|0||style="background: #ff9090"| '''Fail'''||'''1'''
|-
!p<sub>2</sub>
| ||'''0'''||0|| || ||1||0|| || ||0|| style="background-color: #DDDDDD"|0||style="background: #ff9090"| '''Fail'''||'''1'''
|-
!p<sub>3</sub>
| || || ||'''0'''||1||1||0|| || || || || style="background-color: #90ff90"|Pass||0
|-
!p<sub>4</sub>
| || || || || || || ||'''0'''||1||0|| style="background-color: #DDDDDD"|0||style="background: #ff9090"| '''Fail'''||'''1'''
|}
The final step is to evaluate the value of the parity bits (remembering the bit with lowest index is the [[least significant bit]], i.e., it goes furthest to the right). The integer value of the parity bits is 11, signifying that the 11th bit in the data word (including parity bits) is wrong and needs to be flipped.
:{| class="wikitable"
|-
! !!p<sub>4</sub>!!p<sub>3</sub>!!p<sub>2</sub>!!p<sub>1</sub>!!
|-
!Binary
|1||0||1||1||
|-
!Decimal
!8!! !!2!!1!!Σ = 11
|}
Flipping the 11th bit changes 1000110010'''0''' back into 1000110010'''1'''. Removing the Hamming codes gives the original data word of 0110101.
Note that as parity bits do not check each other, if a single parity bit check fails and all others succeed, then it is the parity bit in question that is wrong and not any bit it checks.
Finally, suppose two bits change, at positions ''x'' and ''y''. If ''x'' and ''y'' have the same bit at the 2<sup>''k''</sup> position in their binary representations, then the parity bit corresponding to that position checks them both, and so will remain the same. However, ''some'' parity bit must be altered, because ''x'' ≠ ''y'', and so some two corresponding bits differ in ''x'' and ''y''. Thus, the Hamming code detects all two bit errors — however, it cannot distinguish them from 1-bit errors.
== References ==
* {{cite book
|last=Moon
|first=Todd K.
|coauthors =
|title=Error Correction Coding
|publisher=[[John Wiley & Sons]]
|date=2005
|location=[[New Jersey]]
|url=http://www.neng.usu.edu/ece/faculty/tmoon/eccbook/book.html
|isbn= 978-0-471-64800-0
}}
* {{cite book
|last=MacKay
|first=David J.C.
|authorlink=David MacKay (scientist)
|coauthors =
|title=Information Theory, Inference and Learning Algorithms
|publisher=[[Cambridge University Press]]
|date=September 2003
|location=[[Cambridge]]
|url=http://www.inference.phy.cam.ac.uk/mackay/itila/book.html
|isbn= 0-521-64298-1
}}
* {{cite conference
|author=D.K. Bhattacharryya, S. Nandi
|title=An efficient class of SEC-DED-AUED codes
|pages=410–415
|booktitle=1997 International Symposium on Parallel Architectures, Algorithms and Networks (ISPAN '97)
|doi=10.1109/ISPAN.1997.645128
}}
==See also==
* [[Hamming distance]]
* [[Golay code]]
* [[Reed-Muller code]]
* [[Reed-Solomon code]]
* [[Turbo code]]
* [[Low-density parity-check code]]
== External links ==
* [http://www.ee.unb.ca/cgi-bin/tervo/hamming.pl CGI script for calculating Hamming distances (from R. Tervo, UNB, Canada)]
[[Category:Coding theory]]
[[Category:Error detection and correction]]
[[Category:Computer arithmetic]]
[[cs:Hammingův kód]]
[[de:Hamming-Code]]
[[es:Código Hamming]]
[[eu:Hamming kode]]
[[fr:Code de Hamming]]
[[ko:해밍 부호]]
[[it:Codice di Hamming]]
[[he:קוד המינג]]
[[nl:Hamming-code]]
[[ja:ハミング符号]]
[[pl:Kod Hamminga]]
[[pt:Código de Hamming]]
[[ru:Код Хемминга]]
[[vi:Mã Hamming]]
[[tr:Hamming kodu]]