The Information Structuralist

A graph-theoretic derivation of the Gilbert-Varshamov bound

Posted in Coding Theory, Information Theory, Mathematics by mraginsky on September 23, 2013

Just a quick note for my reference, but it may be of interest to others.

Let ${A_q(n,d)}$ denote the size of the largest code over a ${q}$-ary alphabet that has blocklength ${n}$ and minimum distance ${d}$. The well-known Gilbert-Varshamov bound says that

$\displaystyle A_q(n,d+1) \ge \frac{q^n}{V_q(n,d)},$

where

$\displaystyle V_q(n,d) = \sum^d_{i=0}{n \choose i}(q-1)^i$

is the volume of a Hamming ball of radius ${d}$ in ${\{0,\ldots,q-1\}^n}$. The usual way of arriving at the GV bound is through a greedy construction: pick an arbitrary codeword ${x}$, then keep adding codewords that are at Hamming distance of at least ${d+1}$ from all codewords that have already been picked. When this procedure terminates, the complement of the union of the Hamming balls of radius ${d}$ around each of the codewords should be empty — otherwise, you will have at least one more codeword at distance of at least ${d+1}$ from the ones already picked, and this would mean that the procedure could not have terminated.

As it turns out, there is another way of deriving the GV bound using graph theory that I have learned from a nice paper by Van Vu and Lei Wu. They use this graph-theoretic interpretation to arrive at an asymptotic improvement of the GV bound. Their result, which I will not go into here, extends an earlier result by Tao Jiang and Alex Vardy for binary codes. As far as I can tell, the graph-theoretic ideas go back to the Jiang-Vardy paper as well.

In order to proceed, we need some definitions and a lemma. Let ${G = (V,E)}$ be an undirected graph. A set ${S \subseteq V}$ of vertices is called independent if no two vertices in ${S}$ are connected by an edge. The independence number of ${G}$, denoted by ${I(G)}$, is the cardinality of the largest independent set. The following lemma is folklore in graph theory:

Lemma 1 Suppose that ${G}$ is ${D}$-regular, i.e., every vertex has exactly ${D}$ neighbors. Then

$\displaystyle I(G) \ge \frac{|V|}{D+1}. \ \ \ \ \ (1)$

Proof: Let ${I}$ be a maximal independent set. Any vertex ${v \in V\backslash I}$ is connected by an edge to at least one ${v' \in I}$, because otherwise ${v}$ would have to be included in ${I}$, which would contradict maximality. Therefore, there are at least ${|V \backslash I| = |V| - |I|}$ edges with one vertex in ${I}$ and another vertex in ${V \backslash I}$. On the other hand, because ${G}$ is ${D}$-regular, there can be at most ${D|I|}$ such edges. This means that

$\displaystyle D|I| \ge |V| - |I|.$

Rearranging, we get (1). $\Box$

Now let us construct the following graph (what Jiang and Vardy call the Gilbert graph): associate a vertex to each word in ${\{0,\ldots,q-1\}^n}$, and connect two vertices by an edge if and only if the Hamming distance between the corresponding words is at most ${d}$. This graph has ${q^n}$ vertices, and each vertex has degree ${D = V_q(n,d)-1}$. Moreover, there is a one-to-one correspondence between independent sets of vertices and ${q}$-ary codes of length ${n}$ and minimum distance at least ${d+1}$, and the independence number of the Gilbert graph is equal to ${A_q(n,d+1)}$. The bound (1) is then precisely the GV bound.

Briefly, the Vu-Wu improvement of the GV bound exploits the deep fact that, when the neighborhood of any vertex in a ${D}$-regular graph is very sparse (in the sense that it has a lot fewer than ${{D \choose 2}}$ edges), the lower bound (1) can be significantly tightened. Apparently, actually counting the number of edges in such a neighborhood of any vertex of the Gilbert graph (by regularity, we may as well look at the neighborhood of the all-zero word) is rather complicated; Vu and Wu instead look at a suitable asymptotic regime when ${n}$ is large and ${d = \alpha n}$ for some ${\alpha \le 1-1/q}$ and replace exact combinatorial bounds by entropy bounds.

Lossless source coding at Western Union

Posted in Information Theory by mraginsky on September 19, 2013

… circa 1935. Here is a quote from Single-Story America (translated as Little Golden America), an American travelogue by Ilya Ilf and Yevgeny Petrov:

There is a whole book of readymade telegrams, long and convincing, lavishly composed telegrams for all occasions. Sending such a telegram costs only twenty-five cents. You see, what gets transmitted over the telegraph is not the text of the telegram, but simply the number under which it is listed in the book, and the signature of the sender. This is quite a funny thing, reminiscent of Drugstore Breakfast #2. Everything is served up in a ready form, and the customer is totally freed from the unpleasant necessity to think, and to spend money on top of it.

Typical set encoding, anyone?