Information Structures for Feedback Capacity of Channels with Memory and Transmission Cost: Stochastic Optimal Control & Variational Equalities-Part I

The Finite Transmission Feedback Information (FTFI) capacity is characterized for any class of channel conditional distributions $\big\{{\bf P}_{B_i|B^{i-1}, A_i} :i=0, 1, \ldots, n\big\}$ and $\big\{ {\bf P}_{B_i|B_{i-M}^{i-1}, A_i} :i=0, 1, \ldots, n\big\}$, where $M$ is the memory of the channel, $B^n {\stackrel{\triangle}{=}} \{B_j: j=\ldots, 0,1, \ldots, n\}$ are the channel outputs and $A^n{\stackrel{\triangle}{=}} \{A_j: j=\ldots, 0,1, \ldots, n\}$ are the channel inputs. The characterizations of FTFI capacity, are obtained by first identifying the information structures of the optimal channel input conditional distributions ${\cal P}_{[0,n]} {\stackrel{\triangle}{=}} \big\{ {\bf P}_{A_i|A^{i-1}, B^{i-1}}: i=0, \ldots, n\big\}$, which maximize directed information. The main theorem states, for any channel with memory $M$, the optimal channel input conditional distributions occur in the subset satisfying conditional independence $\stackrel{\circ}{\cal P}_{[0,n]}{\stackrel{\triangle}{=}} \big\{ {\bf P}_{A_i|A^{i-1}, B^{i-1}}= {\bf P}_{A_i|B_{i-M}^{i-1}}: i=1, \ldots, n\big\}$, and the characterization of FTFI capacity is given by $C_{A^n \rightarrow B^n}^{FB, M} {\stackrel{\triangle}{=}} \sup_{ \stackrel{\circ}{\cal P}_{[0,n]} } \sum_{i=0}^n I(A_i; B_i|B_{i-M}^{i-1})$. The methodology utilizes stochastic optimal control theory and a variational equality of directed information, to derive upper bounds on $I(A^n \rightarrow B^n)$, which are achievable over specific subsets of channel input conditional distributions ${\cal P}_{[0,n]}$, which are characterized by conditional independence. For any of the above classes of channel distributions and transmission cost functions, a direct analogy, in terms of conditional independence, of the characterizations of FTFI capacity and Shannon's capacity formulae of Memoryless Channels is identified.