Symbol Detection in Inter-Symbol Interference Channels using Expectation Propagation with Channel Shortening

Iterative message passing detection based on expectation propagation(EP) has demonstrated near-optimum performance in many signal processing and communication scenarios. The method remains feasible even for channel impulse responses (CIRs), where the optimal Bahl-Cocke-Jelinek-Raviv (BCJR) detector is infeasible. However, significant performance degradation occurs for channels with strong inter-symbol interference (ISI), where the initial linear minimum mean square error (LMMSE) estimate is inaccurate. We propose an EP-based detector that operates in a transformed signal space obtained by channel shortening. Specifically, instead of the conventional approach that iterates between an LMMSE estimator and a non-linear symbol-wise demapper, the proposed method iterates between a linear channel shortening filter-based estimator and a nonlinear BCJR detector with reduced memory compared to the actual channel. Additionally, we propose a deliberate mismatch between the initialized messages and the initialized covariance used in the linear estimator in the first iteration for faster convergence. The proposed approach is evaluated for the well-known Proakis-C ISI channel and for CIRs from a wireless measurement campaign. We demonstrate improvements of up to 6dB at 2 bits per channel use and an improved performance-complexity trade-off over conventional EP-based detection.