what-your-optical-fiber-model-isnt-telling-you

Every Simulator Is a Set of Choices About What to Ignore

The GN model, the split-step Fourier method (SSFM), and every tool built on them are not competing for the title of “most accurate.” Each trades a specific set of physical effects for tractability, and each trade is documented — usually in a methods section, rarely in the result someone screenshots into a planning deck. The practical failure mode isn't picking the wrong tool. It's using the right tool outside the regime it was validated for, without knowing you've crossed that line.

The Analytical Shortcut: Where the GN Model Breaks

The GN model treats nonlinear interference (NLI) as an additive Gaussian noise source — a closed-form approximation that holds well for densely loaded, many-span, unshaped WDM systems. Outside that regime, the error isn't small. Carena et al. document a substantial NLI overestimation error, especially in the first spans of a link, and show that some analytical correction terms proposed to fix this can, in turn, underestimate NLI over low-dispersion fibers — the correction isn't universally safer than the model it corrects.

Probabilistic constellation shaping (PCS) violates a different GN assumption. Hansen et al. show that Maxwell-Boltzmann constellations optimized under an AWGN assumption increase the signal's higher-order statistical moments, pushing the noise measurably away from Gaussian as launch power rises — quantified directly via KL divergence from a Gaussian reference. In their nonlinear-dominated regime, an SNR-optimized shaped constellation designed the conventional way underperformed an unshaped, unoptimized one. A model built on the Gaussian-noise assumption doesn't just misestimate margin here — it can point a design toward the wrong constellation entirely.

The Numerical Ground Truth Has Its Own Blind Spots

SSFM is usually treated as the reference against which analytical models like GN are checked. That status is conditional on a parameter most results don't report: step size. Sinkin et al.'s foundational study on step-size optimization found that a constant step size is the least efficient choice in most cases, and that accuracy has to be actively controlled via a local-error bound rather than assumed from the method's name. An SSFM run configured for speed over a local-error target can be less trustworthy than the analytical model it's meant to validate — its accuracy is a configuration choice, not a property of the algorithm.

A second, less-discussed simplification: system-level SSFM tools commonly solve the Manakov equation — a PMD-averaged approximation of the full vector nonlinear Schrödinger equation — for computational tractability. That's a reasonable engineering choice, but it's still an excluded degree of freedom, and it's rarely stated alongside the results it produced.

The Transceiver Front End Doesn't Disappear Because the Model Ignores It

Most channel-level simulators — analytical or numerical — treat the transmitter and receiver as ideal, folding transceiver impairments into a flat implementation-penalty constant, if at all. Chen et al. experimentally quantified what that constant is hiding: implementation penalties from limited ADC effective number of bits (ENOB), measured directly across Nyquist-shaped 16-QAM, 64-QAM, and 256-QAM. The penalty is real, measurable, and format-dependent — and it grows exactly where operators are pushing hardest: higher-order modulation at high baud rate, which is precisely where DAC bandwidth roll-off, clipping under high-PAPR shaped waveforms, and I/Q or polarization skew also concentrate. None of that lives in a channel model that starts at the fiber's near end.

Partial Views Don't Sum to a Complete One

The compounding problem is architectural, not just a list of missing effects. A typical pipeline — GN model for network-wide screening, SSFM for a handful of candidate links, ideal Tx/Rx throughout — combines three separately-scoped approximations without ever validating the combination. Errors from each exclusion don't necessarily average out; they can compound in the same direction or mask each other, and there's no way to know which until the result is checked against a measured link. A simulator's accuracy is unknown until it's calibrated against field data — that calibration step is the difference between a model and an assumption wearing a model's clothes.

The Metric Nobody's Model Computes

Even a perfectly validated pipeline stops at a physical-layer number: OSNR margin, Q-factor, pre-FEC BER. None of that is the unit a capacity-planning or investment decision is made in. A margin number doesn't natively translate into $ per bit, capex deferred on a fiber build, or SLA penalty exposure from a given outage probability — that translation is a separate, usually manual, step that most simulation workflows don't attempt and most simulator vendors don't claim to solve. It's worth naming as its own gap rather than assuming it happens automatically once the physics is right.