Analysis and Design of Delayed Neural Networks
Neural nets where time-delay is the hard problem, not an afterthought.
Mechanisms
Leakage delay in the negative feedback
Feedback paths have their own delay mode — distinct from hop-to-hop lag.
In addition, a representative delay that is fundamentally different from both transmission delay and distributed delay often appears in the negative feedbacks (i.e., leakage terms), which is named leakage delay.
Why it matters
Feedback paths have their own delay mode — distinct from hop-to-hop lag.
Use when: Use when stability fails and everyone only inspects forward-path latency.
Arguments
Delay is inevitable — model it
Inevitability forces a different math, not a hope that latency goes away.
Since delay is inevitable, it is necessary to model NNs by functional differential/difference equations, that is, differential/difference equations with delayed states.
Why it matters
Inevitability forces a different math, not a hope that latency goes away.
Use when: Use when a design assumes instantaneous communication between nodes.
Stability that should survive small undulation
Robustness is the requirement: small hardware tolerance must not flip stability.
However, it is desired that the stability of NNs will not change with small undulation of structure or parameters.
Why it matters
Robustness is the requirement: small hardware tolerance must not flip stability.
Use when: Use when a model works in sim and dies on slightly different chips.
Definitions
Three kinds of delay
"Delay" is not one bug — it is a taxonomy with different physics.
Delays resulting from the communication mode among neurons are classified as transmission delay, leakage delay, and distributed delay.
Why it matters
"Delay" is not one bug — it is a taxonomy with different physics.
Use when: Use when a model lumps all latency into one parameter.
Transmission delay as communication latency
Latency is renamed as a first-class delay type in the network itself.
In the process of communication among neurons, the latency issue is inevitable, which is identified as transmission delay.
Why it matters
Latency is renamed as a first-class delay type in the network itself.
Use when: Use when "network lag" is treated as outside the model.