激活函数Activation Functions

函数Activation Functions"/>

激活函数Activation Functions

激活函数Activation Functions

sigmoid

σ(x)=1(1+e−x) σ ( x ) = 1 ( 1 + e − x ) $$\sigma(x) = \frac{1}{(1 + e^{-x})}$$

• 每个元素被压缩到[0,1]范围内 Squashes numbers to range [0,1]
• 它曾经一度非常流行，因为它有一个很好的解释就像神经元的饱和放电率Historically popular since they have nice interpretation as a saturating “firing rate” of a neuron

3 problems:

1、饱和神经元将使得梯度消失 Saturated neurons “kill” the gradients

2、sigmoid是一个非零中心的函数 Sigmoid outputs are not zero-centered

考虑会发什么，当输入的神经元总是正数…

f(∑i