Perceptons(感知机)

感知机 input is 0 or 1, output is 0 or 1. + threshold value(阈值) + weights(权重) + bias(偏置) : measure of how easy it is to get the perceptron to output a 1

Activation Function

sigmoid neuron.

sigmoid neuron

sigmoid Function Just like a perceptron, the sigmoid neuron has inputs, x1,x2,… . But instead of being just 0 or 1 , these inputs can also take on any values between 0 and 1 . So, for instance, 0.638… is a valid input for a sigmoid neuron. Output is between 0 and 1.

\[ \begin{eqnarray} \sigma(z) \equiv \frac{1}{1+e^{-z}}. \end{eqnarray} \]

Tanh

image

\[ \tanh (x)=\frac{2}{1+e^{-2 x}}-1 \]

ReLU

image

\[ \begin{eqnarray} \text{ReLU}(x) \equiv \max(0, x) \end{eqnarray} \]

COEN 6331