Neural nets

6440 /www/nrich/html/content/id/6440/ 3 0000-00-00T00:00:00 <?xml version="1.0" encoding="UTF-8"?> <mdoxml version="1.0"> This problem will introduce you to the ideas behind neural networks. The tasks become more difficult as the problem progresses! In standard mathematical logic, a logic gate activates, or 'fires' depending on whether certain inputs are either ON of OFF. Here's an example of an AND gate: (made using our <a href="http://nrich.maths.org/5974">Circuit Maker</a> program) <mdo:image alt="" src="andgate.png" style="width: 448px; height: 208px;"></mdo:image> In the human brain, there are around $10^{11}$ neurons. Each of these is connected to many other neurons by synapses, the junctions between cells, which allow signals to be transmitted from cell to cell electrically or chemically. Each neuron has a resting potential (the value at which it remains unless stimulated) and a threshold potential (the value at which it will transmit a signal). Each input can carry potential differences; if the sum of the inputs raises the potential of the neuron to the threshold potential, the neuron fires. Imagine neurons with three inputs x, y, z. By treating x, y and z as coordinates in three dimensions, describe geometrically the regions for which the neurons fire in the follow pair of examples: <mdo:image alt="" src="neuron.png" style="width: 405px; height: 208px;"></mdo:image><mdo:image alt="" src="neuron2.png" style="width: 405px; height: 208px;"></mdo:image> We can model the electrical circuits mathematically using?an artificial neural network. These contain input and output neurons, and also may contain hidden neurons. Each neuron has a threshold value $T$, and also a weight factor $W$, by which its output is amplified or deamplified. Here's an example of an OR gate. (The initial inputs are $0$ or $1$) <mdo:image alt="" src="gate1.png" style="width: 350px; height: 219px;"></mdo:image> Verify that the output behaves as claimed. (i.e. C=1 if at least one of A or B is equal to 1) Here's another network. Which logical operation does this represent? <mdo:image alt="" src="gate2.png" style="width: 180px; height: 324px;"></mdo:image> ? Can you create similar networks which replicate the behaviour of other logical operations? So far, these are all examples of feed-forward circuits, i.e. the flow is all in one direction. We can also have feedback circuits, where the output of a neuron can feed back into the network at an earlier point. These can model the biological phenemenon of positive feedback, and the circuit can get extremely complicated very quickly. In more complicated applications we can consider the neuron as a function producing a numerical output from a vector of inputs. Consider the following network in which the output feeds back into the input. Suppose that the output of the neuron is a function of the two input currents R and B. Suppose the the neuron system is initially charge free and then at some time a constant current of 1 is fired in A and an instantaneous pulse $P$ of current 1 is generated. <mdo:image src="feedback1.png"></mdo:image> Consider the behaviour of the system in the case that $g(R,B)=\frac{1}{R+B}$ for different values of the initial pulse $P$. Next consider the following similar system, which starts in a configuration with $A=C=1$ and $B=D=0$ and that $g(C, D)=C+D$ and $h(A,B) = 2^{-(A+B)}$ At the two junctions where the path splits into two, assume that half of the current flows down each. Where the two branches merge, the output is the sum of the two currents in the branches. <mdo:image src="feedback2.png"></mdo:image> What is the output (to 6 decimal places)? Suppose that the current in $C$ can be varied from $0$ to a large number. What range of outputs can be produced? Suppose that $C$ and $A$ are initial pulses of current. Explore and explain your findings. Explore for various functions $g$ and $h$. <div class="framework" style="text-align: left;">Notes and Background A perceptron is a model of a neural network developed by Frank Rosenblatt in 1957. The output of each neuron is determined by a formula that determines the output $f(x)$ given an input vector $x$: $$f(x) = \left\{\begin{array}{c l}1 & w\cdot x + b &gt; 0\\0 & \mathrm{otherwise} \end{array}\right.$$ where $w$ represents the weight of each of the inputs and $-b$ the threshold value. It's possible to teach a perceptron to give the correct outputs for various inputs, given a few initial results. There's an <a href="http://en.wikipedia.org/wiki/Perceptron">article here</a> giving more details. The function $f$ is called the activation function, and defines the output of a neuron given the inputs. The most simple example is a translation of the Heaviside step function: $$f(v) = \left\{\begin{array}{c l}1 & v &gt; b\\0 & v &lt; b \end{array}\right.$$ where $v$ is the sum of the inputs and $b$ is the threshold value. It turns out if we use this to model the neural network, we need to use a great number of neurons. A solution to this problem is to use a sigmoid function, which can realisitically model this behaviour.</div> </mdoxml> <?xml version="1.0" encoding="UTF-8"?> <mdoxml version="1.0">Perceptrons and feedback circuits In standard mathematical logic, a logic gate activates, or 'fires' depending on whether certain inputs are either ON of OFF Example of a logical AND gate. In an artificial neural network, this concept is extended: inputs can carry differing levels of current so that a gate (neurone) fires provided that the overall input level exceeds a certain threshold. For what ranges of input levels will these neurones fire? SOME EXAMPLES SOME PROBLEM WITH XOR A feedback circuit ACTIVATION FUNCTION -- SIGMOIDS </mdoxml> <?xml version="1.0" encoding="UTF-8"?> <mdoxml version="1.0">Treat the systems as evolving according to discrete timesteps and use a spreadsheet for the calculations.</mdoxml> <?xml version="1.0" encoding="UTF-8"?> <mdoxml version="1.0"> If the three inputs add up to at least 15mV, the neuron will fire. If two of the inputs are $x$mV and $y$mV, the third input needs to be at least $(15-x-y)$mV. We find the truth table for the OR function, as expected. <table border="1" cellpadding="1" cellspacing="1" style="width: 100px;"> <tbody> <tr> <td>A\B</td> <td>0</td> <td>1</td> </tr> <tr> <td>0</td> <td>0</td> <td>1</td> </tr> <tr> <td>1</td> <td>1</td> <td>1</td> </tr> </tbody> </table> Here's the truth table for the second network, which turns out to be the XOR function. <table border="1" cellpadding="1" cellspacing="1" style="width: 100px;"> <tbody> <tr> <td>A\B</td> <td>0</td> <td>1</td> </tr> <tr> <td>0</td> <td>0</td> <td>1</td> </tr> <tr> <td>1</td> <td>1</td> <td>0</td> </tr> </tbody> </table> Here's a sketch of the first graph: <mdo:image alt="" src="tanh.png" style="width: 415px; height: 175px;"></mdo:image> We can see the limit as $x\rightarrow \infty$ is $1$ and it's $-1$ as $x\rightarrow -\infty$. Here's the second function, called the logistic function: <mdo:image alt="" src="logistic%20function.png" style="width: 431px; height: 127px;"></mdo:image> It tends to $0$ as $x\rightarrow -\infty$ and $1$ as $x\rightarrow \infty$. At $0$, the function's $\frac{1}{2}$. These have a continous derivative, which is very useful when calculating the weights in the "learning" stage. The fact that the gradient is positive models the idea that the neuron fires more rapidly as the input increases. The asymptotic behaviour is consistent with the fact a neuron has a maximum fire rate. </mdoxml> 2 3 0 0 0 0 1 Neural nets Find out some of the mathematics behind neural networks. Decision Mathematics and Combinatorics Networks/Graph Theory Applications biology Applications physics