{"id":3312,"date":"2020-09-24T16:33:19","date_gmt":"2020-09-24T16:33:19","guid":{"rendered":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/"},"modified":"2022-12-14T17:30:36","modified_gmt":"2022-12-14T17:30:36","slug":"c-neural-network-in-a-weekend","status":"publish","type":"post","link":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/","title":{"rendered":"C++ Neural Network in a Weekend"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"3312\" class=\"elementor elementor-3312\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-61fece0 elementor-section-boxed elementor-section-height-default elementor-section-height-default qodef-elementor-content-no\" data-id=\"61fece0\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-0ec905f\" data-id=\"0ec905f\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-52bd272 elementor-widget elementor-widget-shortcode\" data-id=\"52bd272\" data-element_type=\"widget\" data-widget_type=\"shortcode.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-shortcode\">[latexpage]<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-3a18233a elementor-section-boxed elementor-section-height-default elementor-section-height-default qodef-elementor-content-no\" data-id=\"3a18233a\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-7ba2762a\" data-id=\"7ba2762a\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-531d4c25 elementor-widget elementor-widget-text-editor\" data-id=\"531d4c25\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.21.0 - 25-04-2024 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<h2 id=\"introduction\">Introduction<\/h2>\nWould you like to write a neural network from start to finish? Are you perhaps shaky on some of the fundamental concepts and derivations, such as categorical cross-entropy loss or backpropagation? Alternatively, would you like an introduction to machine learning without relying on &#8220;magical&#8221; frameworks that seem to perform AI miracles with only a few lines of code (and just as little intuition)? If so, this article was written for you.\n\nDeep learning as a technology and discipline has been booming. Nearly every facet of deep learning is teeming with progress and healthy competition to achieve state of the art performance and efficiency. It\u2019s no surprise that resources tend to emphasize the &#8220;latest and greatest&#8221; in feats such as object recognition, natural language parsing, &#8220;deep fakes&#8221;, and more. In contrast, fewer resources expand as much on the practical <em>engineering<\/em> aspects of deep learning. That is, how should a deep learning framework be structured? How do you go about rolling your own infrastructure instead of relying on Keras, Pytorch, Tensorflow, or any of the other dominant frameworks? Whether you wish to write your own for learning purposes, or if you need to deploy a neural network on a constrained (i.e. embedded) device, there is plenty to be gained from authoring a neural network from scratch.\n\nThe neural network outlined here is hosted on <a href=\"https:\/\/github.com\/jeremyong\/cpp_nn_in_a_weekend\">github<\/a> and has enough abstractions to vaguely resemble a production network, without being overly engineered as to be indigestible in a sitting or two. The training and test data provided is the venerable <a href=\"http:\/\/yann.lecun.com\/exdb\/mnist\/\">MNIST<\/a> dataset of handwritten digits. While more exotic (and original) datasets exist, MNIST is chosen here because its sheer ubiquity guarantees you can find corresponding literature to help drive further experimentation, or troubleshoot when things go wrong.\n<h2 id=\"background\">Background<\/h2>\nThis section serves as a moderately high-level description of the major mathematical underpinnings of neural networks and may be safely skipped by those who prefer to jump straight to the code.\n\nSuppose we have a task we would like a machine learning model to complete (e.g. recognizing handwritten digits). At a high level, we need to perform the following tasks:\n<ol type=\"1\">\n \t<li>First, we must conceptualize the task as a &#8220;function&#8221; such that the inputs and outputs of the task can be described in a concrete mathematical sense (amenable for programmability).<\/li>\n \t<li>Second, we need a way to quantify the degree to which our model is performing poorly against a known set of correct answers. This is typically denoted as the <em>loss<\/em> or <em>objective<\/em> function of the model.<\/li>\n \t<li>Third, we need an <em>optimization strategy<\/em> which will describe how to adjust the model after feedback is provided regarding the model\u2019s performance as per the loss function described above.<\/li>\n \t<li>Fourth, we need a <em>regularization strategy<\/em> to address inadvertently tuning the model with a high degree of specificity to our training data, at the cost of generalized performance when handling inputs not yet encountered.<\/li>\n \t<li>Fifth, we need an <em>architecture<\/em> for our model, including how inputs are transformed into outputs and an enumaration of all the adjustable parameters the model supports.<\/li>\n \t<li>Finally, we need a robust <em>implementation<\/em> that executes the above within memory and execution budgets, accounting for floating-point stability, reproducibility, and a number of other engineering-related matters.<\/li>\n<\/ol>\n<em>Deep learning<\/em> is distinct from other machine learning models in that the architecture is heavily over-parameterized and based on simpler <em>building blocks<\/em> as opposed to bespoke components. The building blocks used are neurons, or particular arrangements of neurons, typically organized as layers. Over the course of training a deep learning model, it is expected that <em>features<\/em> of the inputs are learned and manifested as various parameter values in these neurons. This is in contrast to traditional machine learning, where features are not learned, but implemented directly.\n<h3 id=\"categorical-cross-entropy-loss\">Categorical Cross-Entropy Loss<\/h3>\nMore concretely, the task at hand is to train a model to recognize a 28 by 28 pixel handwritten greyscale digit. For simplicity, our model will interpret the data as a flattened 784-dimensional vector. Instead of describing the architecture of the model first, we\u2019ll start with understanding what the model should output and how to assess the model\u2019s performance. The output of our model will be a 10-dimensional vector, representing the probability distribution of the supplied input. That is, each element of the output vector indicates the model\u2019s estimation of the probability that the digit\u2019s value matches the corresponding element index. For example, if the model outputs:\n\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9e79b3b elementor-widget elementor-widget-text-editor\" data-id=\"9e79b3b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\(M(I) = [0, 0, 0.5, 0.5, 0, 0, 0, 0, 0, 0]\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-df0b14c elementor-widget elementor-widget-text-editor\" data-id=\"df0b14c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tfor some input image \\(I\\), we interpret this to mean that the model believes there is an equal chance of the examined digit to be a 2 or a 3.\n\nNext, we should consider how to quantify the model\u2019s loss. Suppose, for example, that the image \\(I\\) actually corresponded to the digit &#8220;7&#8221; (our model made a horrible prediction!), how might we penalize the model? In this case, we know that the <em>actual<\/em> probability distribution is the following:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cb97aac elementor-widget elementor-widget-text-editor\" data-id=\"cb97aac\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\([0, 0, 0, 0, 0, 0, 0, 1, 0, 0]\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-aef03c8 elementor-widget elementor-widget-text-editor\" data-id=\"aef03c8\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThis is known as a &#8220;one-hot&#8221; encoded vector, but it may be helpful to think of it as a probability distribution given a set of events that are mutually exclusive (a digit cannot be both a &#8220;7&#8221; <em>and<\/em> a &#8220;3&#8221; for instance).\n\nFortunately, information theory provides us some guidance on defining an easy-to-compute loss function which quantifies the dissimilarities between two probability distributions. If the probability of of an event \\(E\\) is given as \\(P(E)\\), then the <em>entropy<\/em> of this event is given as \\(-log P(E)\\). The negation ensures that this is a positive quantity, and by inspection, the entropy increases as an event becomes less likely. Conversely, in the limit as \\(P(E)\\) approaches \\(1\\), the entropy shrinks to \\(0\\). While several interpretations of entropy are possible, the pertinent interpretation here is that entropy is a <em>measure of the information conveyed when a particular event occurs<\/em>. That the &#8220;sun rose this morning&#8221; is a fairly mundane observation but being told &#8220;the sun exploded&#8221; is sure to pique your attention. Because we are reasonably certain that the sun rises each morning (with near 100% confidence), that &#8220;the sun rises&#8221; is an event that conveys little additional information when it occurs.\n\nLet\u2019s consider next entropy in the context of a probability distribution. Given a discrete random variable \\(X\\) which can take on values \\(x_0, &#8230;, x_{n-1}\\) with probabilities \\(p(x_0), &#8230;, p(x_{n-1})\\), the entropy of the random variable \\(X\\) is defined as:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fafb343 elementor-widget elementor-widget-text-editor\" data-id=\"fafb343\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\(H(X) = &#8211; \\sum_{x\\in X}^{}p(x)\\text{log}p(x)\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d713a55 elementor-widget elementor-widget-text-editor\" data-id=\"d713a55\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tFor example, suppose \\(W\\) is a binary random variable that represents today\u2019s weather which can either be &#8220;sunny&#8221; or &#8220;rainy&#8221; (a binary random variable). The entropy \\(H(W)\\) can be given as:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8309db9 elementor-widget elementor-widget-text-editor\" data-id=\"8309db9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\(H(W) = -S\\text{log}S &#8211; (\\text{1} &#8211; S) \\text{log} (\\text{1} &#8211; S)\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8e5ce6f elementor-widget elementor-widget-text-editor\" data-id=\"8e5ce6f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\twhere \\(S\\) is the probability of a sunny day, and hence \\(1 &#8211; S\\) is the probability of a rainy day. As a binary random variable, the summation over weighted entropies expands to only two terms. What does this quantity mean? If we were to describe it in words, each term of the sum in the entropy calculation corresponds to the information of a particular event, weighted by the probability of the event. Thus, the entropy of the distribution is literally the <em>expected amount of information contained in an event<\/em> for a given distribution. If we plot \\(-S\\text{log} S &#8211; (\\text{1} &#8211; S) \\text{log}(\\text{1} &#8211; S)\\) as a function of \\(S\\), we will see something like this:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8397e44 elementor-widget elementor-widget-image\" data-id=\"8397e44\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.21.0 - 25-04-2024 *\/\n.elementor-widget-image{text-align:center}.elementor-widget-image a{display:inline-block}.elementor-widget-image a img[src$=\".svg\"]{width:48px}.elementor-widget-image img{vertical-align:middle;display:inline-block}<\/style>\t\t\t\t\t\t\t\t\t\t<img fetchpriority=\"high\" decoding=\"async\" width=\"441\" height=\"346\" src=\"https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/binary-entropy.png\" class=\"attachment-large size-large wp-image-13836\" alt=\"\" srcset=\"https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/binary-entropy.png 441w, https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/binary-entropy-300x235.png 300w\" sizes=\"(max-width: 441px) 100vw, 441px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e0c626b elementor-widget elementor-widget-text-editor\" data-id=\"e0c626b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tAs a minor note, while \\(\\text{log}0\\) is an undefined quantity, information theorists accept that \\(\\lim_{p \\to 0} p\\text{log} p = 0\\) by convention. Intuitively, the expected entropy should be unaffected by the set of impossible events.\n\nAs you might expect, when the distribution is 50-50, the uncertainty of a binary is maximal, and by extension the amount of information contained in each event is maximized too. Put another way, if you lived in an area where it was always sunny, you wouldn\u2019t <em>learn anything<\/em> if someone told you it was sunny today. However, in a tropical region characterized by capricious weather, information conveyed about the weather is far more meaningful.\n\nIn the previous example, we weighted the event entropies according to the event\u2019s probability distribution. What would happen if, instead, we used weights corresponding to a <em>different<\/em> probability distribution? This is known as the <em>cross entropy<\/em>:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a96bfdf elementor-widget elementor-widget-text-editor\" data-id=\"a96bfdf\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\(H(p, q) = -\\sum_{x\\in X}^{} p(x)\\text{log} q(x)\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-667afdf elementor-widget elementor-widget-text-editor\" data-id=\"667afdf\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tTo get some intuition about this, first, we note that if \\(p(x) = q(x), \\forall_{x} \\in  X\\), the cross entropy trivially matches the self-entropy. Let\u2019s go back to our binary entropy example and visualize what it looks like if we chose a completely <em>incorrect<\/em> distribution. Specifically, suppose we computed the cross entropy where if the probability of a sunny day is \\(S\\), we weight the entropy with \\(1 &#8211; S\\) instead of \\(S\\) as in the self-entropy formula.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-86b1522 elementor-widget elementor-widget-image\" data-id=\"86b1522\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"441\" height=\"346\" src=\"https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/mismatched-distribution.png\" class=\"attachment-large size-large wp-image-13837\" alt=\"\" srcset=\"https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/mismatched-distribution.png 441w, https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/mismatched-distribution-300x235.png 300w\" sizes=\"(max-width: 441px) 100vw, 441px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-598989f elementor-widget elementor-widget-text-editor\" data-id=\"598989f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tIf you compare the values with the previous figure, you\u2019ll see that the cross entropy diverges from the self-entropy everywhere except (0.5), where \\(S = 1 &#8211; S\\). The difference between the cross entropy \\(H(p, q)\\) and entropy \\(H(p)\\) provides then, a <em>measure of error<\/em> between the presumed distribution \\(q\\) and the true distribution \\(p\\). This difference is also known as the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Kullback%E2%80%93Leibler_divergence\">Kullback-Leibler divergence<\/a> or KL divergence for short.\n\nGiven that the entropy of a given probability distribution \\(p\\) is constant, then \\(H(p)\\) must be constant as well. This is why in practice, we will generally seek to minimize the cross entropy between \\(p\\) and a predicted distribution \\(q\\), which by extension will minimize the Kullback-Leibler divergence as well.\n\nNow, we have the tools to know if our model is succeeding or not! Given an estimation of a sample\u2019s label as before:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-96569d1 elementor-widget elementor-widget-text-editor\" data-id=\"96569d1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\(M(I) = [0, 0, 0.5, 0.5, 0, 0, 0, 0, 0, 0]\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0d4e7ca elementor-widget elementor-widget-text-editor\" data-id=\"0d4e7ca\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\twe will treat our model\u2019s output as a predicted probability distribution of the sample digit\u2019s classification from 0 to 9. Then, we compute the cross entropy between this prediction and the true distribution, which will be in the form of a one-hot vector. Supposing the actual digit is 3 in this particular case \\((P(7) = 1)\\):\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-df6baf7 elementor-widget elementor-widget-text-editor\" data-id=\"df6baf7\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\(\\sum_{x\\in \\left\\{ 0,&#8230;,9 \\right\\}}^{} -P(x) \\text{log} Q(x) = -P(3) \\text{log}(Q(3)) = \\text{log}(0.5) \\approx  0.301\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0a439cc elementor-widget elementor-widget-text-editor\" data-id=\"0a439cc\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tLet\u2019s make a few observations before continuing. First, for a one-hot vector, the entropy is 0 (can you see why?). Second, by pretending the correct digit above is (3) and not, say, (7), we conveniently avoided \\(\\text{log} 0\\) showing up in the final expression. A common method to avoid this is to add a small \\(\\epsilon\\) to the log argument to avoid this singularity, but we\u2019ll discuss this in more detail later.\n<h3 id=\"creating-our-approximation-function-with-a-neural-network\">Creating our Approximation Function with a Neural Network<\/h3>\nNow that we know how to evaluate our model, we\u2019ll need to decide how to go about making predictions in the form of a probability distribution. Our model will need to take as inputs, 28&#215;28 images (which as mentioned before, will be flattened to 784&#215;1 vectors for simplicity). Let\u2019s enumerate the properties our model will need:\n<ol type=\"1\">\n \t<li>Parameterization &#8211; our model will need parameters we can adjust to &#8220;fit&#8221; the model to the data<\/li>\n \t<li>Nonlinearity &#8211; it is assuredly not the case that the probability distribution can be modeled with a set of linear equations<\/li>\n \t<li>Differentiability &#8211; the gradient of our model\u2019s output with respect to any given parameter indicates the <em>impact<\/em> of that parameter on the final result<\/li>\n<\/ol>\nThere are an infinite number of functions that fit this criteria, but here, we\u2019ll use a simple feedforward network with a single hidden layer.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fffd3cc elementor-widget elementor-widget-image\" data-id=\"fffd3cc\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"800\" height=\"623\" src=\"https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/single-hidden-layer-1024x797.png\" class=\"attachment-large size-large wp-image-13838\" alt=\"\" srcset=\"https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/single-hidden-layer-1024x797.png 1024w, https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/single-hidden-layer-300x233.png 300w, https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/single-hidden-layer-768x597.png 768w, https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/single-hidden-layer.png 1063w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-37dc599 elementor-widget elementor-widget-text-editor\" data-id=\"37dc599\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tA few quick notes regarding notation: a superscript of the form \\([i]\\) is used to denote the \\(i\\)th layer. A subscript is used to denote a particular element within a layer or vector. The vector \\(x\\) is usually reserved for training samples, and the vector \\(y\\) is typically reserved for sample labels (i.e. the desired &#8220;answer&#8221; for a given sample). The vector \\(\u0177\\) is used to denote a model\u2019s predicted labels for a given input.\n\nOn the far left, we have the input layer with (784) nodes corresponding to each of the 28 by 28 pixels in an individual sample. Each \\(x_i^{(0)}\\) is a floating point value between 0 and 1 inclusive. Because the data is encoded with 8 bits of precision, there are 256 possible values for each input. Each of the 784 input values fan out to each of the nodes in the hidden layer without modification.\n\nIn the center hidden layer, we have a variable number of nodes that each receive all 784 inputs, perform some processing, and fan out the result to the output nodes on the far right. That is, each node in the hidden layer transforms a \\(\\mathbb{R}^{784}\\) vector into a scalar output, so as a whole, the \\(n\\) nodes collectively need to map \\(\\mathbb{R}^\\to \\mathbb{R}^n\\). The simplest way to do this is with an \\(n \\times 784\\) matrix (treating inputs as column vectors). Modeling the hidden layer this way, each of the \\(n\\) nodes in the hidden layer is associated with a single row in our \\(\\mathbb{R}^{n \\times 784}\\) matrix. Each entry of this matrix is referred to as a <em>weight<\/em>.\n\nWe still have two issues we need to address however. First, a matrix provides a linear mapping between two spaces, and linear maps take 0 to 0 (you can visualize such maps as planes through the origin). Thus, such fully-connected layers typically add a <em>bias<\/em> to each output node to turn the map into an affine map. This enables the model to respond zeroes in the input. Thus, the hidden layer as a whole has now both a weight matrix, and also a bias vector. A linear mapping with a constant bias is commonly referred to as an <em>affine map<\/em>.\n\nThe second issue is that our hidden layer\u2019s now-affine mapping still scales linearly with the input, and one of our requirements for our approximation function was nonlinearity (a strict prerequisite for universality). Thus, we perform one final non-linear operation the result of the affine map. This is known as the <em>activation function<\/em>, and an infinite number of choices present itself here. In practice, the <em>rectifier function<\/em>, defined below, is a perennial choice.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4948ada elementor-widget elementor-widget-text-editor\" data-id=\"4948ada\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\(f(x)=max(0,x)\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-269e440 elementor-widget elementor-widget-image\" data-id=\"269e440\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"440\" height=\"344\" src=\"https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/rectifier-function.png\" class=\"attachment-large size-large wp-image-13839\" alt=\"\" srcset=\"https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/rectifier-function.png 440w, https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/rectifier-function-300x235.png 300w\" sizes=\"(max-width: 440px) 100vw, 440px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-eedc592 elementor-widget elementor-widget-text-editor\" data-id=\"eedc592\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThe rectifier is popular for having a number of desirable properties.\n<ol type=\"1\">\n \t<li>Easy to compute<\/li>\n \t<li>Easy to differentiate (except at 0, which has not been found to be a problem in practice)<\/li>\n \t<li>Sparse activation, which aids in addressing model overfitting and &#8220;unlearning&#8221; useful weights<\/li>\n<\/ol>\nAs our hidden layer units will use this rectifier just before emitting its final output to the next layer, our hidden units may be called <em>rectified linear units<\/em> or ReLUs for short.\n\nSummarizing our hidden layer, the output of each unit in the layer can be written as:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-365373e elementor-widget elementor-widget-text-editor\" data-id=\"365373e\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\(a_i^{[1]} = max(0, W_{i}^{[1]} \\cdot \\mathbf{x}^{[0]} + b_i^{[1]})\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e6c7908 elementor-widget elementor-widget-text-editor\" data-id=\"e6c7908\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tIt\u2019s common to refer to the final activated output of a neural network layer as the vector \\(\\mathbf{a}\\), and the result of the internal affine map \\(\\mathbf{z}\\). Using this notation and considering the output of the hidden layer as a whole as a vector quantity, we can write:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c2c78e1 elementor-widget elementor-widget-text-editor\" data-id=\"c2c78e1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\(\nz^{1} = W^{1}x^{0} + b^{1}\n\\\\ a^{1} = max(o,z^{1})\n\\\\ a^{1},b^{1} \\in R^{n}\n\\\\ W^{1} \\in R^{nx784}\n\\\\ x^{0} \\in R^{784}\n\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-33e0887 elementor-widget elementor-widget-text-editor\" data-id=\"33e0887\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThe last layer to consider is the output layer. As with the hidden layer, we need a dimensionality transform, in this case, taking vectors in \\(\\mathbb{R}^n\\) and mapping them to vectors in \\(\\mathbb{R}^{10}\\) (corresponding to the 10 possible digits in the target output). As before, we will use an affine map with the appropriately sized weight matrix and bias vector. Here, however, the rectifier isn\u2019t suitable as an activation function because we want to emit a probability distribution. To be a valid probability distribution, each output of the hidden layer must be in the range \\([0, 1]\\), and the sum of all outputs must equal (1). The most common activation function used to achieve this is the <em>softmax function<\/em>:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3faefe9 elementor-widget elementor-widget-text-editor\" data-id=\"3faefe9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\(\\mathrm{softmax}(\\mathbf{z})_i = \\frac{\\text{exp}(z_i)}{\\sum_j \\text{exp}(z_j)}\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-548f8bc elementor-widget elementor-widget-text-editor\" data-id=\"548f8bc\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tGiven a vector input \\(z\\), each component of the softmax output (as a vector quantity) is given as per the expression above. The exponential functions conveniently map negative numbers to positive numbers, and the denominator ensures that all outputs will be between 0 and 1, and sum to 1 as desired. There are other reasons why an exponential function is used here, stemming from our choice of a loss function (based on the underpinning notion of maximum-likelihood estimation), but we won\u2019t get into that in too much detail here (consult the further reading section at the end to learn more). Suffice it to say that an additional benefit of the exponential function is its clean interaction with the logarithm used in our choice of loss function, especially when we will need to compute gradients in the next section.\n\nSummarizing our neural network architecture, with two weight matrices and two bias vectors, we can construct two affine maps which map vectors in \\(\\mathbb{R}^{784}\\) to \\(\\mathbb{R}^n\\) to \\(\\mathbb{R}^{10}\\). Prior to forwarding the results of one affine map as the input of the next, we employ an activation function to add non-linearity to the model. First, we use a linear rectifier and second, we use a softmax function, ensuring that we end up with a nice discrete probability distribution with 10 possible events corresponding to the 10 digits.\n\nOur network is small enough that we can actually write out the entire process as a single function using the notation we\u2019ve built so far:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4667339 elementor-widget elementor-widget-text-editor\" data-id=\"4667339\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<center>\\(f(\\mathbf{x}^{[0]}) = \\mathbf{y}^{[2]} = \\mathrm{softmax}\\left( \\mathbf{W}^{[2]} \\left(\\text{max} \n\\left(\\mathbf{0}, \\mathbf{W}^{[1]}\\mathbf{x}^{[0]} + \\mathbf{b}^{[1]} \\right) \\right) + \\mathbf{b}^{[1]} \\right)\\)<\/center>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d093ba4 elementor-widget elementor-widget-text-editor\" data-id=\"d093ba4\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<h3 id=\"optimizing-our-network\">Optimizing our network<\/h3>\nWe now have a model given above which can turn our 784 dimensional inputs into a 10-element probability distribution, <em>and<\/em> we have a way to evaluate how accuracy of each prediction. Next, we need a reliable way to improve the model based on the feedback provided by our loss function. This is known as function <em>optimization<\/em>, and most methods of model optimization are based on the principle of <em>gradient descent<\/em>.\n\nThe idea is quite simple. Given a function with a set of parameters which we\u2019ll denote \\(\\theta\\), the partial derivative of that function with respect to a given parameter \\(\\theta_{i} \\in \\theta\\) tells us the overall <em>impact<\/em> of \\(\\theta_i\\) on the final result. In our model, we have many parameters; each weight and bias constitutes an individually tunable parameter. Thus, our strategy should be, given a set of input samples, compute the loss our model produces for each sample. Then, compute the partial derivatives of that loss with respect to <em>every parameter<\/em> in our model. Finally, adjust each parameter in proportion to its impact on the final loss. Mathematically, this process is described below (note that the superscript \\((i)\\) is used to denote the \\(i\\)-th sample):\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2b6b38a elementor-widget elementor-widget-image\" data-id=\"2b6b38a\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"601\" height=\"305\" src=\"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/cpp-neural-formula-001.jpg\" class=\"attachment-large size-large wp-image-3740\" alt=\"\" srcset=\"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/cpp-neural-formula-001.jpg 601w, https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/cpp-neural-formula-001-300x152.jpg 300w\" sizes=\"(max-width: 601px) 100vw, 601px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e848567 elementor-widget elementor-widget-text-editor\" data-id=\"e848567\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThe last set of equations follow from factorizing and rearranging the expressions preceding it. It\u2019s often confusing to newer practitioners that the partial derivative of softmax needs this unique treatment. The key observation is that softmax is a vector-function. It accepts a vector as an input and emits a vector as an output. It also &#8220;mixes&#8221; the input components, thereby imposing a functional dependence of <em>every output component<\/em> on <em>every input component<\/em>. The lone \\((exp{z_i})\\) in the numerator of the softmax equation creates an asymmetric dependence of the output component on the input components.\n\nFinally, let\u2019s consider the partial derivative of the linear rectifier.\n\n\\(\nbegin{aligned}\nmathrm{ReLU}(z) &amp;= max(0, z) \\\nfrac{partial mathrm{ReLU}(z)}{partial z} &amp;=\nbegin{cases}\n0 &amp; z &lt; 0 \\\nmathrm{undefined} &amp; z = 0 \\\nz &amp; z &gt; 0\nend{cases}\nend{aligned}\n\\)\n\nWhile the partial derivative <em>exactly<\/em> at 0 is undefined, in practice, the derivative is simply assigned to 0. Why the non-differentiability at 0 isn\u2019t an issue has been a subject of practical debate for a long time. Here is a simple line of thinking to justify the apparent issue. Consider a rectifier function that is nudged <em>ever so slightly<\/em> to the right such that the inflection point is \\((epsilon \/ 2)\\), where \\((epsilon)\\) is the smallest positive floating point number the machine can represent. In this case, the model will never produce a value that sits directly on this inflection point, and as far as the computer is concerned, we never encounter a point where this function is non-differentiable. We can even imagine an infinitesimal curve that smooths out the function at that inflection point if we want. Either way, experimentally, the linear rectifier remains one of the most effective activation functions for reasons mentioned, so we have no reason to discredit it over a technicality.\n\nNow that we can compute partial derivatives of all the nonlinear functions in our neural network (and presumably the linear functions as well), we are prepared to compute loss gradients with respect to any parameter in the network. Our tool of choice is the venerable chain rule of calculus:\n\n\\(left.frac{partial f(g(x))}{partial x}rightrvert_x = left.frac{partial f}{partial g}rightrvert_{g(x)} left.frac{partial g}{partial x}rightrvert_x\\)\n\nThis gives us the partial derivative of a composite function \\((fcirc g)\\) evaluated at a particular value of \\((x)\\). Our model itself is a series of composite functions, and as we can now compute the partials of each individual component in the model, we are ready to begin implementation in the next section.\n<h2 id=\"setting-up\">Setting up<\/h2>\nOur project will leverage <a href=\"https:\/\/cmake.org\">CMake<\/a> as the meta-build system to support as many operating systems and compilers as possible. A modern C++ compiler will also be needed to compile the code. As of this writing, the code has been tested with GCC 10.1.0 and Clang 10.0.0. You should feel free to simply adapt the code to your compiler and build system of choice. To emphasize the independent nature of this project, <em>no further dependencies are needed<\/em>. At your discretion, you may opt to use external testing frameworks, matrix and math libraries, data structures, or any other external dependency as you see fit. If you\u2019re a newer C++ practitioner, you are welcome to model the structure of the final project hosted on Github <a href=\"https:\/\/github.com\/jeremyong\/nn_in_a_weekend\">here<\/a>.\n\nIn addition, you will need the data hosted on the MNIST database website linked <a href=\"http:\/\/yann.lecun.com\/exdb\/mnist\/\">here<\/a>. The four files available there consist of training images, training labels, test images, and test labels.\n\nIt is highly recommended that you attempt to clone the repository and get things running (instructions on the README will always be kept up to date). The code presented in this article will not be completely exhaustive, but will touch on all the major points, eschewing only various rudimentary helpers functions or uninteresting details for brevity. Alternatively, a valid approach may be to simply follow along the implementation notes below and attempt to blaze your own trail. Both branches are viable approaches for learning.\n<h2 id=\"implementation\">Implementation<\/h2>\n<h3 id=\"the-computational-graph\">The Computational Graph<\/h3>\nThe network we will be constructing is purely sequential. Inputs flow from left to right and the only connections made are between one layer and the layer immediately succeeding it. In reality, many production-grade neural networks specialized for computer vision, natural language processing, and other domains rely on architectures that are non-sequential. Examples include ResNet, which introduces connections between layers that are not adjacent, and various recurrent neural networks, which have a cyclic topology (outputs of the model are fed back as inputs to the model). Thus, it\u2019s useful to think of the model as a whole as <em>computational graph<\/em>. While we won\u2019t be employing any complicated computational graph topologies here, we will still structure the code with this notion in mind. Each layer of our network will be modeled as a <code>Node<\/code> with data flowing forwards and backwards through the node during training. Providing support for a fully general computational graph (i.e. non-sequential) is outside the scope of this tutorial, but some scaffolding will be provided should you want to extend it yourself in the future. For now, here is the interface we\u2019ll use:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-92c8c2c elementor-widget elementor-widget-code-highlight\" data-id=\"92c8c2c\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>#include <cstdint>\r\n#include <string>\r\n#include <vector>\r\n\r\nusing num_t = float;\r\nusing rne_t = std::mt19937;\r\n\r\n\/\/ To be defined later. This class encapsulates all the nodes in our graph \r\nclass Model;\r\n\r\nclass Node\r\n{\r\npublic:\r\n    Node(Model& model, std::string name);\r\n    \r\n    \/\/ Nodes must describe how they should be initialized\r\n    virtual void init(rne_t& rne) = 0;\r\n    \r\n    \/\/ During forward propagation, nodes transform input data and feed results\r\n    \/\/ to all subsequent nodes\r\n    virtual void forward(num_t* inputs) = 0;\r\n\r\n    \/\/ During reverse propagation, nodes receive loss gradients to its previous\r\n    \/\/ outputs and compute gradients with respect to each tunable parameter\r\n    virtual void reverse(num_t* gradients) = 0;\r\n    \r\n    \/\/ If the node has tunable parameters, this method should be overridden\r\n    \/\/ to reflect the quantity of tunable parameters\r\n    virtual size_t param_count() const noexcept { return 0; }\r\n    \r\n    \/\/ Accessor for parameter by index\r\n    virtual num_t* param(size_t index) { return nullptr; }\r\n    \r\n    \/\/ Access for loss-gradient with respect to a parameter specified by index\r\n    virtual num_t* gradient(size_t index) { return nullptr; }\r\n    \r\n    \/\/ Human-readable name for debugging purposes\r\n    std::string const& name() const noexcept { return name_; }\r\n    \r\n    \/\/ Information dump for debugging purposes\r\n    virtual void print() const = 0;\r\n\r\nprotected:\r\n    friend class Model;\r\n    \r\n    Model& model_;\r\n    std::string name_;\r\n    \/\/ Nodes that precede this node in the computational graph\r\n    std::vector<Node*> antecedents_;\r\n    \/\/ Nodes that succeed this node in the computational graph\r\n    std::vector<Node*> subsequents_;\r\n};<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f8c3b2c elementor-widget elementor-widget-text-editor\" data-id=\"f8c3b2c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThe bulwark of the implementation will consist of implementing this interface for all the nodes in our network. We will need to implement this interface for each of the nodes shown in the diagram below.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1c1bf1b elementor-widget elementor-widget-image\" data-id=\"1c1bf1b\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"565\" height=\"129\" src=\"https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/nodes.png\" class=\"attachment-large size-large wp-image-5277\" alt=\"\" srcset=\"https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/nodes.png 565w, https:\/\/contentlab.com\/wp-content\/uploads\/2020\/09\/nodes-300x68.png 300w\" sizes=\"(max-width: 565px) 100vw, 565px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f7f7d19 elementor-widget elementor-widget-text-editor\" data-id=\"f7f7d19\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThe first node (<code>MNIST<\/code>) will be responsible for acquiring new training samples and feeding it to the next layer for processing. In addition, it will provide an accessor that the final categorical cross-entropy loss node will use to query the correct label for that sample (the &#8220;label query&#8221;). The hidden node will perform the affine transform and apply the linear rectification activation. The output node will also perform an affine transform, but will then apply the softmax function. Finally, the loss node will compute the loss of the predicted distribution based on the queried label for a given sample.\n\nIn the figure above, solid arrows from left to right indicate data flow during the <em>feedforward<\/em> or <em>evaluation<\/em> portion of the model\u2019s execution. Each solid arrow corresponds to a data vector emitted by the source, and ingested by the destination. The dashed arrows from right to left indicate data flow during the <em>backpropagation<\/em> or <em>reverse accumulation<\/em> portion of the algorithm. These arrows correspond to gradient vectors of the evaluated loss with respect to the outputs passed during the feedforward phase. For example, as seen above, the hidden node is expected to forward data to the output node \\(((mathbf{a}^{[1]}))\\). Later, after the model prediction has been computed and the loss evaluated, the gradient of the loss with respect to those outputs is expected \\(((partial J_{CE}\/partial a^{[1]}_i) for each (a_i^{[1]}) in (mathbf{a}^{[1]}))\\).\n\nWhen simply evaluating the model (without training), the final loss node will simply be omitted from the graph. In addition, no back-propagation of gradients will occur as the model parameters are ossified during evaluation.\n\nThe model class interface shown below will be used to house all the nodes in the computational graph, and provide various routines that are useful for operating over all constituent nodes as a collection.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ef74c29 elementor-widget elementor-widget-code-highlight\" data-id=\"ef74c29\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>class Model\r\n{\r\npublic:\r\n    Model(std::string name);\r\n    \r\n    \/\/ Add a node to the model, forwarding arguments to the node's constructor\r\n    template <typename Node_t, typename... T>\r\n    Node_t& add_node(T&&... args)\r\n    {\r\n        nodes_.emplace_back(\r\n            std::make_unique<Node_t>(*this, std::forward<T>(args)...));\r\n        return reinterpret_cast<Node_t&>(*nodes_.back());\r\n    }\r\n\r\n    \/\/ Create a dependency between two constituent nodes\r\n    void create_edge(Node& dst, Node& src);\r\n\r\n    \/\/ Initialize the parameters of all nodes with the provided seed. If the\r\n    \/\/ seed is 0, a new random seed is chosen instead. Returns the seed used.\r\n    rne_t::result_type init(rne_t::result_type seed = 0);\r\n\r\n    \/\/ Adjust all model parameters of constituent nodes using the\r\n    \/\/ provided optimizer (shown later)\r\n    void train(Optimizer& optimizer);\r\n\r\n    std::string const& name() const noexcept\r\n    {\r\n        return name_;\r\n    }\r\n\r\n    void print() const;\r\n\r\n    \/\/ Routines for saving and loading model parameters to and from disk\r\n    void save(std::ofstream& out);\r\n    void load(std::ifstream& in);\r\n\r\nprivate:\r\n    friend class Node;\r\n\r\n    std::string name_;\r\n    std::vector<std::unique_ptr<Node>> nodes_;\r\n};<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-eec7e0e elementor-widget elementor-widget-text-editor\" data-id=\"eec7e0e\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<h3 id=\"training-data-and-labels\">Training Data and Labels<\/h3>\nAll machine learning pipelines must consider how to ingest data and labels. Data refers to the information the model is expected to use to make inferences and predictions. Labels correspond to the &#8220;correct answer&#8221; for each data sample, used to compute losses and train the model. The interface of the MNIST data parser is shows below as an implemented <code>Node<\/code> class.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8adffad elementor-widget elementor-widget-code-highlight\" data-id=\"8adffad\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-c line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-c\">\n\t\t\t\t\t<xmp>class MNIST : public Node\r\n{\r\npublic:\r\n    constexpr static size_t DIM = 28 * 28;\r\n    \r\n    \/\/ The constructor receives an input filestream corresponding to the\r\n    \/\/ data samples and labels\r\n    MNIST(Model& model, std::ifstream& images, std::ifstream& labels);\r\n    \r\n    \/\/ This is an input node and has no parameters to initialize\r\n    void init(rne_t&) override {}\r\n    \r\n    \/\/ Read the next sample and label and forward the data\r\n    void forward(num_t* data = nullptr) override;\r\n\r\n    \/\/ No optimization is done in this node so this is a no-op\r\n    void reverse(num_t* gradients = nullptr) override {}\r\n    \r\n    void print() const override;\r\n\r\n    \/\/ Consume the next sample and label from the file streams\r\n    void read_next();\r\n    \r\n    \/\/ Accessor for the most recently read sample\r\n    num_t const* data() const noexcept\r\n    {\r\n        return data_;\r\n    }\r\n    \r\n    \/\/ Accessor for the most recently read label\r\n    num_t* label() const noexcept\r\n    {\r\n        return label_;\r\n    }\r\n    \r\n    \/\/ Quick ASCII visualization of the last digit read\r\n    void print_last();\r\n    \r\nprivate:\r\n    std::ifstream& images_;\r\n    std::ifstream& labels_;\r\n    uint32_t image_count_;\r\n\r\n    char buf_[DIM];\r\n    num_t data_[DIM];\r\n    num_t label_[10];\r\n};<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cf8e091 elementor-widget elementor-widget-text-editor\" data-id=\"cf8e091\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tIn the constructor, we must verify that the files passed as arguments are valid MNIST data and label files. Both files start with distinct &#8220;magic values&#8221; as a quick sanity check. The sample file starts with 2051 encoded as a 4-byte big-endian unsigned integer, whereas the label file starts with 2049. For the data file, the magic number is followed by the image count and image dimensions. The label file magic number is followed by the label count (expected to match the image count).\n\nTo consume big-endian unsigned integers from the file stream, we\u2019ll use a simple routine:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e605246 elementor-widget elementor-widget-code-highlight\" data-id=\"e605246\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>void read_be(std::ifstream& in, uint32_t* out)\r\n{\r\n    char* buf = reinterpret_cast<char*>(out);\r\n    in.read(buf, 4);\r\n\r\n    std::swap(buf[0], buf[3]);\r\n    std::swap(buf[1], buf[2]);\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8c20211 elementor-widget elementor-widget-text-editor\" data-id=\"8c20211\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tIf you happen to be using a big-endian processor, you will not need to perform the byte swaps, but most desktop and mobile architectures are little-endian.\n\nThe implementation that parses the magic numbers and various other descriptors is produced below:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fea9417 elementor-widget elementor-widget-code-highlight\" data-id=\"fea9417\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>MNIST::MNIST(Model& model, std::ifstream& images, std::ifstream& labels)\r\n    : Node{model, \"MNIST input\"}\r\n    , images_{images}\r\n    , labels_{labels}\r\n{\r\n    \/\/ Confirm that passed input file streams are well-formed MNIST data sets\r\n    uint32_t image_magic;\r\n    read_be(images, &image_magic);\r\n    if (image_magic != 2051)\r\n    {\r\n        throw std::runtime_error{\"Images file appears to be malformed\"};\r\n    }\r\n    read_be(images, &image_count_);\r\n\r\n    uint32_t labels_magic;\r\n    read_be(labels, &labels_magic);\r\n    if (labels_magic != 2049)\r\n    {\r\n        throw std::runtime_error{\"Images file appears to be malformed\"};\r\n    }\r\n\r\n    uint32_t label_count;\r\n    read_be(labels, &label_count);\r\n    if (label_count != image_count_)\r\n    {\r\n        throw std::runtime_error(\r\n            \"Label count did not match the number of images supplied\");\r\n    }\r\n\r\n    uint32_t rows;\r\n    uint32_t columns;\r\n    read_be(images, &rows);\r\n    read_be(images, &columns);\r\n    if (rows != 28 || columns != 28)\r\n    {\r\n        throw std::runtime_error{\r\n            \"Expected 28x28 images, non-MNIST data supplied\"};\r\n    }\r\n\r\n    printf(\"Loaded images file with %d entriesn\", image_count_);\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-493312c elementor-widget elementor-widget-text-editor\" data-id=\"493312c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tNext, let\u2019s implement the <code>MNIST::read_next<\/code>, which will consume the next sample and label from the file streams:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-67a3382 elementor-widget elementor-widget-code-highlight\" data-id=\"67a3382\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>void MNIST::read_next()\r\n{\r\n    images_.read(buf_, DIM);\r\n    num_t inv = num_t{1.0} \/ num_t{255.0};\r\n    for (size_t i = 0; i != DIM; ++i)\r\n    {\r\n        data_[i] = static_cast<uint8_t>(buf_[i]) * inv;\r\n    }\r\n\r\n    char label;\r\n    labels_.read(&label, 1);\r\n\r\n    for (size_t i = 0; i != 10; ++i)\r\n    {\r\n        label_[i] = num_t{0.0};\r\n    }\r\n    label_[static_cast<uint8_t>(label)] = num_t{1.0};\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-378e8ee elementor-widget elementor-widget-text-editor\" data-id=\"378e8ee\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tFor the labels, note that the label is encoded as a single unsigned digit, but we convert it to a 1-hot encoding for loss computation purposes later. If your application can assume that the labels will be one-hot encoded, this conversion may not be necessary and a more efficient implementation is possible.\n\nTo verify our work, let\u2019s write up a quick-and-dirty ASCII printer for the last read digit and try our parser out. If you have a rendering backend (written in say, Vulkan, D3D12, OpenGL, etc.) at your disposal, you may wish to use that instead for a cleaner visualization.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ca702f5 elementor-widget elementor-widget-code-highlight\" data-id=\"ca702f5\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>void MNIST::print_last()\r\n{\r\n    for (size_t i = 0; i != 10; ++i)\r\n    {\r\n        if (label_[i] == num_t{1.0})\r\n        {\r\n            printf(\"This is a %zu:n\", i);\r\n            break;\r\n        }\r\n    }\r\n\r\n    for (size_t i = 0; i != 28; ++i)\r\n    {\r\n        size_t offset = i * 28;\r\n        for (size_t j = 0; j != 28; ++j)\r\n        {\r\n            if (data_[offset + j] > num_t{0.5})\r\n            {\r\n                if (data_[offset + j] > num_t{0.9})\r\n                {\r\n                    printf(\"#\");\r\n                }\r\n                else if (data_[offset + j] > num_t{0.7})\r\n                {\r\n                    printf(\"*\");\r\n                }\r\n                else\r\n                {\r\n                    printf(\".\");\r\n                }\r\n            }\r\n            else\r\n            {\r\n                printf(\" \");\r\n            }\r\n        }\r\n        printf(\"n\");\r\n    }\r\n    printf(\"n\");\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8bf6de0 elementor-widget elementor-widget-text-editor\" data-id=\"8bf6de0\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tOn my machine, consuming the evaluation data and printing it produces the following result (the first sample from the test data is shown):\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f9dafc9 elementor-widget elementor-widget-code-highlight\" data-id=\"f9dafc9\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language- line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-\">\n\t\t\t\t\t<xmp>This is a 7:\r\n                            \r\n       *..                  \r\n      *#####********.       \r\n          .*#*####*##.      \r\n                   ##       \r\n                   #*       \r\n                  ##        \r\n                 .##        \r\n                 ##         \r\n                .#*         \r\n                *#          \r\n                #*          \r\n               ##           \r\n              *#.           \r\n             *#*            \r\n             ##             \r\n            *#              \r\n           .##              \r\n           ###              \r\n           ##*              \r\n           #*<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f84cedf elementor-widget elementor-widget-text-editor\" data-id=\"f84cedf\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tso we can be somewhat confident that our MNIST data ingestor is working properly. The only remaining routine we need to implement is <code>MNIST::forward<\/code> which should consume the next sample, and forward the data to all subsequent nodes in the graph.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f95cba4 elementor-widget elementor-widget-code-highlight\" data-id=\"f95cba4\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>void MNIST::forward(num_t* data)\r\n{\r\n    read_next();\r\n    for (Node* node : subsequents_)\r\n    {\r\n        node->forward(data_);\r\n    }\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3f95a2a elementor-widget elementor-widget-text-editor\" data-id=\"3f95a2a\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tSuch an interface ensures our <code>MNIST<\/code> node will be interoperable with networks that aren\u2019t purely sequential.\n<h3 id=\"the-feedforward-node\">The Feedforward Node<\/h3>\nThe hidden and output nodes have much in common and so will be implemented in terms of a single feedforward node class. The feedforward node will need a configurable activation function and dimensionality. Here\u2019s the interface for the <code>FFNode<\/code>:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b0cdd53 elementor-widget elementor-widget-code-highlight\" data-id=\"b0cdd53\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>enum class Activation\r\n{\r\n    ReLU,\r\n    Softmax\r\n};\r\n\r\nclass FFNode : public Node\r\n{\r\npublic:\r\n    \/\/ A feedforward node is defined by the activation\r\n    \/\/ function and input\/output dimensionality\r\n    FFNode(Model& model,\r\n           std::string name,\r\n           Activation activation,\r\n           uint16_t output_size,\r\n           uint16_t input_size);\r\n\r\n    void init(rne_t& rne) override;\r\n\r\n    \/\/ The input data should have size input_size_\r\n    void forward(num_t* inputs) override;\r\n\r\n    \/\/ The gradient data should have size output_size_\r\n    void reverse(num_t* gradients) override;\r\n\r\n    size_t param_count() const noexcept override\r\n    {\r\n        \/\/ Weight matrix entries + bias entries\r\n        return (input_size_ + 1) * output_size_;\r\n    }\r\n\r\n    num_t* param(size_t index);\r\n    num_t* gradient(size_t index);\r\n\r\n    void print() const override;\r\n\r\nprivate:\r\n    Activation activation_;\r\n    uint16_t output_size_;\r\n    uint16_t input_size_;\r\n\r\n    \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\r\n    \/\/ Node Parameters \/\/\r\n    \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\r\n\r\n    \/\/ weights_.size() := output_size_ * input_size_\r\n    std::vector<num_t> weights_;\r\n    \/\/ biases_.size() := output_size_\r\n    std::vector<num_t> biases_;\r\n    \/\/ activations_.size() := output_size_\r\n    std::vector<num_t> activations_;\r\n\r\n    \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\r\n    \/\/ Loss Gradients \/\/\r\n    \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\r\n\r\n    std::vector<num_t> activation_gradients_;\r\n\r\n    \/\/ During the training cycle, parameter loss gradients are accumulated in\r\n    \/\/ the following buffers.\r\n    std::vector<num_t> weight_gradients_;\r\n    std::vector<num_t> bias_gradients_;\r\n\r\n    \/\/ This buffer is used to store temporary gradients used in a SINGLE\r\n    \/\/ backpropagation pass. Note that this does not accumulate like the weight\r\n    \/\/ and bias gradients do.\r\n    std::vector<num_t> input_gradients_;\r\n\r\n    \/\/ The last input is needed to compute loss gradients with respect to the\r\n    \/\/ weights during backpropagation\r\n    num_t* last_input_;\r\n};<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-94c8ec6 elementor-widget elementor-widget-text-editor\" data-id=\"94c8ec6\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tCompared to the <code>MNIST<\/code> node, the <code>FFNode<\/code> uses a lot more state to track all tunable parameters (weight matrix elements and biases), as well as the loss gradients corresponding to each parameter. The loss gradients must be kept because, remember, utilizing them to actually adjust the parameters is performed only after <code>N<\/code> samples have been evaluated, where <code>N<\/code> is the chosen batch size in our stochastic gradient descent algorithm. If the purpose of some of the class members here is still opaque, they will show up later when implement backpropagation.\n\nFirst, we must decide how to initialize the weights and biases of our node. When deciding on a scheme, there are a few key principles to keep in mind. First, the initialization must exhibit symmetry of any sort. For example, if all the parameters are initialized to the same random value, the loss gradients with respect to all individual parameters will be identical, and our network will be no better than a network with a single parameter. In addition, we do not want the parameters to be initialized such that they are too large, or too small. Most papers that discuss weight initialization strive to ensure that the loss gradients remain in a realm where floating point number retain precision (in the range \\(([1, 2))\\)). The other criteria is that parameters should generally be initialized such that they are roughly similar in magnitude. Parameters that deviate too far from the mean are likely to either dominate loss gradients, or produce too small a signal to contribute. Proper parameter initialization is but a small part of addressing the larger problem common in neural networks known as the problem of <em>exploding and vanishing gradients<\/em>. Here, we present the implementation with a couple references if you wish to dig deeper.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-21da22c elementor-widget elementor-widget-code-highlight\" data-id=\"21da22c\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>void FFNode::init(rne_t&amp; rne)\r\n{\r\n    num_t sigma;\r\n    switch (activation_)\r\n    {\r\n    case Activation::ReLU:\r\n        \/\/ Kaiming He, et. al. weight initialization for ReLU networks\r\n        \/\/ https:\/\/arxiv.org\/pdf\/1502.01852.pdf\r\n        \/\/\r\n        \/\/ Suggests using a normal distribution with variance := 2 \/ n_in\r\n        sigma = std::sqrt(2.0 \/ static_cast&lt;num_t&gt;(input_size_));\r\n        break;\r\n    case Activation::Softmax:\r\n    default:\r\n        \/\/ LeCun initialization as suggested in \"Self-Normalizing Neural\r\n        \/\/ Networks\"\r\n        \/\/ https:\/\/arxiv.org\/pdf\/1706.02515.pdf\r\n        sigma = std::sqrt(1.0 \/ static_cast&lt;num_t&gt;(input_size_));\r\n        break;\r\n    }\r\n\r\n    \/\/ NOTE: Unfortunately, the C++ standard does not guarantee that the results\r\n    \/\/ obtained from a distribution function will be identical given the same\r\n    \/\/ inputs across different compilers and platforms. A production ML\r\n    \/\/ framework will likely implement its own distributions to provide\r\n    \/\/ deterministic results.\r\n    auto dist = std::normal_distribution&lt;num_t&gt;{0.0, sigma};\r\n\r\n    for (num_t&amp; w : weights_)\r\n    {\r\n        w = dist(rne);\r\n    }\r\n\r\n    \/\/ NOTE: Setting biases to zero is a common practice, as is initializing the\r\n    \/\/ bias to a small value (e.g. on the order of 0.01). It is unclear if the\r\n    \/\/ latter produces a consistent result over the former, but the thinking is\r\n    \/\/ that a non-zero bias will ensure that the neuron always \"fires\" at the\r\n    \/\/ beginning to produce a signal.\r\n    \/\/\r\n    \/\/ Here, we initialize all biases to a small number, but the reader should\r\n    \/\/ consider experimenting with other approaches.\r\n    for (num_t&amp; b : biases_)\r\n    {\r\n        b = 0.01;\r\n    }\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c9b8fb2 elementor-widget elementor-widget-text-editor\" data-id=\"c9b8fb2\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThe common theme is that the distribution of random weights scales roughly as the inverse square root of the input vector size. This way, the distribution of the node\u2019s output will fall in a &#8220;nice&#8221; range with respect to floating-point precision. Other initialization schemes are of course possible, and in some cases critical depending on the choice of activation function.\n\nWith weights and biases initialized, it\u2019s time to implement <code>FFNode::forward<\/code>. The straightforward plan is, for both the ReLU and softmax nodes, first perform the affine transform \\((mathbf{W}mathbf{x} + mathbf{b})\\), then perform the activation function which will be one of the linear rectifier or the softmax function. Here\u2019s what this looks like:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7667bac elementor-widget elementor-widget-code-highlight\" data-id=\"7667bac\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>void FFNode::forward(num_t* inputs)\r\n{\r\n    \/\/ Remember the last input data for backpropagation later\r\n    last_input_ = inputs;\r\n\r\n    for (size_t i = 0; i != output_size_; ++i)\r\n    {\r\n        \/\/ For each output vector, compute the dot product of the input data\r\n        \/\/ with the weight vector add the bias\r\n\r\n        num_t z{0.0};\r\n\r\n        size_t offset = i * input_size_;\r\n\r\n        for (size_t j = 0; j != input_size_; ++j)\r\n        {\r\n            z += weights_[offset + j] * inputs[j];\r\n        }\r\n        \/\/ Add neuron bias\r\n        z += biases_[i];\r\n\r\n        switch (activation_)\r\n        {\r\n        case Activation::ReLU:\r\n            activations_[i] = std::max(z, num_t{0.0});\r\n            break;\r\n        case Activation::Softmax:\r\n        default:\r\n            activations_[i] = std::exp(z);\r\n            break;\r\n        }\r\n    }\r\n\r\n    if (activation_ == Activation::Softmax)\r\n    {\r\n        \/\/ softmax(z)_i = exp(z_i) \/ sum_j(exp(z_j))\r\n        num_t sum_exp_z{0.0};\r\n        for (size_t i = 0; i != output_size_; ++i)\r\n        {\r\n            \/\/ NOTE: with exploding gradients, it is quite easy for this\r\n            \/\/ exponential function to overflow, which will result in NaNs\r\n            \/\/ infecting the network.\r\n            sum_exp_z += activations_[i];\r\n        }\r\n        num_t inv_sum_exp_z = num_t{1.0} \/ sum_exp_z;\r\n        for (size_t i = 0; i != output_size_; ++i)\r\n        {\r\n            activations_[i] *= inv_sum_exp_z;\r\n        }\r\n    }\r\n\r\n    \/\/ Forward activation data to all subsequent nodes in the computational\r\n    \/\/ graph\r\n    for (Node* subsequent : subsequents_)\r\n    {\r\n        subsequent->forward(activations_.data());\r\n    }\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8f5b6c7 elementor-widget elementor-widget-text-editor\" data-id=\"8f5b6c7\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tAs before, we forward all final results to all subsequent nodes even though there will only be a single subsequent node in this case. Whenever writing code as above, it is prudent to consider all potential corner cases which could result in the myriad issues that arise in floating-point computation:\n<ul>\n \t<li>Loss of precision<\/li>\n \t<li>Floating point overflow and underflow<\/li>\n \t<li>Divide by zero<\/li>\n<\/ul>\nLoss of precision easily occurs when in a number of situations, such as subtracting two quantities of similar size, or adding and multiplying quantities with greatly different magnitudes. Floating point overflow and underflow occur typically when repeatedly performing an operation such that an accumulator explodes to (infty) or (-infty). In this case, the use of <code>std::exp<\/code> is one operation that sticks out. We will not implement a stable softmax here, but the following identity can be used to improve its stability should you need it:\n\n\\(mathrm{softmax}(mathbf{z} + mathbf{C})_i = mathrm{softmax}(mathbf{z})_i\\)\n\nIn this expression, \\((mathbf{C})\\) is a constant vector where all its elements are equal in value. Expanding the definition of softmax in the LHS gives:\n\n\\(\nbegin{aligned}\nmathrm{softmax}(mathbf{z} + mathbf{C})_i &amp;= frac{exp{(z_i + C)}}{sum_iexp{(z_i + C})} \\\n&amp;= frac\n{exp{z_i}exp{C}}\n{left(sum_iexp{z_i}right)exp C} \\\n&amp;= mathrm{softmax}(mathbf{z})_i &amp;&amp; blacksquare\nend{aligned}\n\\)\n\nThus, if we are considered about saturating <code>std::exp<\/code> with a large argument, we can simply set \\((C)\\) to be the additive inverse of the \\((z_i)\\) with the greatest magnitude within \\((mathbf{z})\\). Performing this each time we apply softmax will usually maintain the arguments of the softmax within a reasonable range (unless elements of \\((z_i)\\) explode in opposite directions).\n\nAs a practical implementor\u2019s trick, it is possible to enable floating point exception traps to throw an exception when a <code>NaN<\/code> is generated in a floating point register. Using libc for example, we can trap floating point exceptions using\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e1ea373 elementor-widget elementor-widget-code-highlight\" data-id=\"e1ea373\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp><cfenv>\r\nfeenableexcept(FE_INVALID | FE_OVERFLOW);<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4e08131 elementor-widget elementor-widget-text-editor\" data-id=\"4e08131\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tIt is also possible to trap exceptions specifically in regions where you anticipate a potential issue (which enhances the overall throughput of the network). In the interest of brevity, please consult your compiler\u2019s documentation for how to do this.\n\nOne observation you might have made is the first line of our routine.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b382829 elementor-widget elementor-widget-code-highlight\" data-id=\"b382829\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>last_input_ = inputs;<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5468cb3 elementor-widget elementor-widget-text-editor\" data-id=\"5468cb3\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tHere, we retain a pointer to the data ingested by the feedforward node for a full training cycle. Before delving into any derivations, let\u2019s first present the code for the backpropagation of gradients through our feedforward node and dissect it immediately afterwards.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1364e14 elementor-widget elementor-widget-code-highlight\" data-id=\"1364e14\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>void FFNode::reverse(num_t* gradients)\r\n{\r\n    \/\/ First, we compute dJ\/dz as dJ\/dg(z) * dg(z)\/dz and store it in our\r\n    \/\/ activations array\r\n    for (size_t i = 0; i != output_size_; ++i)\r\n    {\r\n        \/\/ dg(z)\/dz\r\n        num_t activation_grad{0.0};\r\n        switch (activation_)\r\n        {\r\n        case Activation::ReLU:\r\n            if (activations_[i] > num_t{0.0})\r\n            {\r\n                activation_grad = num_t{1.0};\r\n            }\r\n            else\r\n            {\r\n                activation_grad = num_t{0.0};\r\n            }\r\n            \/\/ dJ\/dz = dJ\/dg(z) * dg(z)\/dz\r\n            activation_gradients_[i] = gradients[i] * activation_grad;\r\n            break;\r\n        case Activation::Softmax:\r\n        default:\r\n            for (size_t j = 0; j != output_size_; ++j)\r\n            {\r\n                if (i == j)\r\n                {\r\n                    activation_grad += activations_[i]\r\n                                       * (num_t{1.0} - activations_[i])\r\n                                       * gradients[j];\r\n                }\r\n                else\r\n                {\r\n                    activation_grad\r\n                        += -activations_[i] * activations_[j] * gradients[j];\r\n                }\r\n            }\r\n\r\n            activation_gradients_[i] = activation_grad;\r\n            break;\r\n        }\r\n    }\r\n\r\n    for (size_t i = 0; i != output_size_; ++i)\r\n    {\r\n        \/\/ dJ\/db_i = dJ\/dg(z_i) * dJ(g_i)\/dz_i.\r\n        bias_gradients_[i] += activation_gradients_[i];\r\n    }\r\n\r\n    std::fill(input_gradients_.begin(), input_gradients_.end(), 0);\r\n\r\n    for (size_t i = 0; i != output_size_; ++i)\r\n    {\r\n        size_t offset = i * input_size_;\r\n        for (size_t j = 0; j != input_size_; ++j)\r\n        {\r\n            input_gradients_[j]\r\n                += weights_[offset + j] * activation_gradients_[i];\r\n        }\r\n    }\r\n\r\n    for (size_t i = 0; i != input_size_; ++i)\r\n    {\r\n        for (size_t j = 0; j != output_size_; ++j)\r\n        {\r\n            weight_gradients_[j * input_size_ + i]\r\n                += last_input_[i] * activation_gradients_[j];\r\n        }\r\n    }\r\n\r\n    for (Node* node : antecedents_)\r\n    {\r\n        node->reverse(input_gradients_.data());\r\n    }\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5d0bdb4 elementor-widget elementor-widget-text-editor\" data-id=\"5d0bdb4\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThis code is likely more difficult to digest, so let\u2019s break it down into parts. During reverse accumulation (aka backpropagation), we will be given the loss gradients with respect to all of the outputs from the most recent forward pass, written mathematically as \\((partial J_{CE}\/partial a_i)\\) for each output scalar \\((a_i)\\). Given that information, we need to perform the following tasks:\n<ol type=\"1\">\n \t<li>Compute \\((partial J_{CE}\/partial w_{ij})\\) for each weight in our weight matrix<\/li>\n \t<li>Compute \\((partial J_{CE}\/partial b_i)\\) for each bias in our bias vector<\/li>\n \t<li>Compute \\((partial J_{CE}\/partial x_i)\\) for each input scalar in the most recent forward pass<\/li>\n \t<li>Propagate all the loss gradients with respect to the inputs in step 3 back to the antecedent nodes<\/li>\n<\/ol>\nAs all outputs pass through an activation function, we will need to compute \\((partial J_{CE}\/partial g(mathbf{z})_i)\\) where \\((g)\\) is one of the linear rectifier or softmax function corresponding to a particular component of the output vector. Both derivatives are computed in the background section, so we\u2019ll just recite the results here. For the linear rectifier, \\((partial J_{CE}\/partial g(mathbf{z})_i)\\) will simply be 1 if \\((a_i neq 0)\\), and 0 otherwise. The softmax gradient is slightly more involved, but because every output of the softmax contributes additively to the loss, we require a sum of gradients here:\n\n\\(frac{partial J_{CE}}{partial mathrm{softmax}(mathbf{z})_i} = frac{partial J_{CE}}{partial a_i}sum_{j} begin{cases}\nmathrm{softmax}(mathbf{z})_ileft(1 &#8211; mathrm{softmax}(mathbf{z})_iright) &amp; i = j \\\n-mathrm{softmax}(mathbf{z})_i mathrm{softmax}(mathbf{z})_j &amp; i neq j\nend{cases}\\)\n\nThe factor \\((partial J_{CE}\/partial a_i)\\) comes from the chain rule and is passed in from the subsequent node. These intermediate expressions are computed, scaled by \\((partial a_i\/partial z_i)\\), and then stored in <code>activation_gradients_<\/code> in the top portion of <code>FFNode::reverse<\/code>. Equivalently by the chain rule, we are caching in <code>activation_gradients_<\/code> \\((partial J_{CE}\/partial z_i)\\) for each \\((i)\\). Because the loss gradients with respect to every parameter and input have a functional dependence on the activation function gradients, all results computed in tasks 1 through 4 above will depend on <code>activation_gradients_<\/code>.\n<h4 id=\"computing-bias-gradients\">Computing bias gradients<\/h4>\nThe bias gradients are the easiest to compute due to how they show up in the expression. Since a node\u2019s output is given as\n\n\\(a_i = gleft(mathbf{W}_i cdot mathbf{x} + b_i = z_iright)\\)\n\nfor some activation function \\((g)\\), the derivative with respect to \\((b_i)\\) is just\n\n\\(\nbegin{aligned}\nfrac{partial{a_i}}{partial b_i} &amp;= frac{partial g}{partial z_i}frac{partial z_i}{partial b_i} \\\n&amp;= frac{partial g}{partial z_i}\nend{aligned}\n\\)\n\nThus we can simply accumulate the result stored in <code>activation_gradients_<\/code> as the loss gradient with respect to each bias. Please take note! The code that performs this update is\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c978772 elementor-widget elementor-widget-code-highlight\" data-id=\"c978772\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>    for (size_t i = 0; i != output_size_; ++i)\r\n    {\r\n        bias_gradients_[i] += activation_gradients_[i];\r\n    }<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2d03019 elementor-widget elementor-widget-text-editor\" data-id=\"2d03019\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThe following code would <em>not<\/em> be correct:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f1a668e elementor-widget elementor-widget-code-highlight\" data-id=\"f1a668e\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>    for (size_t i = 0; i != output_size_; ++i)\r\n    {\r\n        \/\/ NOTE: WRONG! Will only alone batch sizes of 1\r\n        bias_gradients_[i] = activation_gradients_[i];\r\n    }<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f853333 elementor-widget elementor-widget-text-editor\" data-id=\"f853333\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tAs the admonition in the comment suggests, while it\u2019s helpful to conceptualize the loss gradient as something that resets every time we perform a forward and reverse pass of a training sample, in actuality, we require the gradients with respect to the <em>cumulative mean loss accrued while evaluating the entire batch<\/em> for stochastic gradient descent. Luckily, because the losses per sample accumulate additively, the gradients of the loss with respect to all parameters in the model also update additively.\n<h4 id=\"computing-the-weight-gradients\">Computing the weight gradients<\/h4>\nThe weight gradients are slightly more involved than the bias gradients, but are still relatively easy to compute with a bit of bookkeeping. For any given weight \\((w_{ij})\\), we can observe that such a weight participates only in the evaluation of \\((z_i)\\). That is:\n\n\\(\nbegin{aligned}\nfrac{partial mathbf{z}}{partial w_{ij}} &amp;= frac{partial z_i}{partial w_{ij}} \\\n&amp;= frac{partial (mathbf{w}_{i} cdot mathbf{x}) + b_i}{partial w_{ij}} \\\n&amp;= x_j \\\nend{aligned}\n\\)\n\n\\(\nboxed{frac{partial J_{CE}}{partial w_{ij}} = frac{partial J_{CE}}{partial a_i}frac{partial a_i}{partial z_i}x_j}\n\\)\n\nThe boxed result shows the final loss gradient with respect to a weight parameter. The weight gradient accumulation appears in the following code, where all (N times M) weights are updated in a couple of nested loops:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1badef4 elementor-widget elementor-widget-code-highlight\" data-id=\"1badef4\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>    for (size_t i = 0; i != input_size_; ++i)\r\n    {\r\n        for (size_t j = 0; j != output_size_; ++j)\r\n        {\r\n            weight_gradients_[j * input_size_ + i]\r\n                += last_input_[i] * activation_gradients_[j];\r\n        }\r\n    }<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8105b6c elementor-widget elementor-widget-text-editor\" data-id=\"8105b6c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<h4 id=\"computing-the-input-gradients\">Computing the input gradients<\/h4>\nThe last set of gradients we need to compute are the loss gradients with respect to the inputs, to be forwarded to the antecedent node. This calculation is similar to the calculation of the weight gradients in terms of the linear dependence. However, it is important to note that a given input participates in the computation of <em>all<\/em> output scalars. Thus, we expect each individual input gradient to be a summation.\n\n\\(\nfrac{partial J_{CE}}{partial x_i} = sum_j frac{partial J_{CE}}{partial a_j}frac{partial a_j}{partial z_j}w_{ij}\n\\)\n\nThe code that computes the input gradients is defined here:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a323d59 elementor-widget elementor-widget-code-highlight\" data-id=\"a323d59\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>    std::fill(input_gradients_.begin(), input_gradients_.end(), 0);\r\n\r\n    for (size_t i = 0; i != output_size_; ++i)\r\n    {\r\n        size_t offset = i * input_size_;\r\n        for (size_t j = 0; j != input_size_; ++j)\r\n        {\r\n            input_gradients_[j]\r\n                += weights_[offset + j] * activation_gradients_[i];\r\n        }\r\n    }<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5e73a12 elementor-widget elementor-widget-text-editor\" data-id=\"5e73a12\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tNote that unlike the weight and bias gradients which accumulate while training an entire batch of samples, the input gradients here are ephemeral and reset every pass since the only depend on the evaluation of an individual sample.\n\nFinally, to complete the <code>FFNode::reverse<\/code> method, the input gradients computed are based backwards for use in an antecedent node\u2019s gradient update (reproduced below). The code as presented <em>does not work<\/em> with non-sequential computational graphs, but is meant to provide a starting point for futher experimentation.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1a4945e elementor-widget elementor-widget-code-highlight\" data-id=\"1a4945e\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>    for (Node* node : antecedents_)\r\n    {\r\n        node->reverse(input_gradients_.data());\r\n    }<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4080fc3 elementor-widget elementor-widget-text-editor\" data-id=\"4080fc3\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<h3 id=\"the-categorical-cross-entropy-loss-node\">The Categorical Cross-Entropy Loss Node<\/h3>\nThe last node we need to implement is the node which computes the categorical cross-entropy of the prediction. A possible class definition for such this node is shown below:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3881e3c elementor-widget elementor-widget-code-highlight\" data-id=\"3881e3c\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>class CCELossNode : public Node\r\n{\r\npublic:\r\n    CCELossNode(Model& model,\r\n                std::string name,\r\n                uint16_t input_size,\r\n                size_t batch_size);\r\n\r\n    \/\/ No initialization is needed for this node\r\n    void init(rne_t&) override {}\r\n\r\n    void forward(num_t* inputs) override;\r\n\r\n    \/\/ As a loss node, the argument to this method is ignored (the gradient of\r\n    \/\/ the loss with respect to itself is unity)\r\n    void reverse(num_t* gradients = nullptr) override;\r\n\r\n    void print() const override;\r\n\r\n    \/\/ During training, this must be set to the expected target distribution\r\n    \/\/ for a given sample\r\n    void set_target(num_t const* target)\r\n    {\r\n        target_ = target;\r\n    }\r\n\r\n    num_t accuracy() const;\r\n    num_t avg_loss() const;\r\n    void reset_score();\r\n\r\nprivate:\r\n    uint16_t input_size_;\r\n\r\n    \/\/ We minimize the average loss, not the net loss so that the losses\r\n    \/\/ produced do not scale with batch size (which allows us to keep training\r\n    \/\/ parameters constant)\r\n    num_t inv_batch_size_;\r\n    num_t loss_;\r\n    num_t const* target_;\r\n    num_t* last_input_;\r\n    \/\/ Stores the last active classification in the target one-hot encoding\r\n    size_t active_;\r\n    num_t cumulative_loss_{0.0};\r\n    \/\/ Store running counts of correct and incorrect predictions\r\n    size_t correct_   = 0;\r\n    size_t incorrect_ = 0;\r\n    std::vector<num_t> gradients_;\r\n};<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bd6435f elementor-widget elementor-widget-text-editor\" data-id=\"bd6435f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThe <code>CCELossNode<\/code> is similar to other nodes in that it implements a forward pass for computing the loss of a given sample, and a reverse pass to compute gradients of that loss and pass them back to the antecedent node. Distinct from the previous nodes is that the argument to <code>CCELossNode::reverse<\/code> is ignored as the loss node is not expected to have any subsequents.\n\nThe implementation of <code>CCELossNode::forward<\/code> follows from the definition of cross-entropy, recalled here with some modifications:\n\n\\(J_{CE}(hat{mathbf{y}}, mathbf{y}) = -sum_j y_j log{left(max(hat{y}_j, epsilon) right)} \\)\n\n\\((J)\\) is the common symbol ascribed to the cost or objective function, while \\((hat{y})\\) and \\((y)\\) refer to the predicted distribution and correct distribution respectively. In addition, the argument of the logarithm is clamped with a small \\((epsilon)\\) to avoid a numerical singularity. The implementation is as follows:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f1bd296 elementor-widget elementor-widget-code-highlight\" data-id=\"f1bd296\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>void CCELossNode::forward(num_t* data)\r\n{\r\n    num_t max{0.0};\r\n    size_t max_index;\r\n\r\n    loss_ = num_t{0.0};\r\n    for (size_t i = 0; i != input_size_; ++i)\r\n    {\r\n        if (data[i] > max)\r\n        {\r\n            max_index = i;\r\n            max       = data[i];\r\n        }\r\n\r\n        loss_ -= target_[i]\r\n                 * std::log(\r\n                     std::max(data[i], std::numeric_limits<num_t>::epsilon()));\r\n\r\n        if (target_[i] != num_t{0.0})\r\n        {\r\n            active_ = i;\r\n        }\r\n    }\r\n\r\n    if (max_index == active_)\r\n    {\r\n        ++correct_;\r\n    }\r\n    else\r\n    {\r\n        ++incorrect_;\r\n    }\r\n\r\n    cumulative_loss_ += loss_;\r\n\r\n    \/\/ Store the data pointer to compute gradients later\r\n    last_input_ = data;\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-acea2fd elementor-widget elementor-widget-text-editor\" data-id=\"acea2fd\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tAs with the feedforward node, a pointer to the inputs to the node is preserved to compute gradients later. A bit of bookkeeping is also done so we can track accuracy and accumulate loss during batch. The derivative of the loss of an individual sample with respect to the inputs is also fairly straightforward.\n\n\\(\nbegin{aligned}\nfrac{partial J_{CE}}{partial{hat{y}_i}} &amp;= frac{partial left(-sum_j y_jlog{left(max(hat{y}_j, epsilon)right)}right)}{partial hat{y}_i} \\\n&amp;= -frac{y_i}{max(hat{y}_i, epsilon)}\nend{aligned}\n\\)\n\nThe implementation is similarly straightforward. As with the other nodes with loss gradients, the loss gradients with respect to all inputs are forwarded to antecedent nodes.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2e0488e elementor-widget elementor-widget-code-highlight\" data-id=\"2e0488e\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>void CCELossNode::reverse(num_t* data)\r\n{\r\n    for (size_t i = 0; i != input_size_; ++i)\r\n    {\r\n        gradients_[i] = -inv_batch_size_ * target_[i]\r\n            \/ std::max(last_input_[i], std::numeric_limits<num_t>::epsilon());\r\n    }\r\n\r\n    for (Node* node : antecedents_)\r\n    {\r\n        node->reverse(gradients_.data());\r\n    }\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c0721bf elementor-widget elementor-widget-text-editor\" data-id=\"c0721bf\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tOne thing to keep in mind here is that this implementation is <em>not<\/em> the most efficient implementation possible for a softmax layer feeding to a cross-entropy loss function by any stretch. The code and derivation here is completely general for arbitrary sample probability distributions. If, however, we can assume that the target distribution is one-hot encoded, then all gradients in this node will either be 0 or \\((-1\/hat{y}_k)\\) where \\((k)\\) is the active label in the one-hot target. Upon substitution in the previous layer, it should be clear that important cancellations are possible that dramatically simplify the gradient computations in the softmax layer. Here\u2019s the simplification, again assuming that the (k)th index is the correct label:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8cdccb2 elementor-widget elementor-widget-image\" data-id=\"8cdccb2\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"597\" height=\"278\" src=\"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/cpp-neural-formula-002.jpg\" class=\"attachment-large size-large wp-image-3741\" alt=\"\" srcset=\"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/cpp-neural-formula-002.jpg 597w, https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/cpp-neural-formula-002-300x140.jpg 300w\" sizes=\"(max-width: 597px) 100vw, 597px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c0d860b elementor-widget elementor-widget-text-editor\" data-id=\"c0d860b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tWhen following the computation above, remember that \\((partial J_{CE} \/ partial a_i)\\) is 0 for all \\((i neq k)\\). Thus, the only term in the sum that survives is the term corresponding to \\((j = k)\\), at which point we break out the differentation depending on whether \\((i = k)\\) or \\((i neq k)\\).\n\nThis is an elegant result! Essentially, the gradient of a the loss with respect to an emitted probability \\((p(x))\\) is simply \\((p(x))\\) if \\((x)\\) was not the correct label, and \\((p(x) &#8211; 1)\\) if it was. Considering the effect of gradient descent, this should check out with our intuition. The optimizer seeks to suppress probabilities predicted that should have been 0, and increase probabilities predicted that should have been 1. Check for yourself that after gradient descent is performed, the gradients derived here will nudge the model in the appropriate direction.\n\nThis sort of optimization highlights an important observation about backpropagation, namely, that backpropagation does not guarantee any sort of optimality beyond a worst-case performance ceiling. Several production neural networks have architectures that employ heuristics to identify optimizations such as this one, but the problem of generating a perfect computational strategy is NP and so not covered here. The code provided here will remain in the general form, despite being slower in the interest of maintaining generality and not adding complexity, but you are encouraged to consider abstractions to permit this type of optimization in your own architecture (a useful keyword to aid your research is <em>common subexpression elimination<\/em> or <em>CSE<\/em> for short).\n\nThe last thing we need to provide for <code>CCELossNode<\/code> are a few helper routines:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8fa2e2a elementor-widget elementor-widget-code-highlight\" data-id=\"8fa2e2a\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>void CCELossNode::print() const\r\n{\r\n    std::printf(\"Avg Loss: %ft%f%% correctn\", avg_loss(), accuracy() * 100.0);\r\n}\r\n\r\nnum_t CCELossNode::accuracy() const\r\n{\r\n    return static_cast<num_t>(correct_)\r\n           \/ static_cast<num_t>(correct_ + incorrect_);\r\n}\r\nnum_t CCELossNode::avg_loss() const\r\n{\r\n    return cumulative_loss_ \/ static_cast<num_t>(correct_ + incorrect_);\r\n}\r\n\r\nvoid CCELossNode::reset_score()\r\n{\r\n    cumulative_loss_ = num_t{0.0};\r\n    correct_         = 0;\r\n    incorrect_       = 0;\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4360537 elementor-widget elementor-widget-text-editor\" data-id=\"4360537\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThese routines let us observe the performance of our network during training in terms of both loss and accuracy.\n<h3 id=\"gradient-descent-optimizer\">Gradient Descent Optimizer<\/h3>\nAt some point after loss gradients with respect to model parameters have accumulated, the gradients will need to be used to actually adjust the parameters themselves. This is provided by the <code>GDOptimizer<\/code> class implemented as below:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-376a3d5 elementor-widget elementor-widget-code-highlight\" data-id=\"376a3d5\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>class GDOptimizer : public Optimizer\r\n{\r\npublic:\r\n    \/\/ \"Eta\" is the commonly accepted character used to denote the learning\r\n    \/\/ rate. Given a loss gradient dJ\/dp for some parameter p, during gradient\r\n    \/\/ descent, p will be adjusted such that p' = p - eta * dJ\/dp.\r\n    GDOptimizer(num_t eta) : eta_{eta} {}\r\n\r\n    \/\/ This should be invoked at the end of each batch's evaluation. The\r\n    \/\/ interface technically permits the use of different optimizers for\r\n    \/\/ different segments of the computational graph.\r\n    void train(Node& node) override;\r\n\r\nprivate:\r\n    num_t eta_;\r\n};\r\n\r\nvoid GDOptimizer::train(Node& node)\r\n{\r\n    size_t param_count = node.param_count();\r\n    for (size_t i = 0; i != param_count; ++i)\r\n    {\r\n        num_t& param    = *node.param(i);\r\n        num_t& gradient = *node.gradient(i);\r\n\r\n        param = param - eta_ * gradient;\r\n\r\n        \/\/ Reset the gradient which will be accumulated again in the next\r\n        \/\/ training epoch\r\n        gradient = num_t{0.0};\r\n    }\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c9e9152 elementor-widget elementor-widget-text-editor\" data-id=\"c9e9152\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tNot shown is the <code>Optimizer<\/code> class interface which simply provides a virtual <code>train<\/code> method. As you implement more sophisticated optimizers, you will find that more state may be needed to perform necessary tasks (e.g. computing gradient moving averages). Also implicit in this implementation is that our <code>Node<\/code> classes need to provide an indexing scheme for each parameter as well as an accessor for the total number of parameters. For example, accessing the <code>FFNode<\/code> parameters is a fairly simple matter:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-654818b elementor-widget elementor-widget-code-highlight\" data-id=\"654818b\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>num_t* FFNode::param(size_t index)\r\n{\r\n    if (index < weights_.size())\r\n    {\r\n        return &weights_[index];\r\n    }\r\n    return &biases_[index - weights_.size()];\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-88bdb68 elementor-widget elementor-widget-text-editor\" data-id=\"88bdb68\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tThe parameters are indexed 0 through the return value of <code>Node::param_count()<\/code> minus one. Note that the optimizer doesn\u2019t care whether the parameter accessed in this way is a weight, bias, average, etc. As a trainable parameter, the only thing that matters during gradient descent is the current value and the loss gradient.\n<h2 id=\"tying-it-all-together\">Tying it all Together<\/h2>\nNow that we have the individual nodes implemented, all that remains is to wire things up and start training! This is how we can construct a model with a input, hidden, output, and loss nodes, all wired sequentially.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-93b4a70 elementor-widget elementor-widget-code-highlight\" data-id=\"93b4a70\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-cpp line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-cpp\">\n\t\t\t\t\t<xmp>    Model model{\"ff\"};\r\n\r\n    MNIST& mnist = &model.add_node<MNIST>(images, labels);\r\n\r\n    FFNode& hidden = model.add_node<FFNode>(\"hidden\", Activation::ReLU, 32, 784);\r\n\r\n    FFNode& output\r\n        = model.add_node<FFNode>(\"output\", Activation::Softmax, 10, 32);\r\n\r\n    CCELossNode& loss = &model.add_node<CCELossNode>(\"loss\", 10, batch_size);\r\n    loss.set_target(mnist.label());\r\n\r\n    model.create_edge(hidden, mnist);\r\n    model.create_edge(output, hidden);\r\n    model.create_edge(loss, output);\r\n    \r\n    \/\/ This function should visit all constituent nodes and initialize\r\n    \/\/ their parameters\r\n    model.init();\r\n    \r\n    \/\/ Create a gradient descent optimizer with a hardcoded learning rate\r\n    GDOptimizer optimizer{num_t{0.3}};<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fd33545 elementor-widget elementor-widget-text-editor\" data-id=\"fd33545\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tAs mentioned before, the &#8220;edges&#8221; are somewhat cosmetic as none of our nodes actually support multiple node inputs or outputs. An actual implementation that would support such a non-sequential topology will likely need a sort of signals and slots abstraction. The interface provided here is strictly to impress on you the importance of the abstraction of our neural network as a computational graph, which is critical when additional complexity is added later.\n\nWith this, we are ready to implement the core loop of the training algorithm.\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0929780 elementor-widget elementor-widget-code-highlight\" data-id=\"0929780\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>    for (size_t i = 0; i != 256; ++i)\r\n    {\r\n        for (size_t j = 0; j != 64; ++j)\r\n        {\r\n            mnist->forward();\r\n            loss->reverse();\r\n        }\r\n\r\n        model.train(optimizer);\r\n    }<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-13fd08a elementor-widget elementor-widget-text-editor\" data-id=\"13fd08a\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\nHere, we train our model over 256 batches. Each batch consists of 64 samples, and for each sample, we invoke <code>MNIST::forward<\/code> and <code>CCELossNode::reverse<\/code>. During the forward pass, our <code>MNIST<\/code> node extracts a new sample and label and forwards the sample data to the next node. This data propagates through the network until the final output distribution is passed to the loss node and losses are computed. All this occurs within the single line: <code>mnist-&gt;forward()<\/code>. In the subsequent line, gradients are computed and passed back until the reverse accumulation terminates at the <code>MNIST<\/code> node again. After all gradients for the batch are accumulated, the model can <code>train<\/code>, which invokes the optimizer on each node to simultaneously adjust all model parameters for each node.\n\nAfter adding some additional logging, the results of the network look like this:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5ccd6da elementor-widget elementor-widget-code-highlight\" data-id=\"5ccd6da\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language- line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-\">\n\t\t\t\t\t<xmp>Executing training routine\r\nLoaded images file with 60000 entries\r\nhidden: 784 -> 32\r\noutput: 32 -> 10\r\nInitializing model parameters with seed: 116726080\r\nAvg Loss: 0.254111  96.875000% correct<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-482038d elementor-widget elementor-widget-text-editor\" data-id=\"482038d\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tTo evaluate the efficacy of the model, we can serialize all the parameters to disk, load them up, disable the training step, and evaluate the model on the test data. For this particular run, the results were as follows:\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a99308f elementor-widget elementor-widget-code-highlight\" data-id=\"a99308f\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language- line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-\">\n\t\t\t\t\t<xmp>Executing evaluation routine\r\nLoaded images file with 10000 entries\r\nhidden: 784 -> 32\r\noutput: 32 -> 10\r\nAvg Loss: 0.292608  91.009998% correct<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3fcfb28 elementor-widget elementor-widget-text-editor\" data-id=\"3fcfb28\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\tAs you can see, the accuracy dropped on the test data relative to the training data. This is a hallmark characterstic of <em>overfitting<\/em>, which is to be expected given that we haven\u2019t implemented any regularization whatsoever! That said, 91% accuracy isn\u2019t all that bad when we consider the fact that our model has no notion of pixel-adjacency whatsoever. For image data, convolutional networks are a far more apt architecture than the one chosen for this demonstration.\n<h3 id=\"regularization\">Regularization<\/h3>\nRegularization will not be implemented as part of this self-contained neural network, but it is such a fundamental part of most deep learning frameworks that we\u2019ll discuss it here.\n\nOften, the dimensionality of our model will be much higher than what is stricly needed to make accurate predictions. This stems from the fact that we seldom no a priori how many features are needed for the model to be successful. Thus, the likelihood of overfitting increases as more training data is fed into the model. The primary tool to combat overfitting is <em>regularization<\/em>. Loosely speaking, regularization is any strategy employed to restrict the hypothesis space of fit-functions the model can occcupy to prevent overfitting.\n\nWhat is meant by restricting the hypothesis space, you might ask? The idea is to consider the entire family of functions possible spanned by the model\u2019s entire parameter vector. If our model has 10000 parameters (many networks will easily exceed this), each unique 10000-dimensional vector corresponds to a possible solution. However, we know it\u2019s unlikely that certain parameters should be vastly greater in magnitude than others in a theoretically <em>optimal<\/em> condition. Models with &#8220;strange&#8221; parameter vectors that are unlikely to be the optimal solution are likely converged on as a result of overfitting. Therefore, it makes sense to consider ways to constrain the space this parameter vector may occupy.\n\nThe most common approach to achieve this is to add an initial penalty term to the loss function which is a function of the weight. For example, here is the cross-entropy loss with the so-called \\((L^2)\\) regularizer (also known as the ridge regularizer) added:\n\n\\(-sum_{xin X} y_x log{hat{y}_x} + frac{lambda}{2} mathbf{w}^{T}mathbf{w}\\)\n\nIn a slight abuse of notation, \\((mathbf{w})\\) here corresponds to a vector containing every weight in our network. The factor \\((lambda)\\) is a constant we can choose to adjust the penalty size. Note that when a regularizer is used, we <em>expect training loss to increase<\/em>. The tradeoff is that we simultaneously <em>expect test loss to decrease<\/em>. Tuning the regularization speed \\((lambda)\\) is a routine problem for model fitting in the wild.\n\nBy modifying the loss function, in principal, all loss gradients must change as well. Fortunately, as we\u2019ve only added a quadratic term to the loss, the only change to the gradient will be an additional linear additive term \\((lambdamathbf{w})\\). This means we don\u2019t have to add a ton of code to modify all the gradient calculations thus far. Instead, we can simply <em>decay<\/em> the weight based on a percentage of the weight\u2019s magnitude when we adjust the weight after each batch is performed. You will often here this type of regularization referred to as simply <em>weight decay<\/em> for this reason.\n\nTo implement \\((L^2)\\) regularization, simply add a percentage of a weight\u2019s value to its loss gradient. Crucially, do not adjust bias parameters in the same way. We only wish to penalize parameters for which increased magnitude corresponds with more complex models. Bias parameters are simply scalar offsets, regardless of their value and do not scale the inputs. Thus, attempting to regularize them will likely increase <em>both<\/em> training and test error.\n<h2 id=\"where-to-go-from-here\">Where to go from here<\/h2>\nAt this point, our toy network is complete. With any luck, you\u2019ve taken away a few key patterns that will aid in both your intuition about how deep learning techniques work, and your efforts to actually implement them. The implementation presented here is both far from complete, and far from ideal. Critically missing is adequate visualization for the error rate as a function of training time, mis-predicted samples, and the model parameters themselves. Without visualization, model tuning can be time consuming, veering on impossible. In addition, our model training samples are always ingested in the order they are provided in the training file. In practice, this sequence should be shuffled to avoid introducing training bias.\n\nHere are a few additional things you can try, in no particular order.\n<ul>\n \t<li>Add various regularization modes such as \\((L^2)\\), \\((L^1)\\), or dropout.<\/li>\n \t<li>Track loss reduction momentum to implement <em>early stopping<\/em>, thereby reducing wasted training cycles<\/li>\n \t<li>Implement a convolution node with a variable sized weight filter. You will likely need to implement the max-pooling operation as well.<\/li>\n \t<li>Implement a batch-normalization node.<\/li>\n \t<li>Modify the interfaces provided here so that <code>Node::forward<\/code> and <code>Node::reverse<\/code> also pass slot ids to handle nodes with multiple inputs and outputs.<\/li>\n \t<li>Leverage the slots abstraction above to implement a residual network.<\/li>\n \t<li>Improve efficiency by adding support for SIMD or GPU-based compute kernels.<\/li>\n \t<li>Add multithreading to allow separate batches to be trained simultaneously.<\/li>\n \t<li>Provide alternative optimizers that decay the learning rate over time, or decay the learning rate as a function of loss momentum.<\/li>\n \t<li>Add a &#8220;meta-training&#8221; feature that can tune <em>hyperparameters<\/em> used to configure your model (e.g. learning rate, regularization rate, network depth, layer dimension).<\/li>\n \t<li>Pick a research paper you\u2019re interested in and endeavor to implement it end to end.<\/li>\n<\/ul>\nAs you can see, the sky\u2019s the limit and there is simply no end to the amount of work possible to improve a neural network\u2019s ability to learn and make inferences. A good body of work is also there to improve tooling around data ingestion, model configuration serialization, automated testing, continuous learning in the cloud, etc. Crucially though, new research and development is constantly in the works in this ever-changing field. On top of studying deep learning as a discipline in and of itself, there is plenty of room for specialization in particular domains, be it computer vision, NLP, epidemiology, or something else. My hope is that for some of you, the neural network in a weekend may take the form of a neural network in a fulfilling career or lifetime.\n<h4 id=\"further-reading\">Further Reading<\/h4>\nIf you get a single book, <em>Deep Learning<\/em> (listed first in the following table) is highly recommended as a relatively self-complete text with cogent explanations written in a readable style. As you venture into attempting to perform ML tasks in a particular domain, search for a relatively recent highly cited &#8220;survey&#8221; paper, which should introduce you to the main ideas and give you a starting point for further research. <a href=\"https:\/\/arxiv.org\/pdf\/1907.09408.pdf\">Here<\/a> is an example of one such survey paper, in this case with an emphasis on object detection.\n<table><colgroup> <col style=\"width: 33%;\" \/> <col style=\"width: 33%;\" \/> <col style=\"width: 33%;\" \/> <\/colgroup>\n<thead>\n<tr class=\"header\">\n<th>Title<\/th>\n<th>Authors<\/th>\n<th>Description<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"odd\">\n<td><em>Deep Learning<\/em><\/td>\n<td>Ian Goodfellow, Yoshua Bengio, and Aaron Courville<\/td>\n<td>Seminal text on the theory and practice of using neural networks to learn and perform tasks<\/td>\n<\/tr>\n<tr class=\"even\">\n<td><em>Numerical Methods for Scientists and Engineers<\/em><\/td>\n<td>R. W. Hamming<\/td>\n<td>Excellent general text covering important topics such as floating point precision and various approximation methods<\/td>\n<\/tr>\n<tr class=\"odd\">\n<td><em>Standard notations for Deep Learning<\/em> (<a href=\"https:\/\/cs230.stanford.edu\/files\/Notation.pdf\">link<\/a>)<\/td>\n<td>Stanford CS230 Course Notes<\/td>\n<td>Cheatsheet covering standard notation used by many texts and papers<\/td>\n<\/tr>\n<tr class=\"even\">\n<td><em>Neural Networks and Deep Learning<\/em> (<a href=\"http:\/\/neuralnetworksanddeeplearning.com\/index.html\">link<\/a>)<\/td>\n<td>Michael Nielsen<\/td>\n<td>A gentler introduction to the theory and practice of neural networks<\/td>\n<\/tr>\n<tr class=\"odd\">\n<td><em>Automatic Differentiation in Machine Learning: a Survey<\/em> (<a href=\"https:\/\/arxiv.org\/pdf\/1502.05767.pdf\">link<\/a>)<\/td>\n<td>At\u0131l\u0131m G\u00fcne\u015f Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, Jeffrey Mark Siskind<\/td>\n<td>Excellent survey paper documentating the various algorithms used for computational differentiation including viable alternatives to backpropagation<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>[latexpage] Introduction Would you like to write a neural network from start to finish? Are you perhaps shaky on some of the fundamental concepts and derivations, such as categorical cross-entropy loss or backpropagation? Alternatively, would you like an introduction to machine learning without relying on &#8220;magical&#8221; frameworks that seem to perform AI miracles with only [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":3732,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[18],"tags":[],"hashtags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>C++ Neural Network in a Weekend &#183; Technical Content Marketing | ContentLab<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"C++ Neural Network in a Weekend &#183; Technical Content Marketing | ContentLab\" \/>\n<meta property=\"og:description\" content=\"[latexpage] Introduction Would you like to write a neural network from start to finish? Are you perhaps shaky on some of the fundamental concepts and derivations, such as categorical cross-entropy loss or backpropagation? Alternatively, would you like an introduction to machine learning without relying on &#8220;magical&#8221; frameworks that seem to perform AI miracles with only [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/\" \/>\n<meta property=\"og:site_name\" content=\"Technical Content Marketing | ContentLab\" \/>\n<meta property=\"article:published_time\" content=\"2020-09-24T16:33:19+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-12-14T17:30:36+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/240_F_226820318_Xgf7wwzORwdOT7OSoAFfBmIHl0dlZmRx.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"403\" \/>\n\t<meta property=\"og:image:height\" content=\"237\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"ContentLab\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@ContentLabIO\" \/>\n<meta name=\"twitter:site\" content=\"@ContentLabIO\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"ContentLab\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"52 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/\"},\"author\":{\"name\":\"ContentLab\",\"@id\":\"https:\/\/contentlab.com\/#\/schema\/person\/e1b8cacc1019e83e8d6cd9ab1a049532\"},\"headline\":\"C++ Neural Network in a Weekend\",\"datePublished\":\"2020-09-24T16:33:19+00:00\",\"dateModified\":\"2022-12-14T17:30:36+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/\"},\"wordCount\":9239,\"publisher\":{\"@id\":\"https:\/\/contentlab.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/240_F_226820318_Xgf7wwzORwdOT7OSoAFfBmIHl0dlZmRx.jpg\",\"articleSection\":[\"Portfolio\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/\",\"url\":\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/\",\"name\":\"C++ Neural Network in a Weekend &#183; Technical Content Marketing | ContentLab\",\"isPartOf\":{\"@id\":\"https:\/\/contentlab.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/240_F_226820318_Xgf7wwzORwdOT7OSoAFfBmIHl0dlZmRx.jpg\",\"datePublished\":\"2020-09-24T16:33:19+00:00\",\"dateModified\":\"2022-12-14T17:30:36+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#primaryimage\",\"url\":\"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/240_F_226820318_Xgf7wwzORwdOT7OSoAFfBmIHl0dlZmRx.jpg\",\"contentUrl\":\"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/240_F_226820318_Xgf7wwzORwdOT7OSoAFfBmIHl0dlZmRx.jpg\",\"width\":403,\"height\":237},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/contentlab.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"C++ Neural Network in a Weekend\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/contentlab.com\/#website\",\"url\":\"https:\/\/contentlab.com\/\",\"name\":\"Technical Content Marketing | ContentLab\",\"description\":\"Technical Content Marketing Done Right.\",\"publisher\":{\"@id\":\"https:\/\/contentlab.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/contentlab.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/contentlab.com\/#organization\",\"name\":\"ContentLab.io\",\"url\":\"https:\/\/contentlab.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/contentlab.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/favicon-1.png\",\"contentUrl\":\"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/favicon-1.png\",\"width\":768,\"height\":768,\"caption\":\"ContentLab.io\"},\"image\":{\"@id\":\"https:\/\/contentlab.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/ContentLabIO\",\"https:\/\/www.linkedin.com\/company\/17893315\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/contentlab.com\/#\/schema\/person\/e1b8cacc1019e83e8d6cd9ab1a049532\",\"name\":\"ContentLab\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/contentlab.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/3929e740ba8fe876f001a63220500e8e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/3929e740ba8fe876f001a63220500e8e?s=96&d=mm&r=g\",\"caption\":\"ContentLab\"},\"description\":\"ContentLab provides high-quality written articles, tutorials, courses, and other technical marketing materials to industry leaders. We create no-nonsense tech content that\u2019s purpose-built to attract, educate, and engage your technical audience.\",\"sameAs\":[\"https:\/\/contentlab.op\"],\"url\":\"https:\/\/contentlab.com\/author\/contentlab\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"C++ Neural Network in a Weekend &#183; Technical Content Marketing | ContentLab","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/","og_locale":"en_US","og_type":"article","og_title":"C++ Neural Network in a Weekend &#183; Technical Content Marketing | ContentLab","og_description":"[latexpage] Introduction Would you like to write a neural network from start to finish? Are you perhaps shaky on some of the fundamental concepts and derivations, such as categorical cross-entropy loss or backpropagation? Alternatively, would you like an introduction to machine learning without relying on &#8220;magical&#8221; frameworks that seem to perform AI miracles with only [&hellip;]","og_url":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/","og_site_name":"Technical Content Marketing | ContentLab","article_published_time":"2020-09-24T16:33:19+00:00","article_modified_time":"2022-12-14T17:30:36+00:00","og_image":[{"width":403,"height":237,"url":"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/240_F_226820318_Xgf7wwzORwdOT7OSoAFfBmIHl0dlZmRx.jpg","type":"image\/jpeg"}],"author":"ContentLab","twitter_card":"summary_large_image","twitter_creator":"@ContentLabIO","twitter_site":"@ContentLabIO","twitter_misc":{"Written by":"ContentLab","Est. reading time":"52 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#article","isPartOf":{"@id":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/"},"author":{"name":"ContentLab","@id":"https:\/\/contentlab.com\/#\/schema\/person\/e1b8cacc1019e83e8d6cd9ab1a049532"},"headline":"C++ Neural Network in a Weekend","datePublished":"2020-09-24T16:33:19+00:00","dateModified":"2022-12-14T17:30:36+00:00","mainEntityOfPage":{"@id":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/"},"wordCount":9239,"publisher":{"@id":"https:\/\/contentlab.com\/#organization"},"image":{"@id":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#primaryimage"},"thumbnailUrl":"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/240_F_226820318_Xgf7wwzORwdOT7OSoAFfBmIHl0dlZmRx.jpg","articleSection":["Portfolio"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/","url":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/","name":"C++ Neural Network in a Weekend &#183; Technical Content Marketing | ContentLab","isPartOf":{"@id":"https:\/\/contentlab.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#primaryimage"},"image":{"@id":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#primaryimage"},"thumbnailUrl":"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/240_F_226820318_Xgf7wwzORwdOT7OSoAFfBmIHl0dlZmRx.jpg","datePublished":"2020-09-24T16:33:19+00:00","dateModified":"2022-12-14T17:30:36+00:00","breadcrumb":{"@id":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#primaryimage","url":"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/240_F_226820318_Xgf7wwzORwdOT7OSoAFfBmIHl0dlZmRx.jpg","contentUrl":"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/240_F_226820318_Xgf7wwzORwdOT7OSoAFfBmIHl0dlZmRx.jpg","width":403,"height":237},{"@type":"BreadcrumbList","@id":"https:\/\/contentlab.com\/c-neural-network-in-a-weekend\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/contentlab.com\/"},{"@type":"ListItem","position":2,"name":"C++ Neural Network in a Weekend"}]},{"@type":"WebSite","@id":"https:\/\/contentlab.com\/#website","url":"https:\/\/contentlab.com\/","name":"Technical Content Marketing | ContentLab","description":"Technical Content Marketing Done Right.","publisher":{"@id":"https:\/\/contentlab.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/contentlab.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/contentlab.com\/#organization","name":"ContentLab.io","url":"https:\/\/contentlab.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/contentlab.com\/#\/schema\/logo\/image\/","url":"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/favicon-1.png","contentUrl":"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/favicon-1.png","width":768,"height":768,"caption":"ContentLab.io"},"image":{"@id":"https:\/\/contentlab.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/ContentLabIO","https:\/\/www.linkedin.com\/company\/17893315\/"]},{"@type":"Person","@id":"https:\/\/contentlab.com\/#\/schema\/person\/e1b8cacc1019e83e8d6cd9ab1a049532","name":"ContentLab","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/contentlab.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/3929e740ba8fe876f001a63220500e8e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/3929e740ba8fe876f001a63220500e8e?s=96&d=mm&r=g","caption":"ContentLab"},"description":"ContentLab provides high-quality written articles, tutorials, courses, and other technical marketing materials to industry leaders. We create no-nonsense tech content that\u2019s purpose-built to attract, educate, and engage your technical audience.","sameAs":["https:\/\/contentlab.op"],"url":"https:\/\/contentlab.com\/author\/contentlab\/"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/contentlab.com\/wp-content\/uploads\/2021\/11\/240_F_226820318_Xgf7wwzORwdOT7OSoAFfBmIHl0dlZmRx.jpg","_links":{"self":[{"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/posts\/3312"}],"collection":[{"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/comments?post=3312"}],"version-history":[{"count":32,"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/posts\/3312\/revisions"}],"predecessor-version":[{"id":13868,"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/posts\/3312\/revisions\/13868"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/media\/3732"}],"wp:attachment":[{"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/media?parent=3312"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/categories?post=3312"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/tags?post=3312"},{"taxonomy":"hashtags","embeddable":true,"href":"https:\/\/contentlab.com\/wp-json\/wp\/v2\/hashtags?post=3312"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}