This SpringerBrief describes how to build a rigorous end-to-end mathematical framework for deep neural networks. The authors provide tools to represent and describe neural networks, casting previous results in the field in a more natural light. In particular, the authors derive gradient descent algorithms in a unified way for several neural network structures, including multilayer perceptrons, convolutional neural networks, deep autoencoders, and recurrent neural networks. Furthermore, the developed framework is both more concise and mathematically intuitive than previous representations of neural networks. This SpringerBrief is one step towards unlocking the black box of deep learning. The authors believe that this framework will help catalyze further discoveries regarding the mathematical properties of neural networks. This SpringerBrief is accessible not only to researchers.