In this lecture, we will briefly review neural networks and computation graphs. Most of this material should be familiar the class, either from other courses or from working with these ideas.
Lectures
Readings and References
Matrices and Tensors
-
The Matrix Cookbook is an invaluable resource for quickly looking up identities, operations, approximations, etc involving matrices and vectors.
Computation graphs
-
An introduction to computation graphs, by Christopher Olah.
-
Goldberg, Yoav. “A primer on neural network models for natural language processing.” Journal of Artificial Intelligence Research 57 (2016): 345-420. See section 5.
-
Network and loss building blocks in PyTorch:
torch.nn
Automatic differentiation
-
The Wikipedia article on automatic differentiation is quite thorough.
-
Griewank, Andreas. “Who invented the reverse mode of differentiation?” Documenta Mathematica, Extra Volume ISMP 389400 (2012).