Clipping gradients to 5 can still leave a norm of 7.07

For a gradient of (6, 8), clipping each component to 5 gives (5, 5), with an L2 norm of 7.07. Clipping the norm to 5 gives (3, 4), preserving the direction. In PyTorch, clip_grad_norm_ also treats the parameters you pass in as one combined gradient, not a separate limit per tensor. I’m building Glacius, a visual app for learning the math behind ML. We made a worked example with diagrams and Python to make the distinction easier to see. submitted by /u/CarelessAlps [link] [comments]