Comment on Can you think of any now?

missfrizzle@discuss.tchncs.de ⁨1⁩ ⁨week⁩ ago

• bluemellophone@lemmy.world ⁨1⁩ ⁨week⁩ ago

  Before LeNet and AlexNet, SVMs were the best algorithms around. People used HOG+SVM, SIFT, SURF, ORB, older Haar / Viola-Jones features, template matching, random forests, Hough Transforms, sliding windows, deformable parts models… so many techniques that were made obsolete once the first deep networks became viable.

  The problem is your schooling was correct at the time, but the march of research progress eventually saw 1) the creation of large, million-scale supervised datasets (ImageNet) and 2) larger / faster GPUs with more on-card memory.

  source

  • missfrizzle@discuss.tchncs.de ⁨1⁩ ⁨week⁩ ago

    HOG and Hough transforms bring me back. honestly glad that I don’t have to mess with them anymore though.

    I always found SVMs a little shady because you had to pick a kernel. we spent time talking about the different kernels you could pick but they were all pretty small and/or contrived. I guess with NNs you pick the architecture/activation functions but there didn’t seem to be an analogue in SVM land for “stack more layers and fatten the embeddings.” though I was only an undergrad.

    do you really think NNs won purely because of large datasets and GPU acceleration? I feel like those could have applied to SVMs too. I thought the real win was solving vanishing gradients with ReLU and expanding the number of layers, rather than throwing everything into a 3 or 5-layer MLP, preventing overfitting, making the gradient landscape less prone to local maxima and enabling hierarchical feature extraction to be learned organically.

    source

    • bluemellophone@lemmy.world ⁨1⁩ ⁨week⁩ ago

      No, you are correct. Hinton began researching ReLUs in 2010 and his students Alex Krizhevsky and Ilya Sutskever used it to train a much deeper network (AlexNet) to win the 2012 ILSVRC. The reason AlexNet was so groundbreaking was because it brought all of the gradient optimization improvements (SGD with momentum as popularized by Schmidhuber, and dropout), better activation functions (ReLU), a deeper network (8 layers), supervised training on very large datasets (necessary to learn good general-purpose convolutional kernels), and GPU acceleration into a single approach.

      NNs, and specifically CNNs, won out because they were able to create more expressive and superior image feature representations over the hand-crafted features of competing algorithms. The proof was in the vastly better performance, it was a major jump when the ILSVRC was becoming saturated. Nobody was making nearly +10% improvements on that challenge back the , it blew everybody out of the water and made NNs and deep learning impossible to ignore.

      source