Linear and logistic regression are much easier (and less error prone) to implement from scratch than neural network training with backpropagation.

That way you can still follow the progression I suggested: implement those regressions by hand using numpy -> compare against (and appreciate) sklearn -> implement SVMs by hand using cvxpy -> appreciate sklearn again.