Is lambda * W the partial derivative of just L_reg with respect to W?

Yes! That's all there is to it.