This is a fragment of a failed attempt to build an optimizer which anneals using the LP "norm" (here in scare quotes because $p$ was frequently below 1). The idea was simple (although non-functional). To create sparsity in a neural network you could use "weight decay" induced by a norm with a value of $p<2$ but in such a case $p$ will usually be very shallow for weights far from 0. Therefore, the idea was to create an optimizer which begins at $p=2$ and slowly anneals the norm to 0.
To maximize sparsity one might want to find the values of the multiplicative factor $\lambda$ (normally the actual value you would put into the optimizer, e.g. in PyTorch's Adam it is simply called 'weight_decay') which maximizes $w$ such that $w$ is stepped to 0. This was the attempt to find that. I leave it for posterity, and also for myself, since maybe one day I'll revisit this project.
To see which steps in our algorithm yield zeros in the resulting parameter vector, we consider the inequality
$$ w \leq \lambda p w^{p-1}. $$
This inequality, and the curve bounding it
$$ w = \lambda p w^{p-1}, $$
defines the weight size w below which our gradient will step to zero (given that we truncate step sizes if they would overstep zero). Now, for every $\lambda$ there is a particular value of $p$ which maximizes the size of $w$. We can find the value of $p$ which maximizes this $w$ by considering the implicit partial derivative of $w$ with respect to $p$ given by
$$ \frac{\partial w(p, \lambda)}{\partial p} = \frac{\partial}{\partial p} \big( \lambda p w^{p-1} \big), $$
which we can algebraically manipulate into the following form:
$$ \frac{\partial w(p, \lambda)}{\partial p} = \frac{\partial}{\partial p} \Big( \lambda p \, e^{(p-1)\ln(w)} \Big). $$
This will more easily render the derivative by the steps below. First we use the chain rule,
$$ \frac{\partial w(p, \lambda)}{\partial p} = \lambda e^{(p-1)\ln(w)} + \Big( \frac{\partial w(p, \lambda)}{\partial p} \cdot \frac{1}{w} + \ln(w) \Big) \cdot \Big( \lambda p \, e^{(p-1)\ln(w)} \Big), $$
then by additional manipulation
$$ \frac{\partial w(p, \lambda)}{\partial p} = \lambda w^{p-1} + \frac{\partial w(p, \lambda)}{\partial p} \cdot \frac{1}{w} \cdot \lambda p w^{p-1} + \ln(w) \, \lambda p w^{p-1}, $$
$$ \frac{\partial w(p, \lambda)}{\partial p} \Big(1-\frac{1}{w} \lambda p w^{p-1}\Big) = \lambda w^{p-1} + \ln(w) \, \lambda p w^{p-1}, $$
finally yielding
$$ \frac{\partial w(p, \lambda)}{\partial p} = \frac{\lambda w^{p-1} + \ln(w) \, \lambda p w^{p-1}}{1-\frac{1}{W}\lambda p w^{p-1}}. $$
To find this derivative's ostensible maxima, we need to factor the numerator in the previous expression to get $\lambda w^{p-1}(1+p\ln(w))$. Since we are only concerned with values $0 < p \leq 2$ we need only consider when $0 = 1 + p \ln(w)$, which are (indeed) actual maxima in this range. This gives a relationship $w = e^{-1/p}$, but how do we relate this back to $\lambda$? If we insert this value back into the curve equation we can establish the relationship between $\lambda$ and the $p$ value at which our "sparsifying region" is maximized:
$$ e^{-1/p} = \lambda p \, e^{\frac{-1}{p}(p-1)}, $$
so that
$$ \frac{e^{-1/p}}{p \, e^{-\frac{1}{p}(p-1)}} = \frac{1}{p} e^{1-\frac{2}{p}} = \lambda. $$
As mentioned before, we are interested in the segment between $0 < p \leq 2$. This is important, as solving this equation requires the Lambert W function to yield
$$ p = \frac{-2}{W\ \left(\frac{- 2 \lambda}{e}\right)}. $$
In this case, this corresponds to a segment of the left branch of the Lambert W function; in other words we are specifically interested in
$$ p = \frac{-2}{W_{-1}\ \left(\frac{- 2 \lambda}{e}\right)}. $$
The relation above means that, given any particular $\lambda$, we can select an optimal target value $p$ which our annealing process can anneal towards. This value of $p$ represents a maximization of the region around 0 that will be stepped towards 0 at the next iteration. The behavior of these relations can be observed in the figures above.