If you are working on potato workloads, upgrading to an M5 isn't likely to make a big difference either. The issue that many are facing is that they run potato workloads and an M1 has been fast enough for a while, and a several generations old iPhone or Android device was fine as well.

Even if the machine is 100% faster, if the workload doesn't need that speed you didn't need to upgrade and in reality most of the stuff that people do can be performed by much older machines.

That said, there are other reasons than raw CPU that people upgrade. Battery life, weight, aesthetics, GPU performance, etc.