If your 172M is powered by a Lycoming O-320E2D (came stock with this engine) . I've flown an E2D powered 177 for years and here are my thoughts;
(1) The E2D in in many Cessnas came with ONE primer nozzle installed in 1 out of the 4 cylinders, so the manual primer is only priming the #4 cylinder (or is it #2?). Doesn't seem optimal does it? It's NOT. IF equipped better it will have a maximum of 3 primer nozzles, which yield much quicker starts.
(2) The ED2 is likely equipped with Slick magnetos, and only 1 of them is firing when you crank the engine with starter, meaning only 4 of the 8 spark plugs fire while the ignition switch is held in the "START" position, again, not optimal. If that magneto is weak it may be nearly impossible to get the engine started.
(3) Slick magnetos are known for a higher "coming in speed", meaning where it fires all spark plugs continuously without skip. IRRC the coming in speed is ~200 RPM for Slicks vs ~150 RPM for Bendix magnetos. So Slicks aren't ideal for easy starting engines.
(4) Back to the "firing 4/8 plugs during start", if there is fouling or contamination of one or more of those plugs its difficult, especially if the plug located with the manual primer nozzle is shorted, priming won't help.
(5) The carburetor is equipped with an accelerator pump which literally squirts fuel into the intake manifold when you advance the throttle quickly. This in effect can "prime" all 4 cylinders. Since it increases the risk of carburetor fires by loading the intake manifold and everything else up with gas, it's advised to only pump the throttle DURING engine cranking. Some people will flame me for this one but it can save a lot of battery and starter abuse. Leaving the throttle at higher settings between 1 or 2 pumps will help pull more fresh air saturated with fuel into the cylinders. As she starts to pop, slowly pull the throttle to near idle and it will go.
I get perfect starts and sometimes crappy starts. I know how it all works, flown it for years, converted to dual impluses so all 8 plugs fire during cranking, new carburetor, new mags, new harness, and new plugs, it still isn't as easy to start as many other aircraft, but sure runs good.
Hopefully that gives you some insight as to what is occurring when you start. Don't forget that fuel has to vaporize. With the E2D two shots of primer with a warm engine, wait a few seconds then crank. Pushing the primer harder will get a finer mist of fuel from the primer nozzle(s) for better vaporization than slow & easy.