I can't speak to twin engine stuff, but in a single engine I believe that Vy is the number to use once any obstacles have been cleared at Vx. I fly Vy to about 500' - 1000' AGL, the lower the nose a bit for better visibility, cooling, and groundspeed. I don't go to a true cruise climb until I have about 1000' to 1500'.
As was said above, in a single, altitude is life if the engine fails. Consider that if you double your altitude, you get four times the landing options since the area you can reach goes up with the square of your gliding distance. You also increase your time to get a mayday out, secure the engine, and generally breath and think a bit.
It has been pointed out that it's all about energy. If you're flying low and fast you can pull up and convert some of that speed into height. That works a lot better in a modern high performance glider than it does in a typical light single.
Consider that every second that you're flying you're also losing energy. The faster you fly, the faster you lose it. In fact, the rate you lose it goes up with the cube of your airspeed (drag goes up with the square, power is drag times velocity, so the energy lost per second is proportional to the cube of velocity. That assumes I'm flying a bit faster than my best glide speed so that parasite drag dominates). That means that you're much more efficient at lower speeds.
When I start a take-off run, I put in full throttle and basically leave it there. Ignoring the change in power output of a fixed pitch prop as I accelerate, the engine's basically putting energy into the airplane at a constant rate, regardless of my speed. At the same time, drag is taking it out at a rate dependent on my airspeed. If the two rates are matched, I can't accelerate or climb. If I have an excess of power, I can choose to put it into kinetic energy or into altitude. If I choose to put it into more speed, then the rate at which energy is taken out by drag goes up, so that after 2 minutes, for example, I'll have less total energy than if I had climbed at a lower speed for 2 minutes.
Now lets say that after 2 minutes, my engine fails. If I've used a cruise climb, I have less energy overall and I'm lower. So the first thing I want to do is trade some speed for altitude. However, I'm going pretty fast, so as I pull back and start to slow down (trading potential energy for kinetic energy), I'll still be losing energy at a good rate. As a result, I started with less total energy (due to the increased drag during the two minutes my engine was working), and I lose even more as I trade speed for altitude. I'll end up at a lower altitude than if I'd stayed at Vy for that 2 minutes.
As far as the question of how this would be modified with a very long runway, I'm not sure it changes much. Let's say your engine quits 15 seconds after liftoff. If you stayed very low (in ground effect where drag is very low), your total energy will be higher than if you climbed at Vy, and you'll have used up more runway in front of you, so it may actually be harder to land back on the runway before you run out. If you use a cruise climb, you'll have less total energy at that point (which is good in this case), but you'll have used up more runway. Not sure if you come out ahead or behind.
The problem is when your engine fails just as you leave the very long runway behind. Then it doesn't matter if it was a short or long runway, if you were climbing at Vy you'll be higher and have more options and more time. Both will increase your chances of survival.
Chris