You don't have to look too hard at the engines we're flying behind/between and literally every other engine out there to see where things have improved over the years. Most of the improvements that we've seen in the automotive space would help in aviation, but not all. Few basics:
1) Electronic fuel injection and ignition, modernize the fuel and ignition system. Basically multi-port EFI and coil-near-plug/coil pack ignition. Either offers an improvement. Direct injection could also be used instead.
2) Improved combustion chamber design in the cylinders. There are many out there that have come about over the decades. I have a few favorites which are not necessarily ones you see in production today, but given the mixture range that we use in aviation and the lack of emissions requirements, I don't think the newest is necessarily the best.
3) Increased compression ratios.
4) Modern turbocharger designs or supercharger designs. Even the newest aircraft turbos are a very old design. The improvements in design and efficiency are immense and will improve overall performance.
5) More optimized/thought out intake design - again more about efficiency and equalizing the power output.
I have a couple things that I would intentionally not do that have become commonplace:
1) Variable valve timing/variable lift/variable intake: Adds weight and complexity and really is only helpful when you're trying to optimize across a rev range. Aircraft engines do not have a broad enough rev range.
2) Overhead cams. OHC engines are heavier than pushrod engines in effectively all cases, and generally make a larger/heavier engine for the same displacement/horsepower. Obviously devil is in the details, but I'd stick with pushrods. Also, pushrods work fine for the low revs that are well suited to aircraft engines.
Note that none of this is really about improving horsepower out of the engines, or at least not about improving rated horsepower. Sure, you could do that, but you can realistic do that today in other ways (such as just turning up the boost). It's really about improving efficiency, ease of use (single lever control), and performance at altitude. What that turns into is a more efficient engine, and enabling use of lower octane fuels than 100LL, and helping with getting the lead out. Of course the obvious problem there is universal adoption required, one of the big problems the whole 100LL replacement issue has had since before I started my career.
There are a lot of details relating to that which I won't go into here, but I do have a consulting fee for interested companies.
