Many people have converted auto engines for aircraft use, but the only two in widespread use are Volkswagen and Corvair... both air cooled engines already similar to aircraft engines. Others have been used, for example Mazda rotaries, various V-8s, Model A Fords back in the old days, and some others, with varying degrees of success. As you say, cooling at sustained high power settings is usually the biggest issue, with torsional resonance issues also being a frequent problem (broken crankshafts, etc.).