Let's discuss-- Part 103.17-- ultralight flight in Class-E-to-surface "extensions"

[martymayes]

You can legally fly there VFR--without speaking to anyone if you so desire-- but you can't skim the cloud-tops or cloud-bases like you can in Class G. So, there's no reason that these airspaces are fundamentally incompatible with ultralight activity, so long as it is carried out in prudent manner

I think that is what the FAA was addressing in the Part 103 preamble - some ultralight operators were being less than prudent, a NMAC in IFR conditions while in controlled airspace, was one of the specifics. A few bad apples ruined it for everyone.