That's not technically true, but it is difficult to not do otherwise in practice given the lack of uncontrolled airspace in the US.
But even when operating in less than VMC in uncontrolled airspace without a clearance, you still are operating under IFR and must follow all the IFR rules not specifically exempted.
But the truth of the matter, "less than VMC" forces you into following the IFR rules (or the flight is illegal), but otherwise the meteorological conditions do not define instrument flight rules.