I remember someone telling me a long time ago that people in Greenland hang their clothes outside to dry at freezing temperatures without issues. No idea if that's true and I never tried. Anyway, from a theoretical point of view, this should work if the air is dry enough. This means that the water vapor pressure in the air must be lower than the equilibrium water vapor pressure at the air temperature. Or in other words: The relative humidity must be lower than 100 %, the lower the better. It may be a little counterintuitive, but the vapor pressure of frozen water is bigger than you may think, for example approx. 6 mbar at 0 °C. So there can be quite some evaporation over time even if you only use energy from the air in a well-ventilated/windy place and maybe some sunshine. Anyway, you would require some alternative clothes to wear in the meantime, so this may limit the practical use of this method, for instance during backpacking when you carry only very limited spare clothing.
Concerning hitting off frozen water, the Wikipedia page about Inuit clothing states:
Historically, Inuit used two main tools to keep their garments dry and cold. The first was the tiluqtut, or snow beater, a rigid implement made of bone, ivory, or wood. It was used to beat the snow and ice from clothing before entering the home. The second was the innitait, or drying rack. Once inside the home, garments were laid over the rack near a heat source so they could be dried slowly.
So apparently, beating off ice and snow can be a thing. However, I would suspect this will mainly work for ice/snow on the clothes surface, rather than if your clothes are completely soaked.
One word about the technical freeze-drying process because it is mentioned in the comments: To my understanding, vacuum is not used primarily to speed up the process (at least in the most simple version where you do not heat your sample), but rather to keep whatever you dry frozen even if your room temerature is 20 °C or so. Too low pressures will indeed slow down drying because equilibrium temperatures and vapor pressures are very low then and only very little evaporation occurs. No need to carry a freeze-drying apparatus on the trail, luckily.