Downshifting on a 70's 3 speed is very different than a modern 5 or 6 speed. Back then, the gear range, materials, craftmanship, and technology was much different than today.
That said, I'll refer to something my dad used to say about mechanical anythings:
Every piece of equipment has a finite number of uses and you don't know what that number is. The less you use it, the longer it will last.
Technically, using anything mechanical is damaging it... It's just a matter of how much damage. Downshifting won't inherently cause major damage (unless you do it wrong), but your transmission WILL last longer if it is rolling in neutral vs rolling in gear.