moving them from one part of garden to another? seems fine. Assuming the raised bed is still connected to the ground, not elevated.
If you add organic matter to the raised bed, the worms will find it on their own. And if they don't like the raised bed they will leave.
