If it were only potting soil, I'd say no, but if it's mixed with compost and some top soil, I think it's fine. My raised beds are a mix of leftovers... various amendments and compost, used potting soil, amended garden soil...
Keep in mind that potting soil on its own lacks the nutrients that soil and compost contain, which is why plants in pots need regular fertilizing. Your aim will be to create nutritious soil in your bed, keeping it well draining, so maybe don't add heavy clay soil.