Both broccoli and cauliflower usually does very well on clay soil. While you of course should amend the soil for them as well, if you have to prioritize I'd put the most effort into the area you will be growing carrots and potatoes in. In particular it can be a bit problematic to grow good quality carrots on heavy clay soil.
Not sure if you have researched which cultivars you are going to grow or not, but as it has impact on how much you will need to improve the soil, I'd like to mention that there are both carrots (often a bit shorter and thicker) and potato cultivars better suited to clay than others.
