>> I think the obstacles to progress you are talking about in science are what Thomas Kuhn called Paradigms.
Exactly, and I might have read the "dinosaurs / die off" comment in his book.
>> the failure of much of the scientific community to understand living systems
Good point. But you might be able to expand that with some validity.
"everyone's failure to understand complex systems"
"the difficulty of understanding 'systems' at all"
Natural systems are much more complex (and less intuitive) than invented systems like computers.
I think living things are much more complex than any invented things, and systems of living things - forget about it! my ambition would be to learn a little about them, and maybe in some millennium, get a handle on what NOT to do to them ... in the meanwhile, admit how LITTLE we understand them and approach them with humility and intuition as well as our best analytics and modelling. But humility first..
The author of "The Mythical Man-Month" pointed out that it was (at least) three times harder to develop a system as it is to develop a single product. it's hard enough to develop one complicated thing, but when you have so many things that you have a whole, interacting SYSTEM of them, the complexities transcend comprehension (or at least 'management').
Almost everyone has trouble thinking about multiple things at the same time, or guessing at what interactions might exist between and among multiple things.
Software engineers handled it in part by inventing a whole new field of "Systems Engineering". We let THEM worry about the larger, harder issues.
I don't know of anything more complicated than living things, except for systems of living things (e.g. ecology / environmental systems). or perhaps the combination of ecology, public policy and economics.
The two most important requirements are lacking for comprehending those: sufficient humility and sufficient intelligence. Bacteriologists are humble enough to admit that we don't fully understand bacteria. Virologists (I think) still have enough humility to admit that we have a lot to learn about most viruses. Chemists who pay attention to electron orbitals are very up front about admitting that the only atoms they can describe analytically is one neutral hydrogen atom - everything else is approximations and rule-of-thumb estimates.
But I agree with you - many doctors are too arrogant to admit there's anything they don't know.
And if you pay a publicist, he's SURE he can prove whichever side of any environmental issue you want him to prove.
And if you believe politicians (pause for sickly laughter), they know what's best for everyone.
