Clearly you can't cut it back much more!
Try it and let us know how it works for you.
Three to four hours of morning sun and whatever sunlight gets through the poplar should be fine. The shade from the fence may be "dead shade" if no light gets through.
