Manly straight handlers on S swell

You're correct that Manly faces east, but a south swell will still get in, just reduced in size and energy.

Any south swell that reaches the beach has refracted around Blue Fish Point and Fairy Bower, and in the process of bending 90° it's forward speed has slowed while the lines have become more uniformly spaced.

With random packets of energy smoothed out, swell lines approach the beach in one long line, losing their 'peakiness' and the tendency to form shoulders.

Also, during north and east swells the bommie comes into play with swells passing over or near the submerged reef refracting in various ways breaking up straight lines of swell.

well, that's the fancy explanation. but here is the real reason -- manly is the bondi of the northern beaches, hence it closes out in southerlies once it hits two foot.