Calling a girl a lemon over here is apparently very offensive - I have no idea why

Dunno about the IOM, but lemon is cocker-nee rhyming slang - easy-peasy, lemon squeezy. Denoting a lady who is a little free with her affections?
I've read plenty of people on here complaining about Paul Cullen's use of language

Are you sure he didn't say the person was 'indifferent to the range of possible outcomes"? Didn't give a toss doesn't sound very cullenesque?