Sounds like the Controller screwed up to me. He told the guy he'd call his crosswind turn then never did. The pilot called back twice and ATC ignored the call. All the sudden it was the pilots fault he was going into Burbank's (?) class D?
Finally the N560 calls downwind and ATC tells him to extend downwind, I'll call your base. Pilots response should have been: Fool me once, shame on you. Fool me twice, shame on me!