Despite the fact that the mistrust and hate between Harry and Snape is an important theme of the whole series, I do not fully understand Snape's behaviour in OOTP from a logical point of view, especially during the Occlumency lessons.
I collected some facts from the whole series to make my points at the bottom clearer. I am sorry that I can not provide quotes, since I only have the (unabridged) audio books.
General Snape facts:
Situation in OOTP:
So, in summary, Snape should have been motivated as hell to teach Harry Occlumency properly to help the order and to protect Harry. For me, that includes being encouraging and helpful, not behaving as a "slimy git" and putting Harry under pressure with dislike and insults.
Even when Harry is caught using the pensive to explore Snape's memory, Snape could have exploited the situation to make Harry less biased towards him and straighten out their relationship in order to teach him Occlumency properly (i.e. controlling his understandable rage). The cognitive dissonance caused in Harry by seeing James and Sirius bullying Snape even let him take the risk to break into Umbridge's office just to have a chat with Sirius.
Are there any facts I missed that explain Snape's behaviour in a logical way?
Note: I hope I pointed out clearly enough that for me, Snape's general dislike or ambiguity towards Harry does not explain Snape's (irrational) behaviour convincingly.
Further note after comments and ASH-Aisyah's answer:To me, Snape's behaviour seems to be inconsistent. He manages to stick to Dumbledore's plan all the time and against all odds, but does not in this particular and critical case. Regarding his pretended loyality towards Voldemort, he could have lied that someone else told Harry Occlumency, if Voldemort found out.
There are a few reasons that you might have missed out:
Firstly, Snape hated Harry. It's not just ambiguity or even general dislike. He truly hated Harry. Yes, in the end, he came to care for him, but only because he was Lily's son. It's partly because of the fact that Harry was a carbon copy of his hated rival, James. But at the same time, Harry had Lily's eyes, which must have been a constant reminder to Snape that the love of his life married someone else, and then died. Also, Snape might very well have resented Harry for surviving, instead of Lily. I mean, Voldy was going for Harry, but not only did Lily and James die instead, but Harry lived on to torment Snape.
So despite whatever grand plan Dumbledore might have roped Snape into, Snape just couldn't shake the fact that he absolutely hated Harry. In fact, up to the end, he only partly was affected by the news that Harry had to die, on Harry's own account. The main reason he's been saving Harry and he in the end didn't want Harry to die, was because he thought they were keeping Harry safe for Lily. But now 'raising him like a pig for slaughter' seemed blasphemous towards Lily who gave her life for him.
Secondly, Snape still needed to keep up the pretense that he was working for Voldy. Yes, he was instructed to teach Harry Occlumency. Which he partly did. At least to some extent. But if he had truly done a good job at it, and Voldy tried to break into Harry's mind, he might have found out that Snape had taught Harry well, and thus would have suspected Snape and not trusted him anymore. Just like how Snape tried to look into Draco's mind in HBP, but couldn't, and discovered that Bellatrix had been teaching Draco Occlumency.
Snape had to walk a very fine line between protecting Harry, and making sure it looks like he's trying not to. There was one particular scene in the book where Harry was complaining to Ron and Hermione, saying that he doubts Snape is actually trying to help. They were wondering if Snape was actually trying to make it easier for Voldy. And if Voldy had found that thought in Harry's mind, he would have trusted Snape more.
So Snape had to deal with walking that very fine line, while at the same time hating Harry and James and missing Lily, plus the general stresses that Dumbledore was putting on him. I guess it's enough to make anyone caustic. Plus, 'encouraging and helpful' seems completely out of character for him, whereas 'slimy git' practically defines Snape.
With regards to the Pensieve, trying to get chummy with Harry would have gone against his plan for Voldy. It would look like he was befriending his boss's enemy, even if he could stomach it. My guess is it was simply a convenient excuse to stop the Occlumency lessons.
"You raised him like a pig for slaughter!"
This sounds more like Snape feeling a little sorry for the kid, as he would probably have felt for any poor guy in such a situation, and not really due to any special care for Harry
Snape may or may not have really hated Harry, but he sure did dislike him, and also thought him to be academically incapable. He most likely truly believed that Harry would not be able to learn occlumency and block out Voldemort from his mind. Note that Harry notices his dreams about Voldemort becoming more frequent after the occlumency lessons. I don't think this has anything to do with Snape trying to sabotage him, rather, it's probably the side-effect of the pressure of the lessons, coupled with the fact that Harry wasn't very good in occlumency to begin with
Given Snape's personality, from the description of how teaches his class normally, I don't think he would have taken the approach of being "encouraging and helpful" towards his students, when he really wanted them to learn, rather he would be more likely to apply pressure and intimidation as a form of motivation. Moreover, he would never make it look as if he felt any affection for Harry, especially until Harry learned occlumency because during this period Harry was more vulnerable to Voldemort invading his mind, and so they could not afford him sensing anything like affection from Snape towards Harry (remember what Voldemort programmed Wormtail's arm to do when he felt the small sliver of mercy)
Hence, Snape's behaviour, as far during the occlumency classes, does not seem implausible given his personality
What I find more illogical is Snape actually giving up on teching him occlumency solely based on his dislike of Harry and James, given the importance of the occlumency lessons. He was, after all, like you pointed out, "a very rational and controlled character"