Who developed all those Nazi tricks, such as
Ordering the victims to bring possessions with them
Selecting only the tall ones one day and only the short ones the other day so to make impression it is for some kind of work rather than execution.
Asking the victims to write letters to the relatives before being executed.
Make the victims complicit and feel guilty, asking mother to chose which of the two kids to kill, making son to kill father etc.
Elevating and giving uniforms to some victims so they to help.
Issuing "Jewish money" in the ghettos.
Distributing Jewish property to the locals so they to feel complicit as well.
Special shops for Ukrainians in Poland.
Making some non-German people to sign papers that they were Germans so to always feel in danger.
I think some of these techniques are very complicated. For instance, no other prison in history ever enjoyed such level of autonomy and self-government as Nazi concentration camps not to say, ghettos, even though in most prisons people are not killed and usually get released.
How the Nazis managed to make people to help kill themselves, hand over all possessions, report their saviors, trust what Nazis were saying etc?
The question is not limited to treatment of civilians but also the way Nazis made Germans to fight till the end (possibly making all the army complicit in atrocities), how they made foreign leaders to trust them and perceive invasions as "provocations", possibly by makinge foreign powers (Poland, the USSR) complicit as well in partitioning other countries before being invaded themselves etc.
Who developed such a great range of various deception and compelling techniques? Is is somehow connected with how they treated mentally ill? Were there any known psychiatrists who studied human behavior behind this?
The Big Lie was a technique and phrase (German: Große Lüge) coined and described by Adolph Hitler in Mein Kampf in 1925. (My emphasis)
All this was inspired by the principle—which is quite true within itself—that in the big lie there is always a certain force of credibility; because the broad masses of a nation are always more easily corrupted in the deeper strata of their emotional nature than consciously or voluntarily; and thus in the primitive simplicity of their minds they more readily fall victims to the big lie than the small lie, since they themselves often tell small lies in little matters but would be ashamed to resort to large-scale falsehoods. It would never come into their heads to fabricate colossal untruths, and they would not believe that others could have the impudence to distort the truth so infamously.
Hitler was undisputedly mad but, in his madness, was an inspired orator with a natural intuition of how to leverage crowd psychology for his own evil ends. He also, as a flame attracts moths, attracted to himself those like Himmler with similar intentions, intuition, and madness, and who in turn were able to implement and manage the petty details of the Nazi Party's evil agenda.
So to address your penultimate question ("Who developed such a great range of various deception and compelling techniques?") it was not a great range of compelling techniques, but a very simple and narrow technique - simply tell the biggest and most outrageous lie imaginable about your intentions and actions that claims to soothe the listener's fears.