The other day I stumbled upon a youtube video from Bad Internet series: Which of The “Friends” Are You? and it made me remember a previous occurrence of an idea that people are sorted into their roles by some algorithm:
The second one refers to the Futurama series, where people appear to be forced into their jobs.
The problem: Both worlds are a joke. I would like the same, but serious.
Is there a way to get to a situation where everybody (i.e. more than 65% of the population) is selected to do their job, while there is no magic?
Other than this, I set your arms free.
I already have a guess that such a society has to start with an apocalypse and/or long war. I also guess that such a society will be a form of dictatorship. You cannot have democracy in this setup.
I even play with the idea of a theocracy, where such a system will appear to be magical on the outside (You have been selected by God!), but will have some strong computer running somewhere in the background.
The society where each person should work according its role is called the caste system. This worked in India for ages (from 1,000 BCE until the modern era). Most people accepted this system. Although today this is redundant, but it happens especially in villages. I suppose it works mainly because of religious reasons.
It's real, although the role of the people is defined by their birth, not by their personal characteristics.
In the fantasty genre there are many worlds where the system defines which people are the best and assigns their work/role/social rank according to it. One of them is Gattaca with a
society that is driven by eugenics.
What the society needs to establish
sorting is just some kind of danger, possibly in the past. We should act in this manner, because if we don't we'll die. This works with any idea.
TLDR: Track information of generations who had any choice, see where the indicators were that someone would be good at their job. Then apply those rules to new generations, offering incentives to take the job they're predicted to be best at.
In a way I think we do have something like this (though perhaps in a more general way and with the impression that we do have choices). Education pushes people along certain paths and if you choose one path rather than another the general impression is that the other careers are shut off to you now (enough that more than 65% of people stick to the career path their education indicated).
Now I suppose it depends on at what age you want to make the choice of which job someone gets.
You don't want to take away the choice...but you want it to be much easier for them to follow that path. I think people are less likely to choose differently if they think there is an option than if they're told they don't.
I'm not 100% sure what the algorithm would measure...performance in school subjects, attention span, activity, hobbies. It really depends on what information you have on them. Maybe the population is chipped and their vitals are monitored (for health reasons, obviously), we can then track that maybe they're happier doing math than their grades would suggest and, comparing to generations before, we find that actually those people are a lot more efficient workers than those who breeze through exams but couldn't care less for the actual subject.
There are plenty of ways to do this. As AlexP mentions in his answer under Emperor Constantine fathers had to be succeeded by their sons under harsh punishment. Of course such a sorting is at best ineffective. While there is some hereditary favor for certain professions it's small. For example families with surgeons also tend to have a history of butchers. But not the majority, just enough to make an anomaly.
Regardless, it's a poor way of sorting your people for professions. It's much better to take a individual approach. The most basic way would be (personality) tests. Capacity tests could show strength and weaknesses. This could be manditory in highschool. An easy compulsive filter.
If you want to take it further you get to genetic testing or even engineering. With the right tests you can see who's physically adept to what. Olympic athletes tend to deviate from the average human in a way that allows them to excel in their sport. A similar approach could work on a genetic level for professions.
Filter on things like short term memory, eye sight, upper body strength and spatial awareness. Combine that with required roles in your state controlled economy and you should have an idea of what to do.
If you want to go even further you could add another step of genetic engineering. When pregnant the system determines what's needed and will edit the baby's genome where required to be the perfect fit. This is, of course, completely dystopian.
I am outright shocked that no one has mentioned the Chinese in history with their Imperial examinations and their Rank system that formed the basis for the Chinese Meritocracy that roughly lasted for about 2200 years or so.
So yeah, it has been done and can easily be considered to be one of the most successful systems that humanity has tried. Having it be done in an algorithmic fashion with a computer and being even more all encompassing (if that is even possible) than what the Chinese used to do could go a long way in preventing some amount of corruption and having a better functioning system.
The main thing you'll need for such a society is for there to be nowhere else that disaffected citizens can realistically go if they don't like their assigned role. It needs to be a closed society. Otherwise, it will fall apart.
Allowing quantum computers, I'm surprised no one mentioned genetic sorting.
When the algorithm is first conceived, there will only be "predictions" on newborn babies, based solely on the genes. The research is secretly carried out, and soon a database of combination genes affecting personalities are created (based on "right and wrong predictions").
Based on this personality database, new step: tests on newborn babies are planned, that is predicting the behavior, job, and success of these babies. The results are, again, processed by the quantum computer, and its algorithm refined.
Soon, the precision increases to 80% and rising, and the news are spread, and people are asking for job advice to this computer, and the algorithm is refined once more, from the people unsatisfied with their current jobs.
As soon as the precision strikes 95%, the government decided it's better for people to be told what they will work when they become an adult. There are resistances, but the plan continues. The government and an independent organization perform secret eugenic operation throughout the world to eliminate potential rebels (by examining the rebellious personality genes).
Then people will just follow what they are told to do, because they believe what the computer say is best for them, according to their personality.
Note: I know the story has not taken many factors affecting the course of history into consideration, but I just want to give example how a society with predetermined job can develop and also how the algorithm is believable.
Isaac Asimov tackles this exact subject in his short story Profession.
Assume there's a computer capable of analyzing and reconnecting neurons in the brain. It's main use is to teach children to read, test which profession suits well with a brain of an adult, and teach that profession in minutes.
There's a catch: using the machine to learn is expensive.
In Asimov's story, there are multiple inhabited planets, and the Earth is the only place where there are learning machines. People on Earth learn to read at 8 and to perform a job at 18. The other planets then hire skilled workers from Earth.
People are so used to learn instantly that they won't learn from studying. They see it as wasting time, and they can't afford to learn another profession. Therefore once they learn a job, they stay doing that job.
It is actually fairly likely. Professional mobility allows the economy and society to deal with economic and technological disruptions by reallocating "human resources" dynamically. A strong future government might simply avoid such disruptions thru other means.
The first thought for most is probably some sort of centralized planned economy, but a free market economy that is large and diverse enough with strong regulations keeping the markets free and fair could be quite stable. Economies do tend to be self-stabilizing over time, the issues tend to come as consequences of technological disruptions or external factors such as epidemics or colonialism.
If we assume that your empire has reached some sort of technological plateau where disruptions from technology are manageable (and upgrading the skills of employees to keep up with progress is already normal) and the empire is secure otherwise and only expands organically (no colonialism or crusades), we could reasonably assume an economy and society that would be stable, even stagnant from our own viewpoint.
In such society there would be no particular need for people to ever change jobs. You could simply assign jobs by lottery or profession of parent at birth and then reassign if the person turns out unsuited for the job or shows particular talent or interest for something else. Since your people would all be trained for their assigned job from birth, they would be well qualified for it, familiar with it, comfortable with it, and predisposed to find it valuable and rewarding (unless the training is incompetent due to corruption or decadence, if you want a falling empire story).
This should easily keep more than 65% percent in their assigned jobs, even without particular legal or social pressure. People with the drive to start their own business or with special talents that make reassigning them necessary do not make up 35% of population. Similarly, normal jobs for normal people do not have stringent requirements that would make large portion of population unsuited for them. And with future science most such issues could and probably would be fixed.
This obviously assumes that the system works properly. Even without external disruptions from technology or expansion the system will fail eventually if quality of training and education collapses, if the assigned jobs cease to be meaningful and rewarding, or if the system becomes too strict to allow the amount of social mobility that is actually necessary. These will eventually lead to increasing inefficiency and collapse. But the failure would have to start from the political side of the society becoming too corrupt to upkeep the system.
The teens of your world will get a general recommendation for their future after their 10th year of education. Afterwards they will rotate apprenticeship in 12 different jobs in their recommended field for a year (one job per month). Should society have a heightened need in certain field, the bias to get sorted into that field will (surprise!) rise accordingly.
The masters (if thinking medieval) will then select between them, who they will take on as a real apprentice for training. In more modern times your bosses will give recommendations which will decide your future (training on the job, university education, ... in the given field).
You cannot study something if you were not pre-selected for eligibility. You could still try something in an entirely self-taught way, but the majority won't take this risk. Especially, since you cannot prove your worth in the given field without the papers you received after proper training/education. Who would trust you except some risk-takers or people with dire need?
=> More than 65% would stay in their chosen job.
The way to be sorted into your general field would be part educated guessing by your teachers, supported by (if modern times) some kind of artificial intelligence working in the big data field. I don't necessarily mean a self-aware AI, the contemporary AIs of Google, IBM, Wolfram, ... should already be enough if trained properly.
The initial part of this answer was in part inspired by the Riftwar Saga by Raymond E. Feist.
Expanding the Myers-Briggs Personality Test
The Myers-Briggs Type Indicator (MBTI) is an assessment that is believed to measure psychological preferences in how people perceive the world and make decisions. According to the Myers-Briggs test, there are 16 different types of personalities. The test consists of a series of questions, and your answers determine what type of personality you have and provides general assumptions about how your personality type is best suited for success in terms of careers, communication, etc.
So, in MBTI, each personality usually have a few careers choice Maybe in the future there would be a great Psychologist which expanding these 16 personalities into say, 132, maybe. And then some another great Psychologist expanding these 132 personalities into much more. And from those personalities, each one only have one career choice.
All kids at 10th or so grades would take this test to determine what their future job/role would be. From that time forward, all they study in school would be about the future job/role.
Like ADS stated earlier, the caste system sounds like it would be a good fit for your requirements. You could also throw in some deep sense of dedication to a greater cause among the population to make it more believable and/or give a reason why individuals accept that system. Plus, if your setup takes place on extended periods of time, members from different castes could have particular physical attributes due to a possible inter-caste breeding restriction.
I'd advise you to check out the Tau caste system from the Warhammer 40.000 series. Maybe you'll find interesting leads for your setup.
Coming from Anime SE, my answer would be, have a supercomputer that can identify your strength and weaknesses and the assign you or at least give you some options on what career you should have.
Your world would need to have the system start with a small population and then prove and publish the result, that the system can correctly assign people to jobs they do best based on their personality and abilites.
After continuous publicity on the system's ability to correctly do such a thing, people would start to accept the system more. Then, after the majority of the society accepts and believe in the system, start enforcing the system so that it has more power over the society by doing propaganda, i.e. people that ain't using the system are hardly successful, that they would only bring trouble to companies, etc.
After several decades, the system would have grown into the de facto and de yure guide for people's lifelong career path.
See Psycho Pass anime for an example of how such a system can work.
The easiest way would be with tests e.g. a personality test to see whether someone is right brain or left brain or type A etc. and an IQ test to see whether they should be a scientist or something like a taxi driver.