Progress in science (and society) is made by taking note of details that a previous theory is unable to account for.
The history of modern science testifies to the truth of the above statement. But it also applies to social/political development and progress.
Every system or theory, whether scientific or political, has rules. The Torah, with its complex set of precepts for living a good life, has a myriad of rules. They are compiled in the Shulchan Aruch, the universally accepted code of Jewish law and custom.
But every rule has an exception (whether or not this rule has an exception gives rise to a logical paradox, but God is above logic and His Torah draws His infinite light, above logic, into finite strokes of logic).
Most good, God-fearing Jews live by the rules, as indeed God desires. They are generally unaware of the exceptions (and often are inclined to repress into their subconscious the very existence of exceptions). But the great tzadikim (righteous souls) of every generation are always conscious of exceptions to the rules, and to a large extent devote their lives to the exceptions.
A congregation of souls, all together, creates a rule. But each individual soul, when observed out of the context of the congregation, is an exception to the rule.
Political systems, based on political philosophy, tend to weave a network of policy fit for the rule, the congregation (the klal), but often lose sight of the individual (the prat), the detail that does not fit the rule. The true leader is the one who studies/judges individual cases (as portrayed in the Bible in the figures of King David and King Solomon) and bases his leadership around them. He is aware of a deep secret: The exception (the seeming insignificant detail that for some 'mysterious' reason does not fit the rule) enriches the rule and motivates progress.
And so it is with regard to the organic development of the Oral Tradition. In every generation new, individual situations arise, calling for new, inspired judgment. This finds expression in the wealth of responsa literature, the true sign of progress in the Torah itself.