Broadly speaking there are two poles to training: kata and randori.And it is not just true of the martial arts. Other disciplines also show this division.
If it's pre-arranged, it's kata. Otherwise there must some uncertainty (i.e. "ran" or chaos) so it's randori.
Take music: Classical music is primarily pre-arranged; jazz is more improvised. But it is not entirely clear-cut: A classical concerto usually has a section -- the cadenza -- where the soloist gets to show-off with an improvised solo. Conversely, there are jazz standards that jazz musicians improvise around time and again.
Take theater: Plays are usually written down and brought to life in productions, but in impro(v) there is no script (really!) and invention occurs on the spot. Again, the division is not absolutely clear: The expression "the show must go on on" comes from theater, and seasoned performers must improvise in written-out plays when something goes wrong. In an interview with the actor Michael Caine he explained how he learned to "use the obstacle". For example if a chair unexpectedly obstructed his entry into a scene in a play Caine would either get angry with it (in a drama) or fall over it (in a comedy). On the impro side there are many games (think Theatresports) and set-ups to give some structure for the creativity to hang off.
Take science: From school science many (most?) get the impression that science is about learning what has gone before -- Newton's laws, rules of chemical composition, what bits make up a frog -- or following set recipes to recreate an experiment. But original science is about discovery, formulating and testing hypotheses, making sense of a bunch of observations, bringing order to chaos.
Which brings me to my first point:
Once upon a time the structured stuff was the product of a creative act (or acts), more akin to improvisation than recapitulation.
The masters who formulated and codified katas, Mozart, Shakespeare, Newton etc. although "standing on the shoulders of giants" were playing with chaos, and bringing forth structure.
What about the rest of us? Perhaps partly because of the lofty standards set by the luminaries who came before we shy away from even attempting to be original, or deliberately put down the old stuff as being worthless or out-of-date, but surely both attitudes are too extreme:
Most of us will do well -- in whatever field -- to study what came before, and also attempt to not only recreate, but create anew.Pablo Picasso was first a wonderful draftsman, in a realistic style, and then he started to play with the rules.
Personally, in the areas where I have manifested more talent -- that is, where things came easily -- I have wanted less structure. Let me figure it out myself! Let me play! On the other hand, where I have shown less ability, then I have craved structure to help me learn.
The traditional martial arts are complex and difficult. Structure is needed, but so is chaos. Traditionally randori, sparring, push hands, etc. have supplied the chaotic element in the form of competition. But alternatives like simple games; figuring out combinations,counters, and self-defence applications; researching bunkai for kata movements; inventing new kata; etc. offer additional scope.
This year I will be giving more time to tuning the balance between structure and chaos in my own training and also in my teaching.