With five years of practice under my belt (or wherever it goes), I am now able to share with the public the three stages of improvisation.
Firstly, the beginners stage. You are focused on playing notes. Your conscious attention is devoted to finding a bit of improvised melody to play, and while you’re doing this the chord suddenly changes and catches you unawares – bother! So you respond by playing up and down each chord in turn, which is not great, but passable.
Next, the intermediate stage. After a few years, you make a transition into a realm in which you’re no longer having to worry too much about where melody comes from – it’s beginning to come automatically.
So you find yourself entering a stage in which you’re playing, not notes, but whole scales. Particularly as you play fast numbers, like I Got Rhythm or How High The Moon or anything bebop, you can feel the scales slipping by, and you only have to think of a scale to have all the notes of that scale under your fingers. Which you then express in melody.
This is facilitated by the jazz practice of moving from scale to scale by adding one flat at a time, so for example D7, G7, C7, F7 (found in I Got Rhythm played in the key of B flat). Each scale is just like the preceding one but with one note different, which is obviously confusing at first but comes eventually.
An alternative way of thinking of it, which works great musically, is, in the example given: D7, Dm7, C7, Cm7, because Dm7 (dorian mode, i.e. with major sixth) has the same selection of notes as G7, and Cm7 the same as F7.
This is where I am now (on a good day with a fair wind).
But off in the distance I can see the third stage through the mists, the real jazz playing stage. Here you’ve internalised not only melody but also the pattern of chord sequences, and your conscious attention is now (I speculate) focused on the piece as a whole. So it’s a process of internalising technique and moving one’s consciousness awareness and intention up onto higher levels of more comprehensive overview each time.
An analogy which occurred to me recently is that jazz improvisation is very much like ordinary conversation. Conversation is basically language improvisation, in other words, you know what the context is (a social group, a discussion on a particular topic), and you have the intention to say something hopefully relevant and interesting, but you don’t know until you open your mouth which actual words or grammatical structures you’re going to use.
Of course you assimilate the mechanics of speaking at an early age, so you can focus entirely upon meaning, like an advanced musician. But not if you’re still learning the language as an adult – then you’re constantly being held up by technical problems: finding the right word, identifying its gender, putting it into the right case, and so on.
So if musical improvisation is like conversation, playing classical or folk music is like reading out poetry aloud. Two different art forms, but both interesting – and even jazz usually begins with a statement of the theme, even if here one is free to rephrase the musical poetry.