I suspect that what you read was referring to the evolution from the trochos or wheel system to the diapason or octave system. This is unrelated to modes.
In the wheel system, the scale is diatonic and repeats every five notes. E.g. D E F G A B c d e f# g and so on. Note the F#, which would not occur in the octave system. This system is also called tetraphonia because it repeats every four intervals. That is, the interval between D and E is the same as the interval between A and B, and the same as between e and f#. It's also called pentachordal because it repeats every five notes.
In the octave system, the scale repeats every eight notes like in Western music.
The eight mode system was well-established by the time the Papadike manuscripts were written and I've never heard of an earlier four mode system.
Technically, if you mean by scale, we only originally had 4 modes.
The authentic (kyrioi echoi) are distinct from the plagals (plagioi echoi) by their melodic progression:
- the authentic modes go down and up
- the plagal modes go up and down
The tetraphonic system is the standard way of seeing the modes, because most melodies only go up and down on only one fifth.
When the melody would go below or above the boundaries of the fifth, you can modulate wherever you want, because you are no longer bounded by anything.
Let's say you go from the 1st Plagal up to the 1st authentic:
D-F-Eqb-F-G-A-A (some stereotypical melodic progression).
Above the A, you can modulate into anything:
- Plagios tou protou: A - Bqb - C - D - E - A
- Plagios tou deuteroy: A - Bqb - Cq# - D - E - A
- Barys: A - B - C - Dq# - E - A
- Plagios tou tetartou: A - B - Cq# - D - E - A
etc.
This is why you see so many modulations in Byzantine Music: outside the boundaries of the fifth, the author is free to do whatever he wants.