Spoiler for solution
They assign values to each color: yellow = 0, green = 1, red = 2, blue = 3. The first prisoner adds the values of all the other prisoners' hats (modulo 4), then finds the corresponding color to that number and stands in that row. By doing this, everyone else will know the color of their own hat. Now the second prisoner does the same thing as the first one did. This tells the first prisoner the color of his/her hat. Then the first two prisoners change their minds if necessary, moving to the right row. And everyone else can move to the right row as well.