Well, something must be wrong, since you got two different answers IMHO, as I have already stated, your initial assumption that A and B are the closest (or farthest) points seems to be wrong. I'll try to justify my point in a subsequent post, please give me some time.
And on the solution above, you are solving a different problem How can you be sure that the solution to both the problems are the same?
I'm going to attempt what bononova suggested, perhaps you can try that too. Solve first for three points in a circle of radius R, instead of an infinite plane.