For a given NxN square there are N diagonals, starting with N in the top right corner. Each number in the sequence is below and one to the left of the previous number. So it is N - 1 more than the previous number. For a given N this forms an arithmetic progression with d = N -1.
So S = (n/2) (2a + (n-1)d)
where S = 5335, n = N, a = N, d = N - 1
Substitute in and arrange to get:
N3 + N = 10670
Solve for N and find N2 for the bottom right entry.
This equation is a depressed cubic (no N2 term) and can be solved by the Cardano method ( http://www.sosmath.com/algebra/f