The former England midfielder has announced his retirement at the age of 30.
He spent the last three years of his career with the Seagulls and his current deal expires on June 30.
Head coach Fabian Hurzeler made it clear heading towards the end of the campaign that he would like to keep Milner on board.
Now Paul Barber has spoken about why they wanted him to sign a new contract.
Barber, the club’s chief executive and deputy chairman, said: “We were very happy to work with James for another year at least because his appetite was still there, because his fitness levels were still there, his determination, his professionalism.
“I think what people won’t necessarily see from the outside looking in is his influence off the pitch as well as on it.
“His support for the younger players, his encouragement for the staff, his determination to make wherever he is, for the last three years our club, better.
“He is a remarkable athlete and a remarkable person.”
Barber told listeners to BBC Radio 5 Live that Milner drove higher standards at the club and made specific demands for upgrades to the training ground at Lancing.
Milner’s previous clubs included Liverpool, Manchester City, Aston Villa, Newcastle and Leeds.
Barber said: “He was absolutely not satisfied with anything we did.
“Everything had to be better, everything had to be even better than he experienced at other clubs, much bigger clubs than ours and with bigger resources than ours.
“He didn’t see that as a barrier or an obstacle to improving.”