I think if you asked most people, they tend to think the Bates House from
Psycho was THE inspiration for Phantom Manor.  I wouldn't be surprised
if it wasn't one of the things Disney looked at when coming up with the
final design for Phantom Manor.  They both sit up menacingly on a hill, and
there are the three arched windows on the tower's second floor with both
buildings. The thing to keep in mind is that the specific house that Phantom
Manor is supposed to be never existed. However the idea it was based on the
Bates House can be understood.  With Phantom Manor, it looks like Disney
took the best of several designs, and built one of the greatest looking haunted
houses of all time