Bance Island House  is the headquarters building where the chief agent and his senior men lived on the upper floor. A two-story dwelling, it was built in the style of a West Indian plantation house, with an elevated veranda that wrapped around on three sides. The chief agent could walk onto the veranda to observe activities inside the the fort, or watch for slave ships sailing upriver toward the castle. From the rear windows, he would also observe hundreds of captives in the slave yard behind the house.