Colony House slated to leave city