It would have needed to already be falling apart. When someone damages a building through looting or it starts to fall apart due to population, it is basically put back onto the "building" list but with zero workers. For some buildings like inns that automatically build when population is sufficient, the game may put workers onto the task, otherwise it is up to you to assign workers or the building will continue to take damage until it is gone. For some reason we get constant messages about watch towers that are suffering from this, but not buildings within our settlements.
