The Perils of Forgetting Fairness