The contacts you remove remain in your Gmail contact list but aren't part of the label until you re-add them. Creating email ...

With Gmail's safe listing, you can create a list of trusted contacts and domains to ensure you never miss important emails.