haul down one's colors
1. To pull down a flag, showing you are beaten and want to stop fighting.
After a long battle, the pirate captain hauled down his colors.
2. To admit you are beaten; say you want to quit.
After losing two sets of tennis, Tom hauled down his color.