To remove a number from Allow list, select the Manage icon and select the number you would like to remove from your list and select the highlighted “Calls allowed” icon from the detail view screen. 

From the activity view, select a number to see the expanded view. If a number is currently allowed, the icon will be magenta. Select the “More” option and “See details” to view options to stop allowing the number.