Around 7:15pm PST tonight we saw a lot of failures on client sites across the spectrum. After digging around I found/suspected that AWS EC2 region in Northern California was having problems. Nothing showed up on the AWS Dashboard but it had to be the issue. Sure enough I checked later and found they network issues from 7:31pm PST to 7:45pmPST.
7:31 PM PST We are investigating elevated packet loss for Internet connectivity in the US-WEST-1 Region.
7:45 PM PST Between 7:14 PM and 7:23 PM PST we experienced elevated packet loss to the Internet for some instances in the US-WEST-1 Region. The issue has been resolved and the service is operating normally.
That matches what we saw on our end. Problem solved.