Three agents already have left the service, including a man who joked on Facebook that he checked out former vice presidential candidate Sarah Palin while on assignment during the 2008 campaign.
The controversy has embarrassed the nearly 150-year-old agency that protects the president and investigates criminal activity, and raised questions of a possible security breach immediately proceeding
President Barack Obama's trip to Cartagena, Colombia, for last week's Summit of the Americas.
Another concern is that the alleged heavy drinking and consorting with prostitutes could be part of the culture of agents prided for their dedication and discipline.
Two Secret Service supervisors no longer with the agency in the wake of the incident were identified as David Chaney and Greg Stokes, a source familiar with the investigation told CNN national security contributor Fran Townsend on Thursday.
In a photo posted on his public Facebook page in January 2009, Chaney is seen standing behind Palin, wearing dark glasses and what appears be a wedding ring.
Under the photo, Chaney posted a comment that said, "I was really checking her out, if you know what I mean?" That drew a strong response from Palin on Thursday night. (Continues)