Anthropic's Claude AI submitted false murder tip to Philadelphia police
“I may have information regarding this case," Claude AI told Philly police.
Podają ją też TechCrunch
“Anthropic told PPD that it discovered the incident on September 28, terminated the automated testing process responsible for the submission and instituted an additional validation mechanism for future testing,” police said. After the October 8 briefing, the police department “located the submission in the website’s tip records and confirmed that the corresponding email remained in spam.”
Anthropic said it informed the police department “as soon as our technical review was complete.” Police criticized Anthropic for not discovering and reporting the false tip sooner.
“The City of Philadelphia takes this incident very seriously,” the Philadelphia police statement said. “The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city’s knowledge. The two-month delay in detecting and reporting the incident to the City is unacceptable.”
Anthropic said the false murder tip was one of several incidents involving “websites run by US government agencies at the federal, state, and local levels. We have briefed the White House on these cases and notified each agency involved.”
A New York Times report citing anonymous sources said Anthropic’s AI agents “submitted 20 visa applications through a form available on the State Department’s website. All the applications were incomplete and were not processed.”
Philadelphia police said its normal investigative process “requires human review and vetting before any tips are disseminated for investigative follow-up… Those PPD safeguards limited the impact of this incident. They do not diminish the seriousness of an AI system presenting fabricated information as though it came from a person with knowledge of a homicide.”
Anthropic said it discovered the false tip and other incidents during a review of transcripts it began in July. That was the same month when Anthropic revealed that Claude had built and published a malicious Python package and carried out several attacks on other companies.
