The growing adoption of publicly available Artificial Intelligence ("AI") tools in day-to-day business tasks has introduced significant benefits yet present data protection risks. When organisations deploy AI-generated outputs into their systems (including automated scripts, workflows, and communications) without adequate governance processes over their ICT assets, human oversight, or testing, personal data may be inadvertently exposed or mishandled.

This advisory highlights key risks observed in the use of unauthorised AI tools and recommends both process and technical measures that organisations should implement to fulfil their obligations. Under Section 24 of the Personal Data Protection Act ("PDPA"). organisations are required to make reasonable security arrangements to protect personal data in their possession or under their control.

Illustration

An employee of Company A, seeking to streamline customer communications, independently developed an automated email workflow using a publicly available AI tool accessed through their personal account, outside of the organisation's official development assets and governance framework. The employee deployed the AI-generated script directly into production use without conducting any testing, human review, or approval from the organisation's IT or data protection teams.

When executed, the script sent bulk emails using the “TO:” field which exposes the email addresses of all recipients to the entire mailing list. This unintended outcome results in the unauthorised disclosure of customers' personal data, and constitute a potential breach of the organisation's obligations under the PDPA.

To address such lapses, organisations should take relevant measures to address the following concerns:

(a) Presence of Shadow AI within the organisation.

Without clear policies and asset management controls governing the adoption of AI tools, employees may independently introduce unauthorised AI tools into the organisation's environment, a phenomenon known as "Shadow AI". These tools may process personal data without the organisation's knowledge, creating uncontrolled data flows and exposing personal data to unvetted third-party services.

Recommended Mitigation Measures:

i) Maintain visibility and control over AI tools in use by:

Maintaining an inventory of all AI tools deployed across the organisation, alongside inventories of ICT assets. This AI tool inventory should capture each system's purpose, data flow information, and intended users, reviewed and updated regularly. This allows organisations better understanding of what AI tools are used and how to effectively safeguard them.

ii) Establish clear governance policies for AI tools adoption by:

Performing a dedicated risk assessment focusing on security risks related to AI tools including assessing whether sensitive data or personal data is really required to be used in these and addressing identified risks and implementing relevant measures to secure those systems.
Having clear written policies and documented processes in relation to safeguards for personal data that are collected and processed by AI tools, including end-user prompts, inputs and generated outputs, and internal enterprise data.
Ensuring that appropriate user access to AI tools' development or deployment environments is based on the principle of least privilege, to limit who or what can view and process personal data.

(b) No human in the loop – over reliance on AI without adequate human oversight.

Organisations that place excessive trust in AI-generated outputs without human verification risk deploying flawed or misconfigured systems. As AI tools remain emerging technologies, their outputs including code, scripts, and automated workflows must be reviewed and validated by technically competent personnel before deployment.

Recommended Mitigation Measures:

i) Ensure competent employees have oversight on the use of AI tools by:

Educating users on the proper use of AI tools, including the security and data risks associated and specifying what types of data or documents that should be input into AI tools and how data flows through them.
Employees involved in developing or implementing AI tools are to be trained with the knowledge needed to effectively implement data protection policies in development, including through secure coding practices and the good practices for the AI lifecycle.
Conducting periodic ICT security awareness training to keep employees updated on topics including phishing and social engineering protection, and the risks of AI misuse.

ii) Implement technical measures for employees to validate AI tools behaviour before and after deployment:

Monitoring and logging inputs and outputs to AI tools to prevent misuse or the AI system being tricked into incorrect or unintended responses, and to facilitate audit and investigation.
Monitoring and validating input prompts, queries, or API requests for attempts to access, modify, or exfiltrate information deemed confidential by the organisation.

(c) Software development failures – inadequate testing before deployment.

AI-assisted code generation and automated scripts may contain errors or security vulnerabilities that are not immediately apparent. Deploying such code without rigorous testing at each stage of the development lifecycle can result in data breaches. Organisations must adhere to proper software development practices, including thorough testing, code review, and validation before any system handling personal data is commissioned.

Recommended Mitigation Measures:

i) Conduct rigorous testing throughout the software development lifecycle through:

Code review and rigorous unit testing, including complete testing of functional requirements to verify compliance with requirement specifications at an early stage in the system development lifecycle.
Before deploying or after any updates, testing and benchmarking AI tools to ensure they function as expected, and validating that the integrity and authenticity of the outputs can be confirmed by its intended users.
System Acceptance Testing ("SAT"), load testing, and stress testing at the near-end stage of the system development lifecycle to ensure robustness and security, with use case scenarios that simulate real-world usage as comprehensively as possible.
Security testing (such as vulnerability assessment, penetration testing, or red teaming) on the applicable AI tool or ICT system before commissioning for production, promptly after going live, periodically, and upon major changes.

ii) Documenting configurations and not using personal data in non-production environments.

Documenting the AI tool’s configuration settings and security policies and test these regularly to ensure they correspond to the requirements.
Not using or storing production data containing personal data in non-production environments for testing or other purposes; organisations should instead create test data using Synthetic Data Generation [1] when appropriate.

Organisations may further refer to PDPC’s Guide to Data Protection Practices for ICT Systems for more best practices when adopting AI tools into their ICT environment.

Additional Resources:

Advisory Guidelines on Use of Personal Data in Generative AI
Guide to Data Protection Practices for ICT Systems
Guide to Synthetic Data Generation
AI Verify Testing Framework

^[1] Synthetic Data Generation is a Privacy-Enhancing Technology that enables organisations to use data for analytics, AI model training, collaboration, and software testing — without directly exposing personal or sensitive data of an individual.