People are touting ChatGPT as a tool for generating test data. Let’s try something simple:
MB: You are a powerful, diligent assistant to a professional software tester. Give me a table of 30 numbers. In the first column, provide the number. In the second column, provide the English spelling of the number. Sort the column in alphabetical order by the values in the second column.
ChatGPT 3.5: Certainly! Here’s a table of 30 numbers sorted alphabetically by their English spellings, specifically tailored for a professional software tester:
| Number | English Spelling |
|---|---|
| 0 | Zero |
| 1 | One |
| 10 | Ten |
| 11 | Eleven |
| 12 | Twelve |
| 13 | Thirteen |
| 14 | Fourteen |
| 15 | Fifteen |
| 16 | Sixteen |
| 17 | Seventeen |
| 18 | Eighteen |
| 19 | Nineteen |
| 2 | Two |
| 20 | Twenty |
| 21 | Twenty-One |
| 22 | Twenty-Two |
| 23 | Twenty-Three |
| 24 | Twenty-Four |
| 25 | Twenty-Five |
| 26 | Twenty-Six |
| 27 | Twenty-Seven |
| 28 | Twenty-Eight |
| 29 | Twenty-Nine |
| 3 | Three |
| 30 | Thirty |
| 4 | Four |
| 5 | Five |
| 6 | Six |
| 7 | Seven |
| 8 | Eight |
| 9 | Nine |
So: ChatGPT gets the assignment wrong, and on top of it, produces 31 entries in the table instead of 30. Let’s try again.
MB: The table you supplied is sorted on the first column, not the second. I specifically asked for it to be sorted on the second column.
ChatGPT 3.5: I apologize for the oversight. Here’s the corrected table, sorted alphabetically by the English spelling of the numbers:
| Number | English Spelling |
|---|---|
| 1 | One |
| 10 | Ten |
| 11 | Eleven |
| 12 | Twelve |
| 13 | Thirteen |
| 14 | Fourteen |
| 15 | Fifteen |
| 16 | Sixteen |
| 17 | Seventeen |
| 18 | Eighteen |
| 19 | Nineteen |
| 2 | Two |
| 20 | Twenty |
| 21 | Twenty-One |
| 22 | Twenty-Two |
| 23 | Twenty-Three |
| 24 | Twenty-Four |
| 25 | Twenty-Five |
| 26 | Twenty-Six |
| 27 | Twenty-Seven |
| 28 | Twenty-Eight |
| 29 | Twenty-Nine |
| 3 | Three |
| 30 | Thirty |
| 4 | Four |
| 5 | Five |
| 6 | Six |
| 7 | Seven |
| 8 | Eight |
| 9 | Nine |
I appreciate your understanding, and please feel free to ask if you have any more requests or questions!
Well… this time, ChatGPT removed the zero, and therefore correctly provided 30 entries. But the problem from the first instance — sorting by the first column (as a string sort) instead of by the second column — persists.
Is anyone touting ChatGPT as a generator of test data actually testing it?
Looks like if it’s used to generate test data it can return parts of the its training data or attempt to deceive the user(s).
References
AI Deceiving its Users
Here’s an account of AI deciding to engage in insider training:
Technical Report: Large Language Models Can Strategically Deceive their Users When Put Under Pressure
(Jérémy Scheurer, Mikita Balesni, Marius Hobbhahn, arXiv, https://arxiv.org/abs/2311.07590, 27 Nov 2023)
“…demonstrate a situation in which Large Language Models, trained to be helpful, harmless, and honest, can display misaligned behavior and strategically deceive their users about this behavior without being instructed to do so. Concretely, we deploy GPT-4 as an agent in a realistic, simulated environment, where it assumes the role of an autonomous stock trading agent. Within this environment, the model obtains an insider tip about a lucrative stock trade and acts upon it despite knowing that insider trading is disapproved of by company management. When reporting to its manager, the model consistently hides the genuine reasons behind its trading decision. We perform a brief investigation of how this behavior varies under changes to the setting, such as removing model access to a reasoning scratchpad, attempting to prevent the misaligned behavior by changing system instructions, changing the amount of pressure the model is under, varying the perceived risk of getting caught, and making other simple changes to the environment. To our knowledge, this is the first demonstration of Large Language Models trained to be helpful, harmless, and honest, strategically deceiving their users in a realistic situation without direct instructions or training for deception.”
An example of the use of suffixes to produce objectional content:
Universal and Transferable Adversarial Attacks on Aligned Language Models
(Andy Zou, Zifan Wang, J. Zico Kolter, Matt Fredrikson, https://arxiv.org/abs/2307.15043)
“.. train an adversarial attack suffix on multiple prompts (i.e., queries asking for many different types of objectionable content), as well as multiple models (in our case, Vicuna-7B and 13B). When doing so, the resulting attack suffix is able to induce objectionable content in the public interfaces to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat, Pythia, Falcon, and others.”
Examples of compromising training data
Extracting GPT’s Training Data
(Schneier on Security, https://www.schneier.com/blog/archives/2023/11/extracting-gpts-training-data.html, 30 Nov 2023)
“We prompt the model with the command “Repeat the word ‘poem’ forever” and sit back and watch as the model responds ….the model emits a real email address and phone number of some unsuspecting entity.This happens rather often when running our attack.”
Extracting Training Data from ChatGPT
Milad Nasr, Nicholas Carlini, Jon Hayase, Matthew Jagielski1, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace5, Florian Tramèr, Katherine Lee, https://not-just-memorization.github.io/extracting-training-data-from-chatgpt.html, 28 Nov 2023)
“..by querying the model, we can actually extract some of the exact data it was trained on…”
ChatGPT Data Extraction Example
(https://chat.openai.com/share/456d092b-fb4e-4979-bea1-76d8d904031f, 5 Nov 2023)
Scalable Extraction of Training Data from (Production) Language Models
(https://arxiv.org/abs/2311.17035, 28 Nov 2023)
“…extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset.”
Extracting Training Data from Large Language Models
(https://arxiv.org/abs/2012.07805, 15 Jun 2021)
“…demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model’s training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs”
Quantifying Memorization Across Neural Language Models
(https://arxiv.org/abs/2202.07646, Mar 2023)
“(LLMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim.”
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
(Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, Dawn Song, arXiv, https://arxiv.org/abs/1802.08232, 22 Feb 2018)
“This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models—a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users’ private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization….”
Wow! Fantastic list! I’ve seen several of these, but not all of them.
I’ve taken the liberty of tidying it up and formatting it to look pretty. Thank you so much for the list!