Using DALL·E to Generate Photorealistic Stimuli for Cognitive Psychology Experiments – Part 1

By Aedan Li – July 17, 2022

As someone who spends a lot of time manually creating stimuli, I am well aware of the challenge (and time) it takes to handcraft images for cognitive psychology and neuroscience experiments. What if there were a way to exponentially speed up the creation of complex photorealistic stimuli? Critically, tools like DALL·E may be able to accelerate the stimulus design process (e.g., Goetschalckx, Andonian, & Wagemans, 2021)

DALL·E is a 12-billion parameter version of GPT-3 trained to generate images from text descriptions. Basically, you input a sentence and the algorithm will automatically generate high-resolution photorealistic images corresponding to the input text.

I received beta access and like a kid eating candy for the first time, I spent the past few days obsessively exploring whether DALL·E could be used to generate stimuli for my experiments.

As a prelude, the results were even better than I had originally expected. Every image below was generated by DALL·E. 🤯

1. Object Categories

In the most straightforward application, I could create photorealistic object images, including animals, scenes, plants, and tools. This functionality may be important for any general cognitive experiment, like studies manipulating object exemplars and categories (Murphy, 2022). In the below, each image took 10 seconds to create.

2. Similar Object Lures

Another experimental manipulation involves showing participants similar images of study objects (i.e., a similar lure; Kim & Yassa, 2013). These object images may take considerable time to develop, especially for experiments with hundreds of study objects.

DALL·E provides a “variations” option that allows the generation of similar variants of any image with a single mouse click.

You can also generate variations of your own uploaded image. To test this capability, I uploaded a fribble (Barry, Griffith, De Rossi, & Hermans, 2014). See the results for yourself.

3. Object Position

In some cases, the experimenter may wish to display objects in different spatial positions. For example, testing object-in-place memory, thought to be linked to the human medial temporal lobes (e.g., Yeung et al., 2019).

DALL·E can generate the relative locations of realistic objects using modifiers such as “on top” or “below”.

Furthermore, object position can be directly edited by the experimenter. I uploaded an image of a farmers market and then changed the location of a cat. Can you spot the change? (e.g., Simons, Franconeri, & Reimer, 2000).

4. Object Features

DALL·E is fascinatingly good at editing the features associated with object concepts. For example, I manipulated the texture of the frogs below, which would be challenging and time consuming to create manually with existing tools.

In addition to features like texture, you can manipulate abstract features like complexity, emotional content, and even traits like “old-fashioned”.

5. Object Dimensionality

I was able to easily generate line drawings as well as more complex 3D renders. This latter capability might be especially useful for future studies pairing realistic objects with virtual reality (e.g., Bohil, Alicea, & Bioccca, 2011).

6. Object Relations in Complex Scenes

Intriguingly, DALL·E could manipulate the position and quality of multiple objects in a scene, including their realism, resolution, locations in the real world, and even whether a photo of an image was poorly taken or not. The one limitation was that comprehensible language could not be displayed on signs or buildings.

7. General Image Style

The original design of DALL·E may have been to generate art, and indeed, the algorithm is fantastic at doing so. This functionality may be applicable for experiments that manipulate the general category shared across multiple images.

Conclusion – Part 1

Some researchers have suggested that artificial intelligence is the “new electricity“, which will transform every industry and create enormous economic value. In this preliminary investigation, I have likely only scratched the surface of what artificial neural networks can do for stimulus generation in cognitive psychology and neuroscience experiments.

It was really fun exploring DALL·E (thank you to OpenAI)! If anyone reading has future ideas or general thoughts about where to take this project next, please feel free to shoot me a message on Twitter or by email (aedanyue.li@utoronto.ca).