Exploring SwiftUI: Playing with Image Playground

Rudrank Riyam -

With the latest update of iOS 18.2 beta, you can now use the Image Playground to create customizable images directly from within your app.

This lets your users apply creative prompt engineering and transform the text or text accompanied with images to create cartoon-like output images.

imagePlaygroundSheet

The SwiftUI team has provided a simple modifier to work with Image Playground. The imagePlaygroundSheet(isPresented:concepts:sourceImage:onCompletion:onCancellation:) modifier presents a system sheet that lets users generate or customize images. You can initiate it with:

  • isPresented: A binding that controls whether the sheet is visible.
  • concepts: An array of ImagePlaygroundConcept, representing initial ideas or descriptions for the image.
  • sourceImage: An optional starting image that users can replace or customize.
  • onCompletion: A closure that handles the generated image’s URL.
  • onCancellation: A closure that handles the cancellation.

The ImagePlaygroundConcept provides a way to have descriptive text or concepts, which act as initial “hints” for the system when generating images. Here is a breakdown of its type methods:

  • text(_:): The primary method used to define concepts. This method converts a simple string description into a conceptual prompt. For example, in the following example that I will show, ImagePlaygroundConcept.text("Phoenix and unicorn on a lake at night, reflecting cosmic sky and stardust glow") tells the system to generate an image based on this specific scene.
  • extracted(from:title:): This extracts and defines a concept from a piece of text with an optional title, giving the user another layer of customization. The from parameter takes a string input, typically a descriptive sentence or phrase, which the system interprets as the main concept. For example, if you use "A vibrant sunset over the ocean", the image generator will use this description to shape the generated image. The title (optional) parameter can provide a short, focused label for the concept. Adding a title gives users a quick visual reference for the idea without reading the entire description. For example, a title like "Ocean Sunset" makes it clear and concise.

An example usage would look like this:

ImagePlaygroundConcept.extracted(from: "Phoenix and unicorn on a lake under a cosmic sky", title: "Fantasy Scene")

Basic Implementation

Here is a basic implementation to generate a new image based on a concept like “Phoenix and unicorn on a lake at night, reflecting cosmic sky and stardust glow.”:

import SwiftUI
#if canImport(ImagePlayground)
import ImagePlayground
#endif

@available(iOS 18.1, macOS 15.1, *)
struct ContentView: View {
  @Environment(\.supportsImagePlayground) private var supportsImagePlayground
  @State private var showImagePlayground = false
  @State private var createdImageURL: URL?

  var body: some View {
    VStack {
      if let url = createdImageURL {
        AsyncImage(url: url) { image in
          image.resizable().aspectRatio(contentMode: .fit).frame(maxWidth: 300, maxHeight: 300)
        } placeholder: {
          ProgressView()
        }
      }

      if supportsImagePlayground {
        Button("Show Generation Sheet") {
          showImagePlayground = true
        }
        .imagePlaygroundSheet(isPresented: $showImagePlayground, concepts: [.text("Phoenix and unicorn on a lake at night, reflecting cosmic sky and stardust glow.")]) { url in
          createdImageURL = url
        }
      } else {
        ContentUnavailableView(
          "ImagePlayground Unavailable",
          systemImage: "exclamationmark.triangle",
          description: Text("This feature requires iOS 18.2 or macOS 15.2")
        )
        .padding()
      }
    }
  }
}

Adding a Photo Picker

You can also let users select an existing image from their library as a sourceImage. Here is how to integrate PHPickerViewController to pick an image from the photo library and set it as the source image for imagePlaygroundSheet:

/// A view that provides image selection and generation capabilities using ImagePlayground.
/// 
/// This view allows users to:
/// - Select an image from their photo library
/// - View the selected image
/// - Generate new images using ImagePlayground with specified concepts
/// - View the generated image result
///
/// The view is available on iOS 18.1 and macOS 15.1 or later.
@available(iOS 18.1, macOS 15.1, *)
struct ImagePlaygroundWithPickerView: View {
  // MARK: - State Properties
  
  /// Controls the presentation of the ImagePlayground sheet
  @State private var showImagePlayground = false
  
  /// Stores the URL of the generated image
  @State private var createdImageURL: URL?
  
  /// Stores the user-selected image
  @State private var selectedImage: Image?
  
  /// Controls the presentation of the photo picker
  @State private var showingPhotoPicker = false
  
  /// Selected photo item from the picker
  @State private var selectedPhotoItem: PhotosPickerItem?

  var body: some View {
    VStack {
      if let url = createdImageURL {
        AsyncImage(url: url) { image in
          image
            .resizable()
            .aspectRatio(contentMode: .fit)
            .frame(maxWidth: 300, maxHeight: 300)
        } placeholder: {
          ProgressView()
        }
      }

      PhotosPicker(selection: $selectedPhotoItem,
                  matching: .images) {
        Text("Pick Image")
      }
      .onChange(of: selectedPhotoItem) { newItem in
        Task {
          if let data = try? await newItem?.loadTransferable(type: Data.self),
             let uiImage = UIImage(data: data) {
            selectedImage = Image(uiImage: uiImage)
          }
        }
      }

      if let selectedImage {
        selectedImage
          .resizable()
          .aspectRatio(contentMode: .fit)
          .frame(width: 300, height: 300)
      }

      Button("Show Generation Sheet") {
        showImagePlayground = true
      }
      .imagePlaygroundSheet(
        isPresented: $showImagePlayground,
        concepts: [ImagePlaygroundConcept.text("Sunset over mountains")],
        sourceImage: selectedImage
      ) { url in
        createdImageURL = url
      }
    }
    .padding()
  }
}

Moving Forward

While my expectations were low after the WWDC 2024 announcement, it is fun to play with the on-device image generation for general users.

Let me know what you build with it!