AnyGPT:

Unified Multimodal LLM with Discrete Sequence Modeling

Demonstrations

• Example-1: Speech conversations (Voice Clone)

voice prompt (to clone):

(Transcription: write a spring poem)

(Transcription:

Spring's beauty,
The grass is green,
Colors abound,
A blooming spring we'll see,
Nature's miracle,
A new season of grace,
Spring's here, it's Spring!)

angry voice prompt1 (to clone):

(Transcription: Generate an angry dramatic character line)

(Transcription: I'm not going to take this anymore! I know it's a long-running joke but please, let me live in peace!)

• Example-2: Speech Instruction → text +Image + Music + Speech Response

(Transcription: I've been too tired recently. Where do you think suitable to relax?)

(Transcription: How about a serene lakeside?)

(Transcription: Can you give me some relaxing music?)

(Transcription: Enjoy this peaceful setting.)

(Transcription: Can you draw me a picture of sunny beach?)

(Transcription: Give me a similar style of music?)

• Example-3: Speech Instruction + Music → Text + Image + Speech Response

(Transcription: Can you convert the feeling of this music into an image?)

(Transcription: The music feels energetic and triumphant.)

• Example-4: Speech Instruction + Image → text + Music + Speech Response

(Transcription: What do you see in this picture?)

(Transcription: Unity and the warmth of human connection shining brightly out here. Want to feel it too?)

(Transcription: Generate a music for it.)

• Example-5: Text → Image + Music

Can you show me a picture of a sunny beach?

Can you give me a piece of similar style of music?.

Draw a scene of a mysterious forest where night has fallen and moonlight filters through the treetops. In the center of the image, a winding stream quietly flows, reflecting the moonlight.

Compose a piece of music for this scene.

• Example-6: Text + Image → Music

Generate a piece of background music for this image.

(case1)

(case2)

(case3)

Generate a piece of music based on the picture's emotion.

• Example-7: Text + Image → Text + Music

Can you translate the emotion in this picture into music?

The image exudes tranquility and meditation.

Understand this picture, then generate a piece of matching music. Please think step by step.

The painting is abstract with bold use of color. Here's the music

• Example-8: Text + Music → Text + Image

What instrument is in this piece of music?

The bass drum is prominently featured, often indicating a strong beat in music.

Draw a picture about it.

Can you find an image to match this song's mood?

The music feels tender and nostalgic.

• Example-9: Text + Music → Muisc

can you generate similar music I might enjoy?