Category: AI / Artificial Intelligence

In this post, I’ll show how I made this AI video containing a consistent main character.

Note: as I made this video, I kept finding and learning better ways to make it, so this video is not as good as could be had I started over using the techniques in this post.

Decide on a theme/setting

I wanted to make a music video using the song “When You Believe” from the movie “The Prince of Egypt”. For that reason, I chose ancient Egypt for my setting.

Create your main character

I wanted to see if I could create an AI version of me with changes to suit my desired main character, i.e., I wanted the character’s face to look as close as possible to me, but have other features like physique be more athletic and sculpted. To create the main character, I chose a picture of me that I liked and I asked ChatGPT to give me a prompt to create an image of a person that matches an example image I found.

ChatGPT analyzed the outfit in the image I uploaded and gave me a prompt to generate an image with that outfit applied to a reference image I would upload. For the reference image, I chose this picture of me.

In OpenArt, I did the following

1. Image > Create Image
2. Model = Nano Banana
3. Uploaded my reference image to the Omni Reference input
4. Clicked “Create”

Nano Banana gave me this:

It was good, but I ended up making various changes until I got this:

This final character image is based on another photo of mine. Tweaking the character photo was a lot more difficult than it should have been. I had to tell Nano Banana to make my arms a little darker and have me facing straight at the camera. When you create your main character, whether from a photo of a real person or not, try to have the character standing straight and facing the camera with a neutral facial expression. In my image above, I have a little smile, which kept getting carried over into various AI image generations. To fix this, I had to update every prompt that used this image to include the comment, “Subject is not smiling”. Also, AI generations would often remove my sunglasses, even though my main character reference image includes sunglasses. To fix this, I had to update every prompt that used this image to include the comment, “Subject is wearing glasses or his sunglasses.”

Note: Make sure you are satisfied with your main character image because you will use it to generate many images and videos with it as a reference.

Decide on your video content structure

In my case, I didn’t want to show viewers the main character non-stop throughout the video, so I decided to have alternating clips showing clips with and without the main character, i.e.

Generate image and video prompts

Generating images is much cheaper than generating videos. For example, in OpenArt, here’s what I paid in credits for each type:

For this reason, I created an image of the first frame of each scene (video clip) before creating each video clip. If I didn’t like an image that was generated, I could regenerate it, which is much cheaper than regenerating a video.

For the scenes without the main character, I didn’t need a reference image, so I used the Flux Pro model, which was cheap (5 credits) and produced great results.

For the scenes containing the main character, I needed to upload my reference image. Flux Pro doesn’t support reference images, so I used the Nano Banana model, which does support reference images.

UPDATE: actually, SeeDream 4.0 is just as good as Nano Banana. Use both SeeDream and Nano Banana, then choose which result you prefer.

For video generation, I chose Kling 2.5 because it supports a first-frame image option and produced good results.

Since we want the final video not to show a bunch of random content, I had each pair of consecutive scenes with and without the main character be relevant to each other. To do this, and to generate detailed prompts that matched my theme and setting, I used ChatGPT. For example, I asked ChatGPT the following:

Give me 10 pairs of cinematic video prompts along with their first-frame image prompt. For each pair, the first set of image and video prompts will not include my main character reference image, and the second set of image prompts will include my main character reference image.

I will use Flux Pro to generate images that don’t include the main character, I will use Nano Banana to generate images with the main character reference image, and I will use Kling 2.5 to generate each video using the corresponding image as the first frame. Every detail that will be referenced in the video prompt should be included in the image prompt.

Since I am making a music video of the song “When You Believe – The Prince of Egypt”, the theme/setting of the scenes should be ancient Egypt. The scenes should include popular landmarks, like the pyramids and the Nile River, people in a busy market, ancient mosques, and, of course, a royal palace for the prince. Each pair of prompts with and without the main character should be relevant to each other so as to flow as a story. If possible, all scenes should flow together as a cohesive story.

After answering a few follow-up questions from ChatGPT, it gave me the prompts, like the following:

1. The Desert Procession (without the prince)

Image Prompt (Nano Banana)
Epic realism, ancient Egyptian soldiers and servants walking through golden desert dunes at dusk, carrying ornate banners and treasures toward a distant palace on the horizon. The sky glows crimson-orange, sand whipping lightly in wind, dust haze around the caravan, footprints trailing through rippled dunes. Cinematic depth, ultra-detailed, 50mm lens realism, warm atmospheric glow.

Video Prompt (Kling)
Dynamic tracking shot alongside the caravan; camera glides parallel to marching servants, banners flapping, sand swirling, sunlight catching on gold ornaments; as the procession nears the palace, a subtle lens flare crosses frame, suggesting the arrival of royalty; ambient desert wind and drum rhythm underscore the movement.

2. The Prince’s Arrival (with the prince)

Image Prompt (Nano Banana)
Epic realism, regal Egyptian prince wearing a black tunic with gold trim, ornate jewelry, and a linen cloak, descending from his chariot before the grand palace gates. Behind him, the same caravan from previous scene arrives in the background. The palace gleams under the last light of sunset, palm trees swaying gently, atmosphere majestic and warm. The prince’s face is clearly visible, confident and composed, cinematic lighting on his features, 50mm shallow depth.

Video Prompt (Kling)
Camera begins low behind the prince’s boots stepping onto polished stone, then cranes upward to reveal his full figure framed against the glowing palace. Cloak moves naturally in breeze, chariot wheels still turning in background, banners from previous scene flutter in golden light. Subtle push-in as the prince lifts his gaze toward the palace — the soundtrack swells, symbolizing destiny and return.

Generate Images and Videos

Now that we have our inputs (prompts and main character image), we can generate images and videos for each scene.

For the images without the main character, I used Flux Pro like this

In this example, I got the following image:

I then used that image as the first frame to generate a video using Kling 2.5 as follows:

This produced the following video:

For the images with the main character, I used Nano Banana like this:

Notice that I uploaded my main character reference image in the Omni Reference field. I also uploaded the image from the previous prompt showing the caravan since this prompt references it. This prompt generated the following image:

Again, I then used that image as the first frame to generate a video using Kling 2.5 as follows:

which generated the following video:

Repeat for all prompts

I then just repeated the previous image and video generation step for all the prompts that ChatGPT gave me.

Create final video

Lastly, I added all assets (video clips, music) to a video editing to (I used Capcut), put them on the timeline in my desired sequence, and exported the final video.

Create a poster image

Since I published my video on YouTube, I wanted an attractive thumbnail / poster image. I scrubbed through the video in the timeline in Capcut until I found a frame that I like, and then exported that frame as a still image. I then added the title of the song (“When You Believe”) to it in Photoshop. This is the poster image I ended up with.

In this post, I’ll explain how to make a music video using AI like the one below.

Get a song

The first step is to get a song. Here are some options:

1. Buy a song, e.g., on Amazon
2. Download a song from YouTube, e.g., using a YouTube music downloader
3. Create a song using AI, e.g., using Suno or TopMediAI

I chose the song “California Love” and I had it in mp3 format.

Optionally, separate vocals from backing track

Sometimes, when you upload a song to generate a lipsync video, the tool can’t distinguish between the vocals and the backing track, resulting in imperfect lip movements. For example, using this song segment

we see that the lipsync results near the end are inaccurate.

If you created the song using Suno, TopMediAI, or any other AI music generation tool, you may be able to cleanly separate the vocal track. Look for “get stems”. Otherwise, you can try to separate the vocals from the backing track using UVR Online.

Here’s how the same lipsync video looks using this vocal track separated using UVR Online.

In this case, the lip movements at the end look accurate, but the beginning looks imperfect, which is why it’s better to get the vocal track from the source.

Get a photo of your singer

In the video above, I wanted to be the singer, so I found a clear, front-facing photo of myself. I then removed the background. Make sure to have a clear photo for better results.

Generate AI prompts for different scenes of the music video

When it comes to generating AI prompts, I find it easier to use AI (I use ChatGPT) to generate the prompts. Later, we will use AI to generate a lip sync of the singer singing the song. Unfortunately, at the time of this writing, AI is not good enough to lip sync images of subjects where their faces are too small, so for the AI prompts, we want to tell ChatGPT to show a close-up of the subject such that their face occupies 1/3 or 1/2 the image height.

Here’s an example prompt to generate a list of prompts.

Give me 20 prompts to generate 20 images using AI with the following criteria:

• the generated images should be relevant to the theme or lyrics of the song “California Love” by 2pac, ft. Dr. Dre.
• the singer in the generated images will come from an uploaded reference image
• the singer should be shown close up from the waist up so that an AI lip sync tool will produce better results
• the singer should be either facing the camera or at most facing 45 degrees from the camera

ChatGPT may ask for more information or inputs, like the image of the singer or the lyrics to the song.

Here is example output from ChatGPT.

It’s important to ensure the face of the subject is clear so the lip-sync video generation results are good. Here’s a comparison of video results from images with a small and large subject face.

Notice how the videos where the subject is not close up show facial distortion. The videos with the subject close up show much less distortion. The closer the subject (extreme close up), the less the distortion, e.g.,

Generate still images using the prompts and the singer image

I like to use OpenArt and Google Nano Banana to generate the images. In OpenArt, go to

1. Image > Create Image
2. Model: Choose the “Nano Banana” model (I prefer this model for this purpose)
3. Prompt: enter one of the prompts from the previous step
4. Omni Reference: upload the image of the singer from the previous step (you can optionally upload more than one image, e.g., a front-facing image, a side-facing/profile image, etc)
5. Output Size: I prefer 16:9
6. Upscale output: x2 (if you choose x4, the image will be too large for TopMediAI – the lipsync tool – so I choose x2)
7. Number of images: 1 (you can choose more than one, but you’ll pay more. I prefer to start with one, and if the results are okay, I’ll regenerate more images to get a variety)

Here’s an example of the interface.

After you do this for each prompt, you should end up with an array of images.

Note that you will likely need to tweak some of the prompt text and regenerate images until you get results you like.

Organize files

Since you’ll be working with many files, you’ll want to organize them so you don’t get lost and make mistakes. I like to put all files in a folder like this

“California Love” (folder)

• california-love.mp3
• img-01.jpg
• img-02.jpg
• img-03.jpg
• …

Split the song in a video editor

I like to use Capcut for video editing. Import the audio file and all image files to Capcut.

1. Add the audio file to the audio track
2. Play the video (audio only at this point) and place a marker where you want different scenes to appear
3. Listen to the audio and move the marker so that it is located between words or sentences.
4. Split the audio track at the marker
5. Repeat for as many scenes (clips) as you want

Add an image to each audio clip

1. On a video track, add an image for each audio clip such.
2. Set the duration of the image clip to match the duration of the audio clip by dragging either end of the image clip to the left or right.
3. Play the video in the preview window to see if you like the image sequence for the associated music.

Export the audio clips

If you like the preview in the previous step, export each audio segment in mp3 format. I use the left and right arrow keys to jump the playbar to the start or end of each segment and then I hit the “i” key to set the start point and the “o” key to set the end point so that I only export the segment that I want. The other segments will be grayed out.

Click the “Export” button and check only the “Audio” checkbox. Name each export with a number corresponding to its order in the timeline.

When done, your folder structure should look like this:

“California Love” (folder)

• audio-001.mp3
• audio-002.mp3
• audio-003.mp3
• …
• california-love.mp3
• img-01.jpg
• img-02.jpg
• img-03.jpg
• …

Create a lip-sync video from each image clip

There are many AI lip-sync tools, like HeyGen. HeyGen is used 3D modeling, which is good, but the output is more for talking rather than singing. As such, only the subject’s head moves. For the video above, I used TopMediAI. Though it only uses 2D modeling, it does an impressive job at animating an entire image, including the background, the subject’s head, and the subject’s body.

1. Log in to TopMediAI
2. Music AI > Music Tools > Singing Photo Maker
3. Drag/upload audio segment 1 (audio-01.mp3)
4. Drag/upload image 1 (img-01.jpg)
5. Click the “Create” button

When generation is done, preview the lip sync video. If it’s good, save the video as “video-01.mp4” and repeat for all other images. When done, your file structure should look like this:

“California Love” (folder)

• audio-001.mp3
• audio-002.mp3
• audio-003.mp3
• …
• california-love.mp3
• img-01.jpg
• img-02.jpg
• img-03.jpg
• …
• video-01.mp4
• video-02.mp4
• video-03.mp4
• …

Add the lip-sync videos to the video timeline

1. Import all lip-sync videos to Capcut
2. Drag each lip-sync video to a new video track above the track containing images, making sure the video matches the image.
3. Optionally add transitions, e.g., fade in/out audio or still images, crossfade between video clips, etc
4. Disable the audio in the lip-sync video track so that only the audio from the music track is heard.
5. Preview the entire video

If everything looks good, export the video. Click the “Export “button, check the “Video” checkbox, and set parameters (4K resolution, etc).

B-roll

Many videos include B-roll, which is filler content. In my music video above, my B-roll includes video footage of various iconic locations in California. To create this footage, I did the following:

1. Asked ChatGPT to give me a list of the top 20 most iconic landmarks in California
2. Asked ChatGPT to give me an image and video prompt for each of those 20 landmarks
3. Used OpenArt > Image > Google Nano Banana to generate an image using the image prompts from the previous step. Generating images is much cheaper than generating videos, so I prefer to generate images first. If I like an image, I use it as a reference to create a video.
4. Used OpenArt > Video > From Image to create a video from the images and video prompts generated in the previous steps.
5. Added the videos to the timeline in Capcut.

If you want to create B-roll footage using the same character, the following process produced good results.

1. Go to OpenArt > Video > Elements to Video
2. Select the Vidu Q2 model (you can use other models, like Google Veo 3, but they can be way more expensive)
3. Upload your character and optionally some other reference images
4. Enter a text description of the video scene
5. Choose other settings (aspect ratio, duration, resolution)
6. Click “Create”

For example, using this character image

and this description text:

Wide slow-tracking shot following the woman as she walks along glowing dunes. Fire lines cross behind; embers trail — cinematic motion, 8K realism.

I was able to create this video

and many other ones using the same character.

There are many tools for removing objects and people from a video, including Adobe After Effects. So far, the easiest one I’ve found is Fotor. You can also use it to remove watermarks on videos.

1. Upload your video
2. Click the “Remove Object” button
3. Paint over the objects you want to remove (you can adjust the size of the paintbrush, if needed)
4. Click ‘Generate Preview” to see how it looks. If it looks good, you can download the generated video if you are a paying member.

Here’s an example. Below is a video clip from a music video.

I wanted to remove the 2 people, which I identified in Fotor using the purple paintbrush.

Here’s the resulting video.

Usually, people create AI videos by describing what they want using a text prompt. This can be very difficult depending on the results you are looking for. Another way to create an AI video is by creating a “driving performance” video, which shows what movements you want to mimic. For example, if you want to make a video of yourself dancing and lip syncing exactly like someone else in an existing video, you can upload the existing video as a “driving performance” video and upload an image of yourself as the “character image”. This post explains how to do it using Runway Act-Two.

In Runway, click on Act-Two. You will see the UI below.

In the top section, you upload a “driving performance” video, which will contain the body movements and facial gestures you want to copy and apply to your character in the bottom section.

In the bottom section, your character can come from an image or video that you upload. For simplicity, and to match the driving video, I will upload an image containing my character.

For demonstration purposes, I want to make myself sing and dance exactly like the subject in the following video.

Runway Act-Two provides the following recommendations for driving performance videos.

• Feature a single subject in the video
• Ensure the subject’s face remains visible throughout the video
• Frame the subject, at furthest, from the waist up
• Well-lit, with defined facial features and expressions
  • Certain expressions, such as sticking out a tongue, are not supported
• No cuts that interrupt the shot
• Ensure the performance follows our Trust & Safety standards
• [Gestures] Ensure that the subject’s hands are in-frame at the start of the video
• [Gestures] Start in a similar pose to your character input for the best results
• [Gestures] Opt for natural movement rather than excessive or abrupt movement

Since my “driving performance” video has people playing music in the background, I need to remove them. One way is by using Capcut’s Auto Background Removal feature.

While it’s not perfect, it may be sufficient for Runway’s Act-Two. Here are two other AI-based video background removal tools that seem to do a better job.

Unscreen

VideoBgRemover.com

VideoBgRemover.com seems to produce the best results. However, for this test, I used the imperfect results from Capcut.

If you need more accurate subject isolation or background removal, use Adobe AfterEffects’ RotoBrush feature.

DRIVING PERFORMANCE VIDEO

CHARACTER IMAGE

Runway Act-Two provides the following recommendations for character images.

• Feature a single subject
• Frame the subject, at furthest, from the waist up
• Subject has defined facial features (such as mouth and eyes)
• Ensure the image follows our Trust & Safety standards
• [Gestures] Ensure that the subject’s hands are in-frame at the start of the video

For the character image settings, make sure “Gestures” is toggled on.

OUTPUT

To show just how close the generated video matches the source, I overlaid it on the source.

That looks pretty impressive to me.

Here are more examples.

INPUT – Performance Video

INPUT – Character Image

OUTPUT

OUTPUT – Overlaid

INPUT – Performance Video

INPUT – Character Image

OUTPUT

OUTPUT – Overlaid

INPUT – Performance Video

ERROR

INPUT – Performance Video

ERROR

INPUT – Performance Video

ERROR

INPUT – Performance Video

Since the subject in the previous video was too small, I scaled it up in Capcut. Now, Runway Act-Two was able to detect the face.

INPUT – Character Image

OUTPUT

OUTPUT – Overlaid

Since I wasn’t able to put myself behind the metal railing, I scaled up the source video to hide the railing.

INPUT – Performance Video

ERROR

INPUT – Performance Video

INPUT – Character Image

OUTPUT

OUTPUT – Overlaid

INPUT

I used VideoBgRemover.com to remove the background.

OUTPUT

OUTPUT – Overlaid

INPUT – Performance Video

INPUT – Character Image

OUTPUT

OUTPUT – Overlaid