Image-to-Video models as a Game
During the Dotcom hype most business were 'internet-adopted' businesses; businesses that could've existed without internet, but rather added internet as another tool of distribution.
Netflix's business model was 'internet-native'. It couldn't exist without the internet (at least the streaming).
What would an AI-native video game look like? Not one where we create assets on the side, but rather one where the gameplay loop fundamentally depends on AI.
AI's weakness as a feature
I love the fact that the output of AI text, image & video models can be unpredictable. How could we incorporate that into a gameplay loop?
The idea: give two players the same starting image with two characters. They would then be assigned a character, and have to textually input a single weapon (e.g. cotton candy or a pizza cutter).
Text to image would take those inputs and generate a new starting image with the weapons. An image-to-video model then generates a video with the explicit instruction for one to win, hopefully generating fun and unpredictable outcomes in the process.
Through using an Elo score we can create a competitive public leaderboard.
This is what the loop would look like:
Validation
I set out to create two demos to validate:
- A traditional battle, where users define a weapon for their character
- A drag race, where users define a weapon on their car
We'll focus on the 'traditional' battle first for creating a prototype. This was the prompt for the starting image.
Create cute 2d comic video game characters facing each other. Camera perspective should be from the side. First character has a sword coming out of the point of it's right foot. The second character one is yellow and has a unicorn horn coming out of it's forehead. They are facing each other in battle stance.
It ignored my sword-as-feet instruction, but I'll take it.
⚔️ Let's battle!
So, I tested three models: Midjourney, MiniMax Hailuo S2V - Live2D, Seedance 1.0 Lite.
I kept the prompt very simple, to give the models a lot space to play:
Battle until one wins
Let's check out the results!
Midjourney
The videos are cute, but in all instances the second character had a sword hallucinated, and nothing was done with the facial horn.
Hailuo
This model was specifically optimized for 2D. The battle wasn't very exciting to watch (albeit the floating was cool), and there was no winner, which is required for having a gameplay loop.
Seedance 1.0
This model is actually currently leading on the Video model arena leaderboards above Veo 3! Also, it's one of the cheapest to use. I will get more into the costs further below.
Only the Lite model was used for this experiment. Just wanted to see where it would go. There's another experiment coming up where I'll actually use the Pro model!
This one is also most hallucinated. Swords randomly appear, and the yellow guy goes through the floor.
💸💸💸 What about costs?
I'll get to it further down! Spoiler; it's not cost-effective at all for a free-to-play game.
🤔 Let's try this again
I would argue that
- None of the outputs respect the inputs. No video used the facial horn in battle. All hallucinated a sword.
- It's not very entertaining to watch. I was expecting more bloody battles, but the models were not willing to simulate that.
So, I figured; what about a non-violent and more static match? A drag race 🏎️!
The inputs
Have a top down hyperrealistic view of two futuristic looking cars on a racetrack. The whole track should be visible from start to finish.
Car 1 is green and has a rocket launcher on top
Car 2 is yellow and has spikes on it's wheels on the side
This resulted in:
3... 2... 1... race! 🔥
This time I chose more extensive coverage with the racing simulators. I tested the models above, but also Seedance 1.0 Pro and Kling.
The prompt:
Have these two cars battle out a race to the finish. One should win. Let them use their tools to disable the other car.
Let's see the results
Midjourney
Midjourney watched too much tentacle stuff I think.
Hailuo
Well, at least the rocket launched. Kinda.
Seedance 1.0 Lite & Pro - Image to Video
Lite on the left, Pro on the right. I don't even know how to describe what's going on here.
Seedance 1.0 Lite & Pro - Text to Video
For Seedance Pro & Lite I also did a text-to-video prompt instead of image-to-video, just to see what happens.
The entire prompt here is just a combo of the two above:
Have a top down hyperrealistic view of two futuristic looking cars on a racetrack.
Car 1 is green and has a rocket launcher on top
Car 2 is yellow and has spikes on it's wheels on the side
Have these two cars battle out a race to the finish. One should win. Let them use their tools to disable the other car.
The Lite video got Cars themed. I guess the prompt seemed to convey childlike play and friendliness.
The Pro video is the only one that's actually a race start to finish. Sadly, the video is quite uneventful, and I feel like the green car tried to cheat at the end.
Kling
Not sure what it's trying to do.
💸 Costs
Even if the videos were exactly what I was looking for, current costs would still prohibit doing this at scale. The game would have to be pay-to-play on a credits bases, which would kill virality and adoption.
Here's a full breakdown for all models. All were 5 or 6 seconds, and ran at the lowest settings. Cost is for a single full video.
| Model | Cost | Note |
|---|---|---|
| Midjourney | $0.06 | Price is per video (so 4 were $0.24) and there is no API available. Just subscription and capped usage based. |
| Hailuo | $0.50 | Per video, both the 2D and 3D models |
| Seedance Lite | $0.09 | At $0.018 per second of output at lowest resolution. Both Image-to-video and Text-to-video have same pricing |
| Seedance Pro | $0.15 | At $0.03 per second of output at lowest resolution. Both Image-to-video and Text-to-video have same pricing |
| Kling 2.1 Standard | $0.25 | At $0.05 per second of output |
Conclusion
Better prompts would have resulted in better videos. But:
- At the current price point I don't see this scaling.
- I would have expected at least one video to give me a 'wow' effect, but none ever exceeded 'meeh'.
- The inputs weren't respected or properly used in any video.
- There regularly would not be a clear winner.
If I had to choose winners:
- Traditional battle: Midjourney, since even thought swords were hallucinated almost all had a winner and the battles were cute.
- Drag race: Seedance Pro text-to-video. It was the only full race, and also had a winner!
I do think similar efforts can be done for text or voice based games. Will explore this more and report on progress!