An animated mascot in React Native: Wan 2.2 clips shipped as WebP

I animated Raku's mascot with clips that Wan 2.2 generated from an existing 3D image. The clips ship as animated WebP with an alpha channel, and expo-image plays them. The smallest size of all eight expressions adds up to 4.7 MB.

This path does not suit everyone. If your mascot needs eyes that follow a finger or a mouth that moves while it talks, Rive is still the right tool. I only play one expression clip at moments like "lesson finished", "right answer" and "wrong answer", and a video model's output was enough for that.

Raku and its mascot

Raku is a micro-learning app I build together with a partner. There is reading, a short quiz and a streak, and the mascot takes up a large part of the screen. It appears on the welcome screen, while thinking, on the lesson result and on the error screen. Its eight expressions are idle, thinking, happy, celebrate, angry, surprised, sad and wave. The app is still in development.

I tried Rive first

Search for a Duolingo-style mascot and the same advice comes up every time. One dev.to article says it outright: "The answer is: You don't use Lottie. You use Rive." Its reason is in the line "But the real magic is the built-in State Machine."

It has a point. A state machine is a clean way to switch a mascot between expressions by changing one mood value. So I set it up. I added @rive-app/react-native and the Nitro module, and wrote a contract into the RakuMascot component with mood, talking and the triggers tap, correct, wrong and blink. Rive's React Native docs ask for at least React Native 0.78 and Nitro Modules, and Raku's Expo 57 base met that.

What was missing was the .riv file. Someone has to rig the character and animate it, and I had nobody for that. So I looked for a shortcut. I flattened the 3D mascot to 2D, split it into head, arms and tail layers and drove them from code. The result was bad. Flat, stiff, nothing like the 3D mascot.

I didn't like it and dropped it.

My mistake was changing the style without measuring quality. I should have put the two side by side, and I didn't.

The Rive path is still in the code. RakuMascot looks for a clip first, then for a Rive source, and falls back to a static PNG if neither exists.

Step 1: The source image

The first frame of every clip is the existing 3D Raku image, 640 x 640, on a flat cream background (246, 231, 201). I kept the background flat because the next step cuts it out with a mask.

Each expression gets its own source frame, such as src_happy, src_sad and src_thinking. The model builds the motion from that frame, so the pose in it should already be close to the expression.

Step 2: Generating the clip with Wan 2.2

I run the Wan 2.2 14B image-to-video model in ComfyUI. There are two model files, wan2.2_i2v_high_noise_14B_fp8_scaled and wan2.2_i2v_low_noise_14B_fp8_scaled, and each gets lightx2v's 4-step LoRA. The text encoder is umt5_xxl_fp8_e4m3fn_scaled and the VAE is wan_2.1_vae.

The WanImageToVideo node produces 81 frames at 640 x 640. Two KSamplerAdvanced nodes follow. The first runs steps 0 to 2 on the high-noise model and the second runs the rest on the low-noise model. Both use euler, simple and cfg 1.0, with a ModelSamplingSD3 shift of 5.0. I decode with VAEDecodeTiled in 512 tiles and turn the frames into a 16 fps video. ComfyUI starts with --reserve-vram 1.8 --cache-none --disable-pinned-memory, which is what made it fit on my 12 GB card.

A clip takes 120 to 145 seconds on that card. Slow. These numbers come from my hardware and my settings, so expect different ones on yours.

The prompt

I split the prompt in two. The first half is identical for every expression and pins down the character, the style and the background.

A cute 3D animated raccoon mascot in a yellow hoodie with a navy backpack, Pixar style, plain cream background, static camera, full body stays in frame.

The second half describes the expression as a sequence of actions. The thinking clip says "he taps his chin with his finger, looks up to the side with curious eyes, tilts his head". The wrong-answer reaction uses the same pattern, "he winces, scrunches his face with a cute oops expression, shrugs his shoulders and scratches his head".

The right and wrong reactions are 33 frames, about two seconds. Those finish in 45 seconds.

Two places I got stuck

The WanFirstLastFrame node threw an access violation during VAE encoding. I had wanted to give it a first and a last frame, and the problem went away when I went back to WanImageToVideo.

The second was memory. With another ComfyUI instance open for a different project, the Wan job died with hostbuf_file_reader_read failed. The fix was a wait loop at the top of my run script. If another main.py process is running, it checks once a minute, and it starts its own ComfyUI only when the other one is gone.

I liked the clips a lot the first time I saw them.

Step 3: The alpha mask

The clip's background is cream, but in the app the mascot sits on top of a card or a page, so the background has to go. I run every frame through BiRefNet and get a sequence of mask PNGs. That took 27 to 33 seconds per clip.

The surprised expression came out weak on the first try. I generated four variants. b and c are the same prompt with different seeds, and for d I changed the source frame and the sentence. I haven't picked one yet.

Step 4: WebP in three sizes

I hand the mask sequence to ffmpeg with alphamerge. The filter below brings color and mask to the same size and frame rate and merges them.

[0:v]scale=320:320:force_original_aspect_ratio=decrease:flags=lanczos,fps=10,format=rgb24[c];[1:v]scale=320:320:force_original_aspect_ratio=decrease:flags=lanczos,fps=10,format=gray[m];[c][m]alphamerge,format=yuva420p[o]

The output goes through libwebp_anim at quality 55, compression level 6, looping forever. The same script encodes each clip at three tiers.

The low versions of all eight expressions are bundled in the app, 4.7 MB in total. Mid and high are not wired up yet. I plan to download them from a remote server later.

Step 5: Playing it in the app

Because the clip is an animated WebP, the plain Image component from expo-image is enough, and nothing else has to be installed: no video player, no custom native module, just a file reference and a size. The Expo docs describe autoplay like this: "Determines if an image should automatically begin playing if it is an animated image."

The tier comes from the device's memory, year class and screen.

export function computeRakuTier(
  totalMemory: number | null,
  yearClass: number | null,
  screenPx: number,
): RakuTier {
  const old = yearClass !== null && yearClass <= OLD_DEVICE_YEAR_CLASS;
  if ((totalMemory !== null && totalMemory < LOW_MEMORY_BYTES) || old) return 'low';
  if (totalMemory !== null && totalMemory >= HIGH_MEMORY_BYTES && screenPx >= HIGH_SCREEN_PX) return 'high';
  return 'mid';
}

Under 3.5 GB of memory, or a device class of 2018 or older, gets low. 5.5 GB or more with a screen at least 1080 pixels wide gets high, and everything else gets mid. If the requested tier is missing, resolveRakuClip falls back to the next one down. Right now I only have low, so every device sees low.

Switching between expressions uses a 200 millisecond crossfade, otherwise the mascot appears to jump when the clip changes. On the lesson result, the expression follows the score. 75 percent or more is celebrate, 50 percent or more is happy, and below that is sad.

Hidden clips should not play

The onboarding has slides side by side, and each has the mascot on it. The clip on a slide that wasn't on screen kept playing. I added a paused prop to RakuMascot and gave Image autoplay={!paused}. An inactive slide is now still.

I haven't measured what that does to CPU, I only cut work that nobody could see.

Measuring is next.

I haven't tried iOS yet

This part is open.

The Expo docs say Apple's default codec is faster and uses less memory, but it "may render animated images with incorrect blending or play them at the wrong framerate." They add, "Some animated WebP files also decode very slowly with Apple's codec."

An animated WebP with an alpha channel is exactly that case. I've only looked at the Android emulator. If the picture comes out wrong on iOS, I'll have to switch to the alternative codec the docs describe, and I haven't looked at it.

When this method doesn't work

It fails if the mascot has to react to a touch or to text the user typed. A clip is a fixed motion and can't look at anything. It also fails if the mouth has to match audio, since there is no speech here.

A large number of expressions is a problem too. Each new expression means one Wan run (around 2 minutes), one alpha run (half a minute) and three encodes. Eight is fine. At twenty you'd want to revisit it.

Finally, without a good 3D source image the pipeline doesn't start, because I began with a ready-made 3D Raku image and the model builds the motion on top of it.

Where it stands

All eight expressions run in the app at the low tier. Next I'll pick the surprised clip, download the mid and high tiers from a server and check how alpha WebP actually plays on iOS. Two short clips for right and wrong answers are also generated, and I haven't connected them to the screen yet.