Expressions, lip sync and the alternative Live2D path

An avatar can have all the expected facial-control names and still feel lifeless. The runtime has to decide when to move, how strongly, and how competing movements combine. Avashi’s animation work addressed this with procedural layers for idle motion, blinking, gaze, emotion and speech.

The source contains TypeScript runtime components and a Python procedural engine. The latter makes layer priorities, bone rotations, blendshape weights and ADSR envelopes explicit. The value is control over time and ownership: several systems can contribute to the face without silently overwriting one another.

Separate behavior from deformation

Behavior states describe the conversation: idle, listening, thinking and speaking. Deformation controls describe the asset: jaw opening, eyelid motion, head rotation and mouth shape. A state should select animation behavior, while an asset adapter maps that behavior to available controls.

This prevents a conversation service from depending on one vendor’s morph names. It also makes missing capabilities explicit. If a model has no calibrated smile, the renderer should use a supported fallback rather than assume every avatar implements the same expression vocabulary.

A useful state transition is triggered by an actual event. Receiving a model token is not the same as audio playback beginning. Likewise, the final token does not mean the last audio buffer has finished. The visual speaking state should follow the playback lifecycle.

ADSR makes expression timing inspectable

An ADSR envelope splits an expression into attack, decay, sustain and release. Attack raises intensity, decay settles it, sustain holds it and release returns it toward neutral. The expression’s shape and its timing can therefore be edited independently.

The practical advantage is composability. A brief acknowledgment can use a small, short envelope. A longer explanation can retain a softer expression without restarting a full facial movement on every token.

Avoid fixed frame increments such as “add 0.02 per frame.” Their speed changes with device frame rate. Evaluate envelopes against elapsed time, and bound unusually large deltas after a suspended tab resumes.

def release_weight(start_weight, elapsed, release_seconds):
    if release_seconds <= 0:
        return 0.0
    fraction = min(1.0, max(0.0, elapsed / release_seconds))
    return start_weight * (1.0 - fraction)

This standalone example expresses timing behavior, not the full project implementation. The important test is that the same elapsed time yields the same result at different rendering rates.

The mouth needs an arbitration rule

Speech and emotion often drive overlapping controls. Adding a full smile to a full open-mouth viseme can produce an implausible face. The source’s separation between upper-face emotion controls and lower-face speech interactions points toward a region-aware composition rule.

One policy gives speech priority in the mouth while retaining emotion around the eyes and brows. Another attenuates lower-face emotion as speech intensity rises. Both need clipping and asset-specific calibration; simply summing arbitrary weights can exceed the intended deformation range.

Gaze and blinking also interact. A blink layer should not fight a look-down control every frame. Bone rotations need a consistent local coordinate convention, and layers must agree whether they supply absolute values or offsets.

Lip sync has several possible inputs

The repository contains text-to-viseme, audio-analysis and Rhubarb-related paths. These represent different trade-offs. Text-based cues are available early but lack exact pronunciation and timing. Audio-driven cues follow the signal but can arrive later. A phoneme or alignment stream can be more precise when the speech provider supplies it.

For live speech, synchronize cues to the audio playback clock. If a sentence waits in a buffer, animating from its synthesis timestamp makes the mouth move before the user hears it. For prerecorded audio, an offline cue sequence can be inspected and replayed deterministically.

An interruption invalidates both audio and its animation cues. Use a response or generation identity so a late viseme packet from an abandoned turn cannot reopen the mouth during the next one.

Live2D was a different representation experiment

Avashi also contains a separate Live2D playground using Cubism-related loading and PixiJS, with expression and motion libraries. This gave the project another way to explore an expressive character without carrying the same 3D scene.

The trade-off moved rather than disappeared. A Live2D model needs prepared artwork, deformation parameters, expressions, motions and asset-specific tuning. Converting a rendered 3D character into a high-quality Live2D character is not automatic. The integration must understand each model’s parameters and available motions. Live2D SDK documentation

The project’s playgrounds provide evidence of the exploration. They do not establish a measured mobile-performance victory or a completed replacement of the 3D path.

Evaluate motion through fixed clips and interactions

Use a repeatable script: greet, listen, answer, pause, interrupt, resume and return to idle. Keep the input audio and random seed fixed when comparing expression settings. Inspect mouth/audio offset, discontinuities, gaze stability and whether the character settles after cancellation.

Automated tests can check finite weights, allowed ranges, stale-generation rejection and frame-rate independence. Human review should compare naturalness and expression appropriateness. Neither replaces sustained rendering tests on the target phone.

The animation layer’s job is to make conversational state visible without inventing it. That becomes especially important when the next layer introduces streaming text, tool calls and live voice.