Meta's Muse Speaks, Gains Expressive Face (US First)

Meta’s AI Assistant Muse Now Features Real-Time Voice and Expressive Avatars

Meta has enhanced its AI assistant app, Muse, with real-time voice conversation and animated virtual avatars that respond with expressions, allowing users to speak instead of typing. The update, rolled out on November 5th, is currently available only in the United States.

Enhanced Interaction with Muse

Users can now engage in spoken dialogue with Meta’s personal AI agent, Muse, eliminating the need for text-based input. The app also features a virtual avatar that mirrors the user’s tone and emotion during conversations. This new functionality was launched on November 5th in the U.S. and is accessible on the mobile, web, and Mac versions of the Muse app. Users can initiate voice conversations by tapping the phone icon in the upper right corner of the screen and can interrupt the AI at any time, similar to interacting with a human.

Use Cases for Voice Interaction

Meta has highlighted two primary use cases for this new feature. The first is for brainstorming and planning, where users can articulate their thoughts as they develop them. For instance, when planning a trip, a user might start by saying, “I want to wake up somewhere scenic, and I want the whole trip to revolve around a specific meal.” Muse will then process this input and continue the conversation. The second use case is for practice and rehearsal, such as mock interviews, presentation run-throughs, or sales pitches. The ability to receive immediate responses and reply in real-time creates a more authentic experience.

Expressive Avatars and Display Options

The virtual avatars are designed to convey subtle expressions and gestures that align with the conversation’s tone. They also react to the user’s speech, signaling that they are actively listening. Users have three display options: audio only, the avatar positioned alongside the conversation, or a full-screen, face-to-face interaction. A wide selection of hundreds of voices is available.

A demonstration video showcased a user practicing for a job interview with a cartoon character wearing a cowboy hat. The user asked the AI to simulate an interview, and after the practice session, requested it to email the recruiter to confirm the date. Muse, acting as the hiring manager, responded positively to the user’s experience in business, their MBA, and their coffee company, suggesting they lead with the coffee company to highlight their initiative. The avatar initially appeared as a small circular icon above the conversation before expanding to fill the upper half of the screen.

Availability and Future Rollout

Muse was initially launched in the U.S. in September. Meta announced on October 28th that it had also become available in Singapore, with access restricted to users aged 18 and above. However, the voice conversation and expressive avatar features are currently exclusive to the U.S., and Meta has not yet provided a timeline for their release in other regions.

Voice conversation is slated to be supported on AI glasses and the Muse Charm, a pocket-sized device introduced at Connect in September specifically for interacting with Muse. Further details are expected later this year.

Pricing and Token System

Meta’s support documentation indicates that Muse can be used for free with limited usage. The “Power” plan costs $20 per month and provides 500 million Muse tokens per week, while the “Maximum” plan is $100 per month and offers 3 billion tokens per week. However, the documentation does not specify whether voice conversations or avatars incur charges or how they consume tokens. This aspect remains unclear.

Underlying Technology: Real-Time Generation

Meta’s AI research blog explains that Muse Realtime Voice handles understanding and responding to user input, outputting encoded speech known as “voice tokens.” These tokens contain both the spoken content and the intended tone. Simultaneously, these tokens are converted into audio and fed to Muse Realtime Avatar, which generates the visual representation, ensuring lip-sync and synchronized expressions.

Muse Realtime Avatar utilizes a diffusion transformer model. This model generates video in short segments, using the most recent frame as a reference for the next, allowing for continuous generation regardless of conversation length.

A significant challenge has been achieving speed. The original “teacher model” required 120 computations per segment (40 steps, each with 3 iterations). Meta developed a “student model” through distillation, reducing the computations to just 2, a 60-fold improvement, while maintaining near-teacher model quality. The system generates vertical video at 448×768 resolution at 25 frames per second, with an approximate latency of 870 milliseconds from the user finishing speaking to receiving the first frame and audio. On a NVIDIA GB200, generating 8 frames (0.32 seconds of video) per step takes only 20 milliseconds.

In terms of cost efficiency, optimizations such as 4-bit quantization have increased service capacity eightfold. A single GB200 can simultaneously handle 12 real-time conversations. This optimization was developed in collaboration with NVIDIA.

Competitive Benchmarking

Meta conducted its own comparative evaluation against competitors. In tests, evaluators engaged in 2-3 minute conversations with the same character using Muse, Runway Characters, and HeyGen LiveAvatar (which Meta identifies as a leading commercial real-time avatar system). Muse was preferred over Runway Characters 78% to 22% and over HeyGen LiveAvatar 88% to 12% in overall preference, with Muse also scoring higher in visual quality and behavior. However, Meta noted that the difference in “behavior” compared to Runway was not statistically significant.

Unanswered Questions

Several aspects of the new feature remain unaddressed by Meta.

Firstly, regarding avatars, Meta’s research blog states that the model can use “any image” as a reference for real-time animation, with photorealistic portraits displaying subtle expressions and full-body illustrations performing gestures. However, the blog’s demonstrations showcase model capabilities and do not necessarily represent the full range of avatars available within the Muse app. The announcement also does not clarify whether users can upload their own photos or images of others as avatars.

Secondly, concerning watermarking, Meta has stated that generated videos will embed invisible watermarks using Video Seal technology for source traceability, without increasing latency.

Thirdly, regarding cost, Elon Musk commented on X on November 1st that the operational costs of Grok Bot were unsustainably high relative to revenue. Muse’s avatar feature supports only 12 simultaneous conversations per GB200. Meta has yet to disclose the duration of free usage or the token allocation for paid plans.