From a song and a scene prompt, we generate a 360° visual experience whose atmosphere evolves with the music’s structure and estimated emotional trajectory.
Listen to structure
Extract downbeats and organize the song into four-bar musical units.
Translate emotion
Predict valence–arousal conditions and turn them into visual guidance.
Enter the scene
Animate panoramic keyframes into temporally connected 360° video.
Abstract
Music visualization can deepen listeners’ understanding and experience by translating audio into visual form. Yet many existing approaches rely heavily on lyrics or produce flat, non-immersive videos, limiting both their applicability to instrumental music and their ability to create an immersive listening experience.
We propose Bring Music The Horizon, an emotion-aware pipeline for music-driven 360° video generation. Given a song, the system estimates a valence–arousal trajectory for every four bars. EmotiCrafter converts these conditions into emotion-aware visual guidance, SEGA provides fine-grained semantic control for panoramic keyframe generation, and image-to-video models synthesize temporally continuous 360° video. The resulting visualization is designed to reflect the song’s emotional progression and temporal structure.
Why immersive music visualization?
Most music visualization is still framed as a conventional screen-based music video. We investigate a complementary experience: a listener can look around inside a scene while its visual atmosphere changes with the song. The goal of this work is to use musical structure and continuous affect as conditions for an evolving, immersive scene.
Scope. This work focuses on qualitative, immersive music visualization. Our paper presents demonstrations with songs across different genres and qualitative comparison with From-Sound-To-Sight.
Method
Pipeline overview. A song’s downbeats and emotional conditions guide panoramic keyframe generation; the keyframes are then animated into a 360° dynamic scene.
1. Music information retrieval
We use All-In-One to estimate downbeat timestamps and functional segments. The song is grouped into four-bar units, and a Dynamic Valence–Arousal regressor predicts one emotion condition for each unit. This produces a time-varying representation of both the musical organization and affective trajectory.
2. Emotion-guided panoramic keyframes
A user supplies a base prompt that anchors the visual concept. For each musical unit, the base prompt and predicted emotion condition are passed through a retrained EmotiCrafter model to obtain an emotional residual. We use that residual with SEGA as guidance in the diffusion process, then use SDXL 360° LoRA to synthesize an emotion-aligned panoramic keyframe. Retraining EmotiCrafter with a custom dataset is intended to reduce unwanted human-activity artifacts and produce cleaner, scene-focused outputs (this is a limitation of the original paper).
3. Temporally connected 360° video
Each keyframe anchors a four-bar video unit. Wan-I2V generates a dynamic scene for the first three bars, while Wan-flf2v generates a transition clip during the final bar to connect to the next keyframe. Concatenating these clips in temporal order yields the final 360° visualization.
Results and demos
The paper demonstrates that the pipeline can generate 360° videos viewable in a VR headset. The visual atmosphere and transitions are designed to follow the estimated emotion conditions at different musical segments, while the four-bar units align scene generation with the song’s temporal structure.
Queen - Bohemian Rhapsody
- Song length
- 6:00 (2:15 in demo video)
- Text prompt
- ”An ancient medieval European street”
- V-A trajectory
- Open trajectory PNG
Michael Jackson - Stranger In Moscow
- Song length
- 5:35 (1:36 in demo video)
- Text prompt
- ”A gloomy afternoon on a rain-soaked street.”
- V-A trajectory
- Open trajectory PNG
Jay Chou - Rice Field
- Song length
- 3:44 (1:22 in demo video)
- Text prompt
- ”Vast wheat fields”
- V-A trajectory
- Open trajectory PNG
Qualitative comparison
We compare our results with From-Sound-To-Sight, with 3 cases:
Contributions
- Immersive music visualization. We extend music-driven visual generation toward a 360° format intended for VR viewing.
- Dynamic music emotion modeling. The pipeline predicts valence–arousal conditions over four-bar units rather than using a single static mood to represent the whole song.
- Emotion-guided scene synthesis. Emotional residuals guide panoramic keyframe generation while preserving a user-provided base scene concept.
- Structure-aware video assembly. Dynamic scenes and transition clips are arranged according to the song’s temporal order.
Limitations and future work
Although this work addresses emotion-aware, structure-guided 360° music visualization, the present system has several limitations:
- Similar valence–arousal conditions can lead to repetitive keyframes.
- Panoramic boundary inconsistencies can create visible seams.
- Output resolution is still limited.
- Forward-moving visual effects are not fully controllable and may affect motion comfort.
- Long prompts can cause a drift from generating omnidirectional imagery toward flat scenes.
These constraints motivate future work on stronger 360° consistency, higher resolution generation, and more controllable immersive motion.
Materials
Acknowledgements
This research is supported by the Yushan Young Fellow Program of the Ministry of Education in Taiwan under Grant MOE-114-YSFEE-0010-008-P1, and by the National Science and Technology Council, Taiwan, under Grant NSTC 115-2813-C-A49-161-E.