#多模态

@FastCompany just published a great piece on @theworldlabs , @drfeifei , Marble, and the idea that spatial intelligence / world models may be one of the next big shifts in AI. I was happy to be quoted in the article, but I also wanted to share more context about my own experience with World Labs and Marble, and why this direction is especially interesting to me. https://www.fastcompany.com/91549046/fei-fei-li-world-labs-ai-gets-physical-models-spatial-intelligence My starting point: volumetric capture — For the past few years I’ve been exploring and using volumetric capture and reconstruction (photogrammetry, NeRFs, 3D Gaussian Splats) mostly capturing locations around Montreal. Alleys, museums, urban interiors. I love every step of it: the capture itself, the pipeline, and what can be done with the output. Turning real spaces into real-time explorable systems. I do this personally, sharing explorations here, and professionally as chief technologist, and co-founder of Dpt. Physical reality + generative manipulation — In my work I’m especially drawn to mixing physical reality with generative and digital manipulation: using physical interfaces (light, clay, ink, ... ) to drive generative AI pipelines, building mixed reality prototypes that reshape your surroundings, or starting from real captured spaces and transforming them using tools like Marble. Like many people, I saw the World Labs announcement on Twitter in September 2024, and Marble when it surfaced in early December. But by then, I already had a sense something was coming. The first conversation — As someone deep into volumetric capture and radiance fields, I obviously knew about @BenMildenhall and his pioneering work on NeRF. To my surprise, Ben reached out to me in late June 2024. He’d been following some of my experiments and wanted to chat about my process and workflows and how I was using this “stuff” creatively. At that point he didn’t share what he was building, but we had a genuinely great conversation about radiance fields, AI, and my work. He was curious about the creative perspective, not just the technical one. When the World Labs announcement dropped a few months later, it all made sense. I understood what Ben had been working on, and why the creative angle mattered to them. Then in August 2025, he invited me to try the Marble beta, and I’ve been experimenting with it since. Experimenting with Marble — The first thing I used Marble for was materializing scene and world concepts during ideation at the studio, and seeing if and how it could fit into our production pipeline. In parallel, I dove into a series of experiments focused on world manipulation: starting from real captured spaces and transforming them using Marble. I’d already been exploring that idea using img2img diffusion with ControlNet on NeRF renders, real-time video streams, and even mixed reality using headset camera feeds. But Marble brings something different. It generates persistent, spatially cohesive 3D worlds that can be rendered in real time across a wide range of devices. That’s a real shift. Experiment 01: Parallel Realities — The first experiment, Parallel Realities, starts from a volumetric capture of a real location, reconstructed as 3D Gaussian Splats. Using Marble, I generate an alternate version of that same space, something informed by the original architecture: abandoned, nature-reclaimed, alternate era. Then, using Spark (World Labs’ 3D Gaussian Splatting renderer for THREE.js) I make both realities coexist in the same spatial coordinate system. From there, I use a portal UX mechanic to let the user step between the real reconstruction and the Marble-generated version. Experiment 02: Hidden Depth The second experiment, Hidden Depth, does not transform a space as much as expand it. A captured location has a visual boundary (a mural, a doorway, a dark corridor) and Marble generates what exists beyond it. For example: a Montreal alley has a painted mural; step through it and you’re inside a world informed by what is actually depicted there. World Labs showcased part of this work here: https://www.worldlabs.ai/labs/showcase/dormant-memories And in their Spark 2.0 post: https://www.worldlabs.ai/blog/spark-2.0 The project page is here: https://www.smallfly.com/dormant_memories/ Why this matters to me — Being able to start from a real 3D Gaussian Splat scene and manipulate it with Marble opens up a lot of ideas. The 3DGS pipeline is becoming an increasingly compelling foundation for exploration, experimentation, and storytelling. What matters most to me right now is more control. The more I can steer the generated scene or world, the more useful the tool becomes. I want more features like the already existing multiple input images and Chisel, the blockout-based approach. I would like better local control, the ability to expand a generated world more and more while preserving coherence, and the ability to directly import 3D Gaussian Splat scenes to be used as a starting point. I want more ways to shape the result, not just a “prompt and hope” approach. — It is exciting to see this field moving from research and demos toward actual creative workflows.

been testing MiniCPM-V 4.6 locally on my DGX Spark this demo is an industrial gauge reader i generated a synthetic control panel with a pressure gauge, temperature gauge, flow display, and tank level bar. because the panel is synthetic, i know the exact ground truth at every frame. MiniCPM gets a frame and has to return structured JSON: pressure_bar temp_c flow_lpm level_pct then i score each reading live: pass = within 5% of full scale drift = within 10% miss = outside that the scenario i was targeting is industrial visual ops. lots of real systems still have gauges, panels, status screens, meters, and legacy interfaces. i wanted to see if a small open source vision model could read that kind of interface and return values another system could actually use. what i saw in practice: digital displays and tank level were easier. analog gauges were harder, especially between tick marks or while moving. but the important part is that the output is measurable. it is not just a captioning demo. MiniCPM-V 4.6 had to read pixels, follow a schema, and produce structured data i could check against ground truth.

Frontier doesn't only mean Anthropic and OpenAI anymore. I built this black hole simulator by simply dropping an illustration screenshot into Kilo Code, switching to @MiniMax_AI M3, and prompting it to "animate this screenshot into a working black hole simulator". M3's vision capabilities are incredible. Total cost: $0.53 Speaking of stars - Kilo Code is about to hit 25k, and we're giving $500 of AI Credits to 2 people once that happens! Check out our repo here: https://github.com/Kilo-Org/kilocode/

GPT-Realtime 2 是操作系统的未来。 我已经试验了它几周，不得不说，它真是相当不可思议。 打开应用、搜索网页，甚至在 Premiere 里剪辑。全都只用我的声音完成。 而且只需几个提示就能设置好。 在这个视频里，我会向你展示具体怎么做。 0:00 开场 1:46 什么是 GPT-Realtime 2？ 4:46 设置它（无需编程） 7:53 修复常开麦克风（按键说话） 9:41 演示：搜索网页 11:38 演示：通过 MCP 连接应用（Obsidian） 14:13 演示：控制 Premiere Pro（无障碍树） 18:00 诚实的注意事项 19:00 结尾