Malazan Empire: Abyss - Viewing Profile - Malazan Empire

Researchers at Google [...] released an AI model called PaLM-E this week that combines language and vision capabilities to control robots, allowing them to complete tasks autonomously in the real world—from getting a chip bag from a kitchen to sorting blocks by color into corners of a rectangle.

According to the researchers, this is the largest Visual Language Model (VLM) reported to date, with 562 billion parameters. This AI has a “wide array of capabilities” which includes math reasoning, multi-image reasoning, and chain-of-thought reasoning. [...] the AI uses multi-task training to transfer skills across tasks, rather than being trained on individual tasks. [...] the AI model when controlling robots even displays "emergent capabilities like multimodal chain of thought reasoning, and the ability to reason over multiple images, despite being trained on only single-image prompts."

PaLM-E is based on Google’s previous large language model called PaLM and the E in the name stands for “embodied,” and refers to the model’s interaction with physical objects and robotic control. [...]

The robot is able to generate its own plan of action in response to commands using the model. When the robot was asked to “bring me the rice chips from the drawer,” PaLM-E was able to guide it to go to the drawers, open the top drawer, take the rice chips out of the drawer, bring it to the user, and put it down. The robot was able to do this even with a human disturbance, with a researcher knocking the rice chips back into the drawer the first time the robot picked it up. PaLM-E is able to do this by analyzing data from its live camera.

[...] “This enables not only embodied reasoning but also question answering, as demonstrated in our experiments.”

[...] “Perhaps most exciting about PaLM-E is positive transfer: simultaneously training PaLM-E across several domains, including internet-scale general vision-language tasks, leads to significantly higher performance compared to single-task robot models.”

Big Tech Is Now Developing Powerful AI Brains for Real-World Robots