Dream-1 in the loop

Dream Labs foundation model · MuJoCo sim · live connecting
—task —requests 0RTT —row ——

How it works

Dream-1 is a vision-language-action model. It never sees the simulator. Each request is camera frames, joint numbers and a sentence; each answer is the next 2.4 seconds of joint targets for both arms. The loop below runs until the trial ends. The box that's active right now is highlighted.

Control loop: the sim's cameras and joint state go to the dreamlabs client, which sends them with the prompt to the Dream API; the API returns 48-row action chunks that the executor plays into the sim at 20 Hz. frames + joint state, 5 fps HTTPS: prompt + 9 frames/cam + state q_l_chunk, q_r_chunk: 48 × 7 step(): one row every 50 ms MuJoCo sim2 × YAM arm, 3 camerasphysics 500 Hz dreamlabs clientkeeps last 9 frames/camset_observation · get_action Dream APIDream-1 beta (VLA)api.dreamlabs.ai Executor48 rows at 20 Hz = 2.4 sclip to joint limits physics pauses while the model thinks
9 framesper camera per request, 0.2 s apart: 1.8 s of video history
48 × 7rows per arm: 6 joint angles (rad) + gripper (0 closed, 1 open)
20 Hzone row per 50 ms of sim time, so one answer covers 2.4 s
3 camstop (overhead), left wrist, right wrist; RGB only, no depth

Request (what this sim sends)

KeyValue
promptThe task in plain English
image_window.top / left / right9 base64 JPEGs each, 320×176, oldest first
state.arm_l.joint_pos, state.arm_r.joint_pos6 joint angles per arm, radians
state.arm_l.gripper_pos, state.arm_r.gripper_posGripper opening, 0 to 1
robot.arm, robot.exec_head, …yam, joint: tells the server the body, so it computes hand poses from the joints itself

Fine-tuned models use video.<role> and state.<part> keys instead. The live panel above shows exactly what goes out.

Response

KeyValue
q_l_chunk48 rows × 7: left arm joint targets, then gripper
q_r_chunkSame for the right arm

The client blends each new chunk into the previous one so the arm never jumps. The sim plays every row, then sends the next request. Errors come back as codes: DA-1000 bad key, DA-2000 model warming up or busy, DA-4000 malformed request.

FANUC env: bring your own arm

The FANUC CR-7iA/L cell is not a YAM, so it asks the model in hand-pose mode (robot: other_embodiment). It sends state.right_eef_xyz (m), state.right_eef_rot6 (first two columns of the tool rotation), state.right_grip, zeros for the joints, and a fixed idle pose for the missing left arm. The model then reads only the cameras and the prompt, and answers 48 hand poses (action.right_eef_xyz / rot6 / grip). Our IK turns them into the FANUC's 6 joints. A grip below 0.5 switches the magnet on.

Plugging in another simulator

Anything that can render RGB frames and set joint targets works: Isaac, Genesis, PyBullet, a real robot. The API only sees this data. Whether the model succeeds there depends on how close the arm and camera views are to what it was trained on.