Skip to content

XTac UMI G1 Developer Kit

The Developer Kit (PC edition) runs collection on your own x86 collection PC: the grippers and headset connect to it directly by cable, recording runs on open-source LeRobot and produces a LeRobotDataset v3 you can push to the Hugging Face Hub. The collection software and gripper SDK are open source, built for research and algorithm teams who want to customize freely.

If you would rather work without a computer and operate from a tablet in the field, choose the Backpack Kit; the two are compared in Product line and configuration comparison.

The collection workflow

① Getting ready

Once per collection PC and per device

  1. Know the hardware
    Parts, wiring and power-on order
  2. Install the environment
    setup_env.sh or the Docker image
  3. Configure the host
    Serial access, ModemManager off
  4. Set up the Pico4 Ultra Enterprise
    Developer mode, XR app, trackers
  5. Calibrate the leaders
    Zero and travel span, once each

② Collecting

In this order, every session

  1. Power on and connect
    Grippers in, trackers on, PC Wi-Fi off
  2. Start the service and XR app
    PC Service, then the XR app
  3. Preview check
    lerobot-teleoperate in Rerun
  4. Full recording
    lerobot-record, keep XR running

③ The data

After recording

  1. Check completeness
    lerobot-check-dataset
  2. Upload to the Hub (optional)
    lerobot-push-dataset-to-hub
  3. Understand the format
    What each frame records

To mount the follower gripper on a robot and drive it from code, see Follower gripper; to call the gripper SDK directly, see SDK & development.

System components

Collection runs inside one lerobot-record process on the collection PC: each device is read independently, every frame takes the latest value from each stream, and the paired frames are written to the dataset. Hover over (or tap) any block to see where its data comes from and where it goes; the buttons above switch between the three tiers.

  • The leader grippers connect over USB: the opening is read by xense.taccap (100Hz push), the visuotactile images by xensesdk, and each wrist camera is matched to its gripper by serial number.
  • The trackers sit on top of the grippers and are tracked by the headset, which hands the poses to the XenseVR PC Service over a cable (recommended) or Wi-Fi; the collection program then transforms them to the gripper tip (TCP). Tier ③ also brings the headset stereo images and head pose.
  • lerobot-record takes the latest value of every stream each frame, pairs the previous frame's observation with this frame's pose and opening, and writes a LeRobotDataset v3; camera streams are encoded while recording, with hardware encoding on an NVIDIA GPU.

Collection comes in three tiers depending on the connected devices, chosen with --robot.type:

Tier Devices needed Data recorded
① Grippers only Leader grippers Tactile, wrist cameras, opening
② With wrist pose Grippers + trackers + headset Adds the gripper pose
③ Full setup Same as ② Adds headset stereo images and head pose

Commands are in Data collection.

What each frame records

Taking a bimanual setup as the example, each dataset row is the observation from frame t-1 plus the action from frame t. Hover over (or tap) any data item, source or storage block to see where it comes from and where it is stored; the buttons switch between the three tiers.

  • The observation comes from frame t-1 and the action from frame t, so the action leads by one step; the first frame of each episode has nothing to pair with, so each episode is one frame shorter.
  • The poses *_tcp.* and head_camera.* are 9-D each: position x, y, z plus a 6-D rotation, in a world frame of X forward / Y left / Z up.
  • Dimensions per tier: ① state and action 2-D each, 6 image streams; ② 20-D each, 6 streams; ③ 29-D each, 8 streams.
  • With a single gripper (--robot.type=taccap_gripper) the keys have no left_ / right_ prefix and the wrist camera is wrist_cam.

Field details are in What each frame records, the dataset format in Dataset.

Developer Kit open-source repositories

Everything the Developer Kit runs on is open source except xensesdk, whose license is not stated; the table lists each repository and what it is for. The Backpack Kit's collection software is not open source and is not listed here.

Repo / package License Notes
xense-taccap-lerobot Apache-2.0 Data-collection repo, customized from lerobot 0.5.1
xense.taccap Apache-2.0 Gripper SDK, included in the data-collection repo as a submodule; developer docs
xensevr_pc_service_sdk Apache-2.0 Pico4 Ultra Enterprise PC service, installed as a .deb
xensesdk not stated Visuotactile sensor SDK, installed by the setup script; developer docs
xense-lerobot-viewer Apache-2.0 Open-source local visualization tool for browsing LeRobot datasets in the browser