MediaPipe is Google's open-source framework of ready-made, real-time ML perception models — hands, face, pose, objects — tuned to run at 30+ FPS on ordinary laptops, phones, and even in the browser. Its Hand Landmarker takes a plain RGB webcam frame and returns 21 precise 3D points per hand, every frame, no depth camera and no gloves required.
TouchDesigner is a node-based visual programming environment used for live visuals, installations, and stage shows. It thinks in operator families: TOPs (textures/images), CHOPs (channels of numbers), SOPs (3D geometry), DATs (tables/text). The moment hand landmarks become CHOP channels, they can drive anything — particle systems, shader uniforms, lights, sound.
The whole trick of this chapter is one pipeline:
MediaPipe answers "where is the hand?" extremely well but draws nothing. TouchDesigner draws anything but has no idea what a hand is. Wire them together and you get touchless interfaces, theremin-style instruments, and gallery installations — all from a laptop webcam.
Every tracked hand comes back as the same 21 points in the same order, whether the hand is open, fisted, or waving. Index 0 is the wrist; each finger then gets 4 points from knuckle to tip. Memorize just two of them and you can already build gestures: 4 = thumb tip, 8 = index tip.
👉 Hover or tab over any dot in the diagram to see its index and official MediaPipe name:
| Indices | Region | Names (knuckle → tip) |
|---|---|---|
0 | Wrist | WRIST — the anchor every distance is measured against |
1–4 | Thumb | THUMB_CMC → THUMB_MCP → THUMB_IP → THUMB_TIP |
5–8 | Index | INDEX_MCP → INDEX_PIP → INDEX_DIP → INDEX_TIP |
9–12 | Middle | MIDDLE_MCP → MIDDLE_PIP → MIDDLE_DIP → MIDDLE_TIP |
13–16 | Ring | RING_MCP → RING_PIP → RING_DIP → RING_TIP |
17–20 | Pinky | PINKY_MCP → PINKY_PIP → PINKY_DIP → PINKY_TIP |
x, y — normalized 0 to 1 across the image. (0,0) is the top-left; TouchDesigner UVs use bottom-left, so you'll often compute 1 - y.z — depth relative to the wrist, roughly the same scale as x. Smaller = closer to camera. It is not millimeters!Hand detection sounds like one neural network, but MediaPipe splits it into two specialists — and that split is why it's fast enough for live visuals.
A single-shot detector (BlazePalm) scans the whole frame for palms. Palms are rigid squares — way easier to detect than wiggly fingers — so the model stays tiny. It outputs an oriented bounding box.
Cost: the expensive step. Runs on the full image.
Takes only the cropped, rotated palm box and regresses the 21 keypoint positions directly — no per-pixel heatmaps. It also outputs a confidence score and left/right handedness.
Cost: cheap. The crop is small and pre-aligned.
Here's the trick that buys the frame rate: once a hand is found, MediaPipe stops running the palm detector. On the next frame it simply predicts the crop box from the previous frame's landmarks and jumps straight to Model 2. The full-frame detector only wakes up again when the landmark model's confidence drops — i.e., the hand left the frame or got occluded. Tracking a steady hand is therefore dramatically cheaper than finding a new one.
*until you hide your hand behind your coffee mug, at which point the palm detector clocks back in.
The fastest route is the free, open-source MediaPipe TouchDesigner plugin by Torin Blankensmith & Dom Scott (github.com/torinmb/mediapipe-touchdesigner). It runs MediaPipe's GPU web runtime inside a Web Render TOP, so there is zero Python installation pain. Steps:
release.zip from the plugin's GitHub Releases page and unzip it. Keep the .tox files together with their folders.MediaPipe.tox into your TouchDesigner network. A component appears with its own camera selector.hand_tracking.tox in next to it, and wire the MediaPipe COMP's output into it.Because real TouchDesigner screenshots don't survive our comic printing press, here are faithful redrawn network views of what you'll see at each step:
TouchDesigner convention: always export from a Null, never from the source operator. When you later swap the tracker or insert a filter, every downstream reference keeps working because it points at null_hands, not at whatever sits behind it.
Prefer to own the whole pipeline, tweak model options, or run detection on a second machine? Run MediaPipe in a plain Python process and beam landmarks into TouchDesigner over OSC (a tiny UDP protocol TouchDesigner speaks natively). Don't try to pip install mediapipe into TouchDesigner's bundled Python — version mismatches make that a rabbit hole; a separate process sidesteps it entirely.
Step 1 — install the sender's dependencies (system Python 3.10+):
Step 2 — the sender script. Capture, detect, send each landmark as /hand/<index>:
Note the 1.0 - p.y — that single subtraction converts MediaPipe's top-left origin into TouchDesigner's bottom-left world, saving you a Math CHOP later.
Step 3 — receive in TouchDesigner. Drop an OSC In CHOP, set Network Port to 7000, and run the script. Channels named hand/0:0, hand/0:1… appear the instant data flows. Wire it to a Null, same convention as before.
Raw landmarks are just coordinates. Gestures are relationships between landmarks — and the three classics need nothing fancier than a distance formula.
That normalization by palm size (wrist 0 → middle knuckle 9) is the single most-forgotten trick: without it, stepping toward the camera reads as "un-pinching".
Same comparison, flipped: every tip above its PIP joint. Combine with a stable pinch value of ~1.0 and you have a reliable "reset" gesture.
Raw landmarks jitter a few pixels every frame. Before driving visuals, pass channels through a Lag CHOP (0.1–0.2 s) or a Filter CHOP. For on/off gestures add hysteresis: trigger pinch below 0.22 but only release above 0.32 — one threshold means flicker at the boundary.
No webcam handy? Same math, simulated. Pick a pose — the skeleton animates between real landmark configurations, the pinch formula from Section 06 runs live on those 21 points, and its output drives the "Circle TOP" on the right, exactly the way a CHOP export would in TouchDesigner.
This wiring is literally: null_hands → Select CHOP (grab thumb/index tips) → Script CHOP or Expression CHOP (pinch formula) → Lag CHOP (smoothing) → export to the Circle TOP's Radius parameter. Five nodes, touchless volume knob.
max_num_hands at what you actually need — each extra hand is another landmark-model pass per frame.| Symptom | Likely cause | Fix |
|---|---|---|
| No channels appear (Path A) | Camera claimed by another app, or plugin files separated from their folders | Close Zoom/browser tabs using the cam; re-unzip the release keeping folder structure |
| No channels appear (Path B) | Port mismatch or firewall | OSC In CHOP port must equal the script's (7000); allow Python through the firewall |
| Hand found, landmarks jittery | Normal sensor noise | Lag/Filter CHOP downstream; never smooth before gesture thresholds with hysteresis |
| Tracking dies on fast motion | Tracker lost the hand; detector re-arming | Raise camera FPS, lower resolution, raise min_tracking_confidence slightly |
| Y axis feels upside-down | MediaPipe origin is top-left, TouchDesigner is bottom-left | Use 1 - y in the sender, or a Math CHOP: multiply −1, add 1 |
| Everything works, then FPS tanks over minutes | Unthrottled viewer windows and probe overlays | Close operator viewers you're not watching; overlays cost real GPU time |