A Video-based Motion Tracking System

Motion Tracking SetupOn octo­ber 2nd 2007, a modi­fied ver­si­on of the moti­on trac­king sys­tem i ori­gi­nal­ly devel­o­ped for my Light­room pro­ject, will be used in an inter­ac­ti­ve sound-instal­la­ti­on of Wer­ner Urban at the Kin­der­boe­ken­bal 2007 at Muziek­ge­bouw aan ‘t IJ in Amster­dam. For tho­se inte­rested i will sha­re here a litt­le bit of the tech­ni­cal back­grounds of this system.

Hard­wa­re

CameraThe sys­tem con­sists of a fire­wi­re came­ra (from Uni­Brain), with a wide-ang­le lens (Mar­shall Elec­tro­nics), an omnidi­rec­ti­o­nal infra-red illu­mi­na­tor (home­ma­de, for very dark ligh­ting con­di­ti­ons) con­nec­ted via an 11 mtr. long fire­wi­re cable (also from Uni­Brain) to a com­pu­ter. This com­pu­ter is run­ning a soft­wa­re­patch (writ­ten in the Max/Jitter pro­gram­ming envi­ron­ment). I usu­al­ly pla­ce the came­ra direct­ly abo­ve the sce­ne being track­ed, so that the bar­rel dis­tor­ti­on of the wide ang­le lens isn’t too annoying. It can track and report up to 8 dif­fe­rent posi­ti­ons simul­ta­neous­ly (thanks to the excel­lent exter­nal object for Jit­ter writ­ten by Randy Jones: 2up.jit.centroids). The­se coo­r­di­na­tes can then be used for trig­ge­ring all sorts of acti­ons in Max. In the case of the ori­gi­nal pro­ject it trig­ge­red a matrix of 64 lights and the spa­ti­a­li­za­ti­on of sound in a quadrap­ho­nic spea­ker setup.

Soft­wa­re

The trac­king part basi­cally con­sists of three sepe­ra­te max-pat­ches con­nec­ted to each other, one for set­ting came­ra para­me­ters and dimen­si­ons, one for video pre-pro­ces­sing and the last one for the trac­king itself.

Camera & DimensionsSet­ting came­ra para­me­ters and dimensions.
It con­sists of get­ting the live video ima­ge into the patch and set­ting came­ra para­me­ters, which is some pret­ty basic tuning of the jit.grab object (like set­ting dimen­si­ons, fra­me rate and came­ra-spe­ci­fic set­tings like com­pres­si­on etc.) For good trac­king (and a speedy patch) the ima­ge dimen­si­ons don’t have to be big at all. I nor­mal­ly grab a video­st­ream of max. 160 x 120 pixels with a fra­me rate of 10 fps.

Video PreprocessingVideo pre-pro­ces­sing.
Some pre-pro­ces­sing is nee­ded on the video ima­ge to get a good and sta­ble ima­ge for trac­king. What i did was to record (in advan­ce) a short movie from the emp­ty spa­ce being track­ed, load it in to the patch and use it as a refe­ren­ce movie. This (loo­ping) movie is sub­trac­ted from the live input from the came­ra, resul­ting in a video­st­ream that shows only pixels that are dif­fe­rent then the refe­ren­ce movie (i.e. moving objects, peo­p­le, etc.). Final­ly, i nor­mal­ly make the ima­ge mono­chro­me for trac­king only grey­sca­le valu­es (alt­hough you can track colors) and apply some jit.fastblur to get rid of most of the noi­se befo­re sen­ding it to the trac­king part.

TrackingTrac­king.
Here comes in the magic of the 2up.jit.centroids object. I used a lot from the help patch inclu­ded with the object. It all is pret­ty much self-explana­to­ry. You pick a value you want to track and 2up.jit.centroids reports up to 8 coo­r­di­na­tes if it sen­ses this value any­whe­re in the ima­ge. From here on, you can use this coo­r­di­na­tes for any­thing one can think of.

Things to watch out for

Light.
A lot depends on ligh­ting con­di­ti­ons. A small chan­ge in brightness (for instan­ce clouds moving in front of the sun) can ren­der the sys­tem total­ly unu­sa­ble. That’s why i made an infra­red illu­mi­na­tor from a stack of blue and red Lee fil­ters atta­ched around a 150 Watts light bulb, to make the sce­ne as even­ly lit as pos­si­ble even if the sce­ne itself has to stay dark.

Stea­dy­cam.
The came­ra has to be tightly fixed in posi­ti­on, becau­se a small move­ment of the came­ra will make all pixels dif­fe­rent from the refe­ren­ce movie, resul­ting in a jit­ter patch frea­king out.

Back­ground (floor) and dif­fe­rent mate­ri­als (clo­t­hes, hair) affect tracking.
This is the most unre­lia­ble part of the sys­tem. Some­ti­mes a per­son wears clo­t­hes of a cer­tain color or mate­ri­al that blends too much with the back­ground­co­lor/-mate­ri­al, resul­ting in an ima­ge that doesn’t have enough con­trast. Thus making it hard for the trac­king object to track a spe­ci­fic color. Eve­ry situ­a­ti­on demands a litt­le fine­tu­ning; you can set a tole­ran­ce and a tres­hold for 2up.jit.centroids, to set a ran­ge of valu­es to be track­ed and a mini­mum value.


Creative Commons License

A Video-based Moti­on Trac­king Sys­tem is licen­sed under a
Cre­a­ti­ve Com­mons Attri­bu­ti­on 3.0 Nether­lands Licen­se.