Education in 2020: the student-tracking camera I built my 6-year-old son

What is it?

We’ve assembled a kid-tracking (or any-object-tracking) camera rig that can be used as a webcam (with Zoom, Google Meet, etc) on a normal computer (a Macbook in this case).

What does it do?

This camera rig is like a smart, AI-enabled webcam. It tracks my son (pictured) and daughter and keeps them in view of their teacher in their great new school program, Modulo, run by Manisha Snoyer. Now my kids have more freedom to move!

Why did I do this?

COVID has distanced all of us. We’re deep in the world of e-learning, and the screens we once eschewed for our children have become a constant in our lives, so we’d better make the best of it.

Now that we have this camera, my kids can move around the room, jump up and down, romp around the floor, do jumping jacks, and work on their computers, and the camera will follow them around, ensuring that their teacher can see them and talk with them. 

How do I make one?

Parts list:

1. DJI Ronin-S Gimbal 

   1. Note: or a comparable gimbal - anything with sufficient size + load capacity for the camera, phone and other gear. A small gimbal like a DJI Osmo Mobile will probably not work.

   2. Note: your gimbal will need to have “Face Tracking” or “Object Tracking” -- many from DJI and Zhiyun have these features.

2. Sony ZV-1 Camera

   1. Any camera with HDMI out will work, but this camera supports uncompressed HDMI out and Sony seems to be investing in making it work natively as a webcam (right now only for Windows).

   2. I’m going to try out using another phone instead of the ZV-1, and this may also decrease the cost of the build.

3. iPhone 11 Pro (any modern smartphone that works with the Ronin App will work for “Auto Track” / “Face tracking”)

   1. Note: DJI’s app doesn’t seem to work very well on older Android phones - I tried a Nexus 5x and the app constantly failed to orient the camera the right way, sending the gimbal in random directions.

4. Elgato Cam LInk 4k or any HDMI capture interface that produces at least 1080p output and accepts HDMI input.

5. And enough “stuff” to join it all together in a robust way:

   1. A phone mount capable of mounting the phone immediately above the camera lens (so that the lens of the camera and the lens of the phone see roughly the same thing)

   2. A simple hot shoe/ cold shoe adapter that lets us mount the phone holder

   3. A lightweight micro-HDMI to HDMI cable

   4. [Optional] A micro-USB cable to charge the camera (for sessions long than 45 minutes or so this is important)

   5. [Optional] A lightning cable or other charging cable for the phone you’re using (I didn’t encounter an issue with this in >2 hours of use, so I prefer to keep the cables away).

How to assemble:

1. Attach the phone and mount to the ZV-1 camera (or similar camera) via the camera shoe + phone adapters linked above.

2. Attach the ZV-1 to the Ronin-S gimbal (the Ronin-S manual explains balancing, etc, if you haven’t used a gimbal before).

   1. Note: you will want to balance the gimbal with the camera’s screen extended out, and the phone mounted in its “final” configuration, so that your gimbal is properly balanced when you turn it on. Failing to do this might mean the gimbal doesn’t work or cannot overcome the weight imbalance.

3. Plug the micro-HDMI cable into the camera and the capture device (Elgato Camlink 4k or similar) and plug the Camlink into your computer.

4. Verify that the camera outputs video (adjust camera settings if not) -- open Zoom settings → Video (or similar) to confirm.

5. Setup the Ronin app on the phone attached to the camera and go into Create → Active Track (or similar depending on your gimbal manufacturer and model).

6. Enable active tracking on the Ronin app.

7. Your camera will now follow your subject, and output video to your computer for use with Zoom, Google Meet, Facetime, and virtually any other video platform that supports a camera connected to your computer. 

What else did we try?

I tried much smaller/simpler setups at first:

1. Using what should be a perfect, purpose built product: the DJI Osmo Pocket - a gimbal enabled 4k video camera. But this camera cannot be used as a webcam and does not have HDMI out! DJI - please fix this!

2. Attaching a cheap webcam to my phone (couldn’t balance on a smaller gimbal, would likely work with the right rig to tie everything together and a sufficiently strong gimbal)

3. Attaching two phones together (this will work with a gimbal like the Ronin-S but not with smaller gimbals like the Osmo Mobile 4).

4. Investigated the Facebook Portal - this looks like a good product from a functionality perspective, but I agree with CNET and others on the privacy concerns. 

What do we wish we had?

DJI, FIMI, Sony, et al: embrace flexibility!

This whole project would have been trivial if the DJI Osmo Pocket, or the FIMI Palm, or similar cameras simply worked as webcams natively on Mac OS and Windows or if they supported HDMI output.

Others have gone to incredible lengths to get the DJI Osmo Pocket usable as a webcam, but the amount of custom hardware and software required defeats the purpose of using the small camera in the first place. I contacted the author of Cosmostreamer (linked above) and he indicated that his system likely would not work well for the use case we described.

Tell me about what you build!

Have a cool hack to improve your kid’s e-learning experience? Tell me about it!