Author Topic: Virtual control panel using finger tracking  (Read 5052 times)

Offline cloudflint

  • *
  • Posts: 20
  • Karma: +4/-0
    • View Profile
on: June 17, 2009, 03:26:18 PM
Well with my finger tracking system controlling the mouse quite well, I have been doing some more experimentation with finger tracking in general.

I was working on a finger tracking controlled robotic hand using a Lego mind storm kit, but it ran out of batteries. So I decided to take a break and play some flight simulator. I was playing flight simulator when suddenly I thought "Wouldnít this be so much better it felt more like I was sitting at a plane rather than at a computer"

My first thought was to do away with that pesky keyboard; I figure real planes have joysticks or things very similar so that can stay. My next thought was that most planes donít have a big keyboard, they have proper control panels full of buttons and sliders and all sorts of fun things. I know you can buy these things, but that would cost money, after some brain storming however an idea struck me.

Why bother buying or making a complex control panel when I can just draw one and bring it to life instead? Immediately I fired up GlovePIE and got to work. I have been at this a few hours now and have already attained some very positive results.  I started out by sketching out a simple control panel on a piece of A4 paper; this contained a slider for the throttle and 7 buttons.

I then developed a system in GlovePIE whereby I can calibrate the camera to the paper; this allows me to get a reading on where my finger is in relation to the paper. 0, 0 would be the top left and 1, 1 would be the bottom right so 0.5, 0.5 is right in the middle.

Getting readout from GlovePIE I then measured the opposite diagonals of each component. I then developed a simple system where you can hover your hand over a button to "press" it. To press it again you must move your finger from over the button. Pressing a button causes it to simulate a keyboard press; this means it was simply a case of looking up the default keyboard shortcuts in flight simulator and deciding what to make each button on the control panel do.

For the throttle this required something slightly more complex. Using ppjoy I setup a virtual joystick containing a virtual throttle slider like you get on some joysticks. I then mapped the throttle axis to the userís finger position mapped to the height of the throttle.  This was picked up by flight simulator as a throttle automatically meaning the whole system is able to work in flight simulator without making any modifications to flight simulator its self. (Apart for adding keyboard shortcuts for some functions which didnít have any shortcuts assigned)

[youtube]http://www.youtube.com/watch?v=dKeWYBmwhOc[/youtube]

Now that I have buttons and sliders working, I am thinking that I might try out Johnny lees idea of tagging objects on a surface, rather than having a single piece of paper, I could have lots of pieces of paper each with a single control or set of buttons on them.

This means that a person could arrange the controls like they are laid out in the actual plane then "tag" them by touching either corner of them.  This would give the advantages of having a big simulator setup and would have the added advantages of allowing the users to lay out the control any way they wished and would not require expensive equipment.

This project only uses one wiimote so far. It started out as a bit of fun but now is becoming a good example of one of the ways finger tracking could potentially be used in the future! I figure once i have it working i could release it and have a control panel that you can just print off. Then people could download the script, print of the control panel and it would be it ready to go.
« Last Edit: June 17, 2009, 04:18:26 PM by cloudflint »



Offline eyucel

  • *
  • Posts: 39
  • Karma: +1/-0
    • View Profile
Reply #1 on: June 18, 2009, 02:05:52 AM
Very smart idea and of course applied creativity. Please keep us informed.



Offline cloudflint

  • *
  • Posts: 20
  • Karma: +4/-0
    • View Profile
Reply #2 on: June 18, 2009, 11:27:49 AM
Yeah, ill keep you posted on my progress  :)