Author Topic: Finger tracking project im working on  (Read 14613 times)

Offline cloudflint

  • *
  • Posts: 20
  • Karma: +4/-0
    • View Profile
on: April 13, 2009, 09:31:57 AM
Hi, i thought this minght intrest some people, its a project im working on to create a "free air touch screen" system.

What im doing is creating two pieces of software. the first is the finger tracking software to interperet the gestures e.t.c. The second is a filing system optimised for working with the finger tracking system.

Heres the setup im using:

[youtube]http://www.youtube.com/watch?v=CW2ilV8eZ0Q[/youtube]

Ive basically just got a monitor sitting horizontally on my desk, and an infra red array shining on the desk with a wiimote built into the array.

Heres a video of me using the system to browse a web page:

[youtube]http://www.youtube.com/watch?v=gZPs3Rmf53U[/youtube]

At the moment the system still has some limitations. I only have one array at the moment so i can only get 2d tracking. I hope to get a second infrared array soon so i can get full 3d tracking. At the moment i am using the index finger to determine the cursor position and the thumb to click. Unfortunatley you can only do a single click so you cant drag or anything like that yet. Once i have improved the finger tracking software i will be able to use it with the software i have developed to work with the finger tracking.

For this system i used an array of 130 leds which in total runs at 12 volts and draws about 1 amp. The reflective material i am using is from an arm band designed to be worn while bike riding, i bought these for about 4 pounds. The system i demonstrated in the video was programmed using glovepie.

The ultimate aim of this project is to create a virtual desk which will be projected onto a real desk. The user will be able to drag windows out on their actuall desk using their fingers and move the windows around, resize them e.t.c. basically any desk will be able to turn into a 3d touch screen!





Offline Konum

  • *
  • Posts: 5
  • Karma: +1/-0
    • View Profile
Reply #1 on: April 16, 2009, 06:02:13 AM
Woahh! It looks really nice and running smoothly. What code are you using? I'm really interested in this, as i'm working in an university project focused on finger tracking interfaces.

Thinking about it, the clik and drag issue, you could assign the dissapear event (of the thumb) to mouse down and the appear of it as the mouse up, so as your thumb it's "off" you could drag thins around.
« Last Edit: April 16, 2009, 06:39:14 AM by Konum »



Offline cloudflint

  • *
  • Posts: 20
  • Karma: +4/-0
    • View Profile
Reply #2 on: April 16, 2009, 01:35:12 PM
Hey, im doing this for a university project as well :D Im examining wether finger tracking systems can be used as a viable alternative to stuff like touchscreen or the traditional mouse.

The software im using is called glovepie, its a scripting system for creating input emulators.

I managed to solve the dragging isue after reading the glovepie documentation which i probably should have done beforehand. The problem that i was having was that i was saying "if you cant see the thumb set the left mouse button to down" but because of the way glovepie works every frame it simulated a left mouse button down, resulting in it just clicking like mad rather than dragging, i managed to get fix this by only putting the left mouse button down if the left mouse buttons state was false and rather than just saying

Code: [Select]
mouse.leftbutton = true
ive used a toggle function to toggle its state.

The only problem i have found is that when you click as your thumb passes out of view there is some slight "jitter" so to speak which means as you click it does lots of little fast clicks before staying clicked which im currently working on fixing.

Here is the download site for glovepie, it includes lots of sample scripts for the Wiimote adn it has pretty good documentations so its really easy to get stuff going quickly.

http://carl.kenner.googlepages.com/glovepie_download



Offline Konum

  • *
  • Posts: 5
  • Karma: +1/-0
    • View Profile
Reply #3 on: April 16, 2009, 02:06:42 PM
Thank you very much ^^

I hope to have a Wiimote and a IR array next week so i can start playing around with it. I'll share here problems or solutions I find.

Greetings :)



Offline cloudflint

  • *
  • Posts: 20
  • Karma: +4/-0
    • View Profile
Reply #4 on: April 16, 2009, 02:44:01 PM
greetings aswell ;)

In the last ten minutes ive managed to attain right clicking by measuring the size of the points it can see then ive said if the persons "clicks" and their hand is held near the display thats a left click but if they hold their hand higher than a certain point above the display it will do a right click instead. Which is pretty awsome if i do say so my self  :P

Another random idea i just had to make use of the excess wiimote view that i have is to allow people to work the scroll wheel by clicking to the side of the display.

Ill post a video once i get it functioning smoothly.



Offline cloudflint

  • *
  • Posts: 20
  • Karma: +4/-0
    • View Profile
Reply #5 on: April 18, 2009, 03:13:00 PM
Hey heres a video of my using the tracking system to right click and drag and drop and stuff like that.

[youtube]http://www.youtube.com/watch?v=QPy6uy3aNGg[/youtube]

The software I was using there will eventually be projected onto a desk the idea beaing the black background wont show up so the desk will just look like a regular desk but youll be able to have these windows floating around it and stuff and hopefully it will be a fairly big area that wil be covered allowing you to have files floating around your desk and stuff. The software for the virtual desk is actually really bloated and slow, im gonna be doing alot of tidying up so it runs smoother.

Ive also made a virtual scroll wheel by allowing the user to click outside the dsiplay then drag up and down.

I think the camera I used may be interfereing with the tracking since as soon as I start recording the system seems to decide to start performing really badly then when i stop recording it starts working great again :D




Offline Konum

  • *
  • Posts: 5
  • Karma: +1/-0
    • View Profile
Reply #6 on: April 18, 2009, 06:23:35 PM
Looks really, really nice :D Keep up the good work ^^



Offline cloudflint

  • *
  • Posts: 20
  • Karma: +4/-0
    • View Profile
Reply #7 on: April 25, 2009, 10:26:26 AM
Hey, i now have a second infrared array so i now have full 3d tracking and it is good :D

I also learned about system hotkeys so i can now communicate with the filing system software from glovepie much better by simulating combinations of key presses, not the most elegant solution but it works! I have also introduced a "deadspace" to the system so you have to move your finger more than a certain amount before it will move the cursor. This makes it much easier to hold the cursor still while double clicking.

Ive managed to get my menus working how i origionally intended. When you want to manipulate data you are obviously gonna have your hand fairly low for maximum accuracy. So when the hand is below a certain level the menus are hidden, but when you lift your hand a certain amount above the display the menus suddenly appear. This means the menus do not clutter up your view and you cant accidentally click them when doing file operations and stuff like that.

I have added a Z threshhold step to the calibration procedure, basically you hold your hand at whatever height you like. Then that level will be the threshhold for switching between "high" operation where the menus' show up and "low" operation where they are hidden.

All that is left to do now is hook the system up to a projector and make a few more tweaks to the amount of smoothing and I shall be finished!

[youtube]http://www.youtube.com/watch?v=m5XPV6uxP7U[/youtube]



 

« Last Edit: April 25, 2009, 10:29:23 AM by cloudflint »



Offline benpaddlejones

  • Support Admin
  • *
  • Posts: 1178
  • Karma: +49/-0
    • View Profile
    • Inter.Connect.Ed
Reply #8 on: April 25, 2009, 05:34:29 PM
cloudflint
Looks awesome, if you want to share the application fell free to uploaded to the download section. If you do let me or Dice know so we can approve it.

benpaddlejones :)


See my full profile and links on my Google Profile: http://www.google.com/profiles/benpaddlejones


Offline cloudflint

  • *
  • Posts: 20
  • Karma: +4/-0
    • View Profile
Reply #9 on: April 25, 2009, 06:03:26 PM
Hey yeah I will definetly do that, the project as I mentioned is a university project so I have a deadline of may 22nd at which point I will be demonstrating the system at the universities open day. So once I have passed the deadline I shall release it to the world!

Im not sure how much use it will be however since it does require a bit more than what your average user may have to hand but if someone got two wiimotes and two infrared arrays then they could fire up my programs and go.

I am going to get the plans for the infra red arrays from the technicians and including them in my dissertation. Ill have to double check but hopefully I will be able to upload them to the internet as well. When I was looking at building an array I couldnt find any sort of technical specs so hopefully I will be able to make them available so outher developers wanting to try similar stuff can be saved the hassle I went through experimenting with power and the amount of leds and stuff.

Im hopeing to develop this system further on my own time once the project is finally finished. The project is looking at practical aplications of finger tracking technology but I feel like I have barely scratched the surface with regards to interface designes, gestures e.t.c
 
« Last Edit: April 25, 2009, 06:05:25 PM by cloudflint »



Offline eyucel

  • *
  • Posts: 39
  • Karma: +1/-0
    • View Profile
Reply #10 on: April 27, 2009, 08:27:59 AM
Congrats. I cannot see the pictures and videos. Do you have any video about you idea on youtube. Take care



Offline cloudflint

  • *
  • Posts: 20
  • Karma: +4/-0
    • View Profile
Reply #11 on: April 27, 2009, 10:21:49 AM
Hey, thats odd that you cant see them. You can find them all here: http://www.youtube.com/user/cloudflint



Offline eyucel

  • *
  • Posts: 39
  • Karma: +1/-0
    • View Profile
Reply #12 on: April 27, 2009, 10:44:02 AM
Thank you. When the youtube is restricted, I cannot see the movies. Problem is solved. Take care and good luck on your project.



Offline SusanStevens

  • *
  • Posts: 9
  • Karma: +0/-0
    • View Profile
Reply #13 on: June 09, 2009, 12:10:36 PM
Cloudflint,
Update us! How did it go? Do you have video of your final demo? I've been following the progress of your project and I think it seems promising. Are you going to upload your project so others may experiment?

-Susan


Offline cloudflint

  • *
  • Posts: 20
  • Karma: +4/-0
    • View Profile
Reply #14 on: June 09, 2009, 04:31:22 PM
Hey,

Its all been a bit hectic what with finishing university and moving houses and stuff but im settled in and have been working on cleaning up my code and making one video which covers the whole project. I have also written a couple of user manuals for the different bits of software as well.

Hopefully i should be uploading the source within the next day or so so people can experiment with it.

I shall also post the video once I have finished editing it.

Thanks for your interest in the project, im very excited by this, today i hooked up my lego mindstorm to my pc and have been experimenting with controling various robot designs with finger tracking! The scope for development with this technology is pretty much unlimited.