Wiimote Project

General Wiimote and Technology => Project Ideas => Topic started by: GoldenEagle on July 26, 2009, 02:16:10 PM

Title: REAL 1:1 Wiimote Lightgun Tracking
Post by: GoldenEagle on July 26, 2009, 02:16:10 PM
As I'm sure most of you are aware by now, the Wiimote is not capable of 1:1 light gun style tracking of the screen (ie, you can't point it at a specific location on the screen and calculate where exactly it is pointed and show a dot there, for instance). At least, not with the standard hardware setup.

This of course makes "light gun" games (Link's Crossbow Training, HOTD, Umbrella Chronicles, The Conduit, etc) rather un-lightgun like. You can't, for example, use the Nyko Perfect Shot (it's hard to be very accurate with a Wiimote by itself), point it at the screen, and have the cursor track to the physical screen location you're pointing at. It is only capable of tracking relative to the sensor bar, which, though it can be mitigated slightly by providing lots of configuration settings (eg, The Conduit), is still very sub-optimal.

I've figured out how to do 1:1 mapping using two sensor bars (one above and one below the TV) and the Wiimote's ability to track 4 IR points to use them to define a 3D plane, and then map the aiming point to the physical location on the screen. I can fairly accurately hit a 1-2 inch target on a 42 inch LCD from about 10 feet using a slightly modified Nyko Perfect Shot (a piece of paper has been added to the front to provide a front sight).

There are still several problems to be mitigated, but before I spend more time trying to work them out, I would like to be sure I'm not duplicating someone else's efforts. I have Googled around quite a bit, and have not found anything like this (the closest being someone with an HL2 mod designed to use vertical-standing IR bars in combination with a projector pointed at a wall; unfortunately I haven't been able to find it again so I can't provide a link). What I'm working on is of more practical application for the average user, since all you need is a single extra sensor bar (the Nyko Wireless bars work great) to be placed opposite of your regular sensor bar.

I would like to know if anyone is aware of an existing project that does this? If you know of something, please let me know!

Now, in the event that this is actually original work, I'll explain what I'm doing in a little more detail, and list some of the problems. Please let me know if this something that interests you, so I can get an idea of how useful something like this would be to the community.

Q: Why doesn't the Wii already provide 1:1 screen tracking?
A: Because the included sensor bar only provides two IR tracking points. This is sufficient to provide distance, rotation, and relative position. But you need 3 points to define a 3D plane. With only 2 points, it's impossible to tell whether the sensor bar is becoming smaller because you are moving away from it, or because you are stepping sideways and looking at it from an angle (due to perspective foreshortening, the distance between the two points becomes smaller, but without a third reference, it's not possible to tell if this is because you are moving away or to the side).
Q: What about the Wii MotionPlus?
A: We'll probably know by next week (after we've had time to play with Wii Sports Resort... why did they decide to release that on a Sunday?). Right now I think it's safe to say it'll make good sword-fighting games possible, but whether it's accurate enough to map to a screen position? I can't say I'm holding my breath (it's designed to improve angular rotation response, not provide physical positional mapping for objects beyond your body). That said, it may be possible to improve the accuracy of this light-gun technique by utilizing the MotionPlus (see Problems to be Mitigated).

It would be totally awesome if future Wii games could support this sort of light-gun mapping by offering a configuration option to "Enable Light-Gun Tracking with 2nd Sensor Bar". I don't really expect this to actually happen any time soon, due to the Wii only coming with one sensor bar, but one can dream, can't he? Till then, we could use it on the PC to make "real" light-gun games, or mod existing games to provide light-gun style controls.

(How this project works)
As mentioned above, you cannot determine 3D plane from 2 points. But you can do it with three points. Or four points (which is actually easier for our purposes, but also introduces some problems). Since the Wiimote is capable of tracking up to four IR points, this works out great for us. All we need is a second sensor bar, and we can define a 3D plane!

So, by putting one sensor bar above the TV and another below, we can define the plane that the screen exists on. The resulting 4 IR points create a quad that can be then be mapped to screen space (I'm using Johnny Lee Chung's Warper.cs class from his Wiimote Whiteboard demo for this at the moment; I don't know enough fancy math to do it myself). Of course, you can't map the four points to the four corners of the screen (which is the way the Wiimote Whiteboard works). So you have to perform a second calibration mapping step, whereby you indicate four points on the corners of your screen which are then mapped relative to the 4 IR points, and finally turned into screen coordinates. (Sorry if I'm not being real clear here.)

So, long story short, you put a sensor bar above and below your TV, shoot a point in each corner, and you have 1:1 light-gun tracking with your Wiimote! (Sadly, of course, it's not QUITE that simple... continue reading.)

(Problems to be mitigated)
Problem #1: The Wiimote's camera has a rather limited field of view. What if one or more of the IR points moves out of view?
Solution: This is the reason I mentioned that I'm standing 10 feet from the TV.
- If one point is lost, I think it should be reasonably possible to estimate the location of the missing fourth point using the remaining three (remember, we only need a minimum of three points to define a 3D plane).
- If two points are lost, it should still be possible to estimate the location of the remaining two based on their last known location, although with reduced accuracy. This will occur quite often, when pointing toward the top or sides of the screen while standing too close.
- If three points are lost, it may still be possible to estimate the other three based on their last known locations, but accuracy will start to suffer horribly. This will occur if pointing toward the corners of the screen while standing too close. Fortunately, you will likely point toward the corners the least often (mostly it'll be towards the top or sides), and will most likely not remain pointing toward the corners for extended periods of time while moving around the room.
- If all four points move out of view, there is not much we can do. It's probably safe to say you're not pointing the Wiimote at the TV; unless of course you're like 1 foot away or something (it might be possible to use the accelerometer and/or Wii MotionPlus to determine if this is the case).

Problem #2: Currently the calibration only holds up well if you remain standing in the same position as when you calibrated it. It does work from alternate angles, but accuracy begins to suffer (targeting was off by about 2 inches in the corner of the screen when I calibrated from the front, and then tried shooting from about 45 degrees to the side).
Solution: I believe this is because with four points, it's possible to define a creased plane instead of a perfectly flat plane, and thus you have to position your sensor bars very carefully. I believe this could be mitigated by averaging the normals of both triangles which make up the quad, and using that to define a perfect 3D plane, but I don't know how to do that. You also have to try and position them directly opposite each other vertically, or you get a skewed quad, which throws things off (I think it should be possible to adjust for this as well, but again, I don't know how).

Problem #3: Cursor wiggle.
Solution: The Wiimote's camera is not very high resolution, which, combined with the shakiness of your hand, results in a lot of cursor wiggling (which gets worse the further away you are from the IR points). This can be reduced with a smoothing routine (which I've already implemented), but it may be possible to use the accelerometer and/or MotionPlus to provide improved micro-accuracy. As I mentioned, I can fairly reliably hit a 2 inch target on a 42 inch screen from 10 feet as-is; if micro-accuracy could be improved and wiggle reduced this could probably be increased to 1/2 inch or less.

Problem #4: Lag.
Solution: As anyone who's played Link's Crossbow Training is probably aware, there is a slight, but noticeable delay when you swing the cursor from one side of the screen to the other, requiring you to wait a moment for the cursor to "catch up" before pulling the trigger in order to hit what you're aiming for. This seems to be inherent in the system (the Wii's IR sensitivity calibration screen exhibits the same lag), so it may not be possible to do anything about it. On the other hand, if the delay is caused by the amount of time required by the Wiimote to process the IR image (and not the Bluetooth connection or something), and other sensors (I'm thinking MotionPlus here) send data sooner, it may be possible to use that data to estimate the location of the cursor slightly sooner than the actual IR information is available, thereby reducing perceived lag.

And I'm sure there are more problems that I haven't thought of yet... but I think that covers the major ones.

I look forward to your feedback!
Title: MotionPlus Augmentation
Post by: GoldenEagle on July 28, 2009, 08:24:22 AM
Well, that answers that. I picked up a copy of Sports Report, and while the MotionPlus is cool and should open up a range of additional motion-sensing options (arm tracking comes to mind), it's not likely to help much with this.

Firstly, it definitely does not provide any real significant sort of directional aiming, at least of the sort needed for a real lightgun (which has to be based on the physical location of the screen). There is a good possibility that it could be used to provide some additional accuracy as I had suggested, but that leads to...

The second problem: the MotionPlus is designed to actually clip into the two slots on the bottom of the Wiimote, and therefore has two hooks sticking out that go into the slots. Without either A) Clipping off the hooks or B) Drilling holes into the bottom of the Nyko Perfect Shot, there isn't any way to attach it. While this isn't a huge problem for me personally (or, I suspect, many of you), the ultimate idea is to have a library that could be used by a more "average" user to play a light gun game on the computer. I guess the library could be designed with MotionPlus support, and just detect if one is plugged in or not and utilize it if so (assuming of course that it's even worthwhile to do so; it may turn out that the MotionPlus isn't accurate enough to provide any benefit). Something I'll have to think about down the road.
Title: Re: REAL 1:1 Wiimote Lightgun Tracking
Post by: Intermezzo on June 07, 2010, 01:33:14 PM
Well this is a little bit old but excactly the kind of script im searching for.
This kind of script would be great in kombination with motion+.
Is there any code you can give us so i can go on with this project?
Title: Re: REAL 1:1 Wiimote Lightgun Tracking
Post by: lukeheineman99 on November 08, 2010, 12:10:33 PM
I see this is old, but it sounds like you and I are chasing the same dragon here.

I too wish to utilize the wiimote for a "light gun" style of game. I'm currently using wiiFlash and have been reasonably successful with tracking/aiming. I have also run into some of the same snags

What I'm doing is tracking just one of the points. I decided to start here and eliminate all the extraneous data that might interfere with calculations. Long story short, by tracking one point, moving to a spot in front of the TV (42 inch as well) where I can aim at the bottom right hand corner of the TV and the top left corner without losing the ir point, I can "calibrate" my aim. Basically I shoot those corners to create a rectangle. I then can use a ratio to translate the IR data to the coordinates.

1. distance from TV is about 8 - 10 feet as well.
2. rolling the wiimote ever so slightly tremendously effects aim. I think I can counter this, but don't have time to experiment.

So after playing around with the above, I decided to create an IR point on my gun and "flip" the wiimote/sensor bar setup around. Now I have a wiimote underneath the TV and an IR point on the gun in addition to a second wiimote. Using the above method of shooting the TV corners, I can calibrate the aim and do not need to be so far from the television. As a bonus, this solves the "rolling" problem because the wiimote camera doesn't care if the IR point rotates.

Drawback to this method:

1. Leaning forward or backward tremendously effects the aim. Even an extremely SLIGHT movement throws it off enough to prevent an accurate aim.

wii flash keeps track of the size of the IR point, but I don't think it's accurate enough to account for and solve this issue.

I'm considering using method 2 with a second sensor bar and attempting to calculate the distance from the TV using those two points, but then that reintroduces the problem of losing points, etc.

I'd be willing to collaborate
Title: Re: REAL 1:1 Wiimote Lightgun Tracking
Post by: lukeheineman99 on November 08, 2010, 12:14:31 PM
Just a thought: what if three IR points in a triangle formation were placed around the "barrel" of the gun? This would create your 3D plane and would lend itself to solving the distance problem. You'd also be able to find the center of the barrel this way to keep track of where it's pointing.
Title: Re: REAL 1:1 Wiimote Lightgun Tracking
Post by: fernandohildebrand on February 04, 2011, 02:52:12 PM
Err... Actually, the so called "sensor bar", wich is really a sender of IR, the Wiimote that is the receiving end.
But you're not quuuuite right. There are not only 2 points, actually there are 10! Five on one side, five on the other,
and those 5 leds (at each side) are not pointed all to front, only the 2 first leds(counting from the center of the sensor) are straight, the other 3 have different inclinations, they point outwards the sensor. I think the fact that they used 10 and pointed them outwards is to make a more ample field with stronger IR sign.
If you want to get that much accuracy, just put 5 more leds on the bottom of your TV, But I dont know if that will make difference to the regular wii driver.. Think you'll have to code. :)
Title: Re: REAL 1:1 Wiimote Lightgun Tracking
Post by: wiimote hacker on July 14, 2012, 06:30:25 PM
Just a thought: what if three IR points in a triangle formation were placed around the "barrel" of the gun? This would create your 3D plane and would lend itself to solving the distance problem. You'd also be able to find the center of the barrel this way to keep track of where it's pointing.
good concept but that would make it so the cursor didn't even move
the wiimote uses the movment of the IR LED's to know where it is so if we could put an immobile LED in and tell it to reference that one with the other 3 on the wiimote then that MIGHT work but the problem would be it could get confused and then that would also create tracking problems plus the problem of not seeing the immoblie LED and therefor losing tracking