Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Topics - GoldenEagle

Pages: 1
Project Ideas / REAL 1:1 Wiimote Lightgun Tracking
« on: July 26, 2009, 02:16:10 PM »
As I'm sure most of you are aware by now, the Wiimote is not capable of 1:1 light gun style tracking of the screen (ie, you can't point it at a specific location on the screen and calculate where exactly it is pointed and show a dot there, for instance). At least, not with the standard hardware setup.

This of course makes "light gun" games (Link's Crossbow Training, HOTD, Umbrella Chronicles, The Conduit, etc) rather un-lightgun like. You can't, for example, use the Nyko Perfect Shot (it's hard to be very accurate with a Wiimote by itself), point it at the screen, and have the cursor track to the physical screen location you're pointing at. It is only capable of tracking relative to the sensor bar, which, though it can be mitigated slightly by providing lots of configuration settings (eg, The Conduit), is still very sub-optimal.

I've figured out how to do 1:1 mapping using two sensor bars (one above and one below the TV) and the Wiimote's ability to track 4 IR points to use them to define a 3D plane, and then map the aiming point to the physical location on the screen. I can fairly accurately hit a 1-2 inch target on a 42 inch LCD from about 10 feet using a slightly modified Nyko Perfect Shot (a piece of paper has been added to the front to provide a front sight).

There are still several problems to be mitigated, but before I spend more time trying to work them out, I would like to be sure I'm not duplicating someone else's efforts. I have Googled around quite a bit, and have not found anything like this (the closest being someone with an HL2 mod designed to use vertical-standing IR bars in combination with a projector pointed at a wall; unfortunately I haven't been able to find it again so I can't provide a link). What I'm working on is of more practical application for the average user, since all you need is a single extra sensor bar (the Nyko Wireless bars work great) to be placed opposite of your regular sensor bar.

I would like to know if anyone is aware of an existing project that does this? If you know of something, please let me know!

Now, in the event that this is actually original work, I'll explain what I'm doing in a little more detail, and list some of the problems. Please let me know if this something that interests you, so I can get an idea of how useful something like this would be to the community.

Q: Why doesn't the Wii already provide 1:1 screen tracking?
A: Because the included sensor bar only provides two IR tracking points. This is sufficient to provide distance, rotation, and relative position. But you need 3 points to define a 3D plane. With only 2 points, it's impossible to tell whether the sensor bar is becoming smaller because you are moving away from it, or because you are stepping sideways and looking at it from an angle (due to perspective foreshortening, the distance between the two points becomes smaller, but without a third reference, it's not possible to tell if this is because you are moving away or to the side).
Q: What about the Wii MotionPlus?
A: We'll probably know by next week (after we've had time to play with Wii Sports Resort... why did they decide to release that on a Sunday?). Right now I think it's safe to say it'll make good sword-fighting games possible, but whether it's accurate enough to map to a screen position? I can't say I'm holding my breath (it's designed to improve angular rotation response, not provide physical positional mapping for objects beyond your body). That said, it may be possible to improve the accuracy of this light-gun technique by utilizing the MotionPlus (see Problems to be Mitigated).

It would be totally awesome if future Wii games could support this sort of light-gun mapping by offering a configuration option to "Enable Light-Gun Tracking with 2nd Sensor Bar". I don't really expect this to actually happen any time soon, due to the Wii only coming with one sensor bar, but one can dream, can't he? Till then, we could use it on the PC to make "real" light-gun games, or mod existing games to provide light-gun style controls.

(How this project works)
As mentioned above, you cannot determine 3D plane from 2 points. But you can do it with three points. Or four points (which is actually easier for our purposes, but also introduces some problems). Since the Wiimote is capable of tracking up to four IR points, this works out great for us. All we need is a second sensor bar, and we can define a 3D plane!

So, by putting one sensor bar above the TV and another below, we can define the plane that the screen exists on. The resulting 4 IR points create a quad that can be then be mapped to screen space (I'm using Johnny Lee Chung's Warper.cs class from his Wiimote Whiteboard demo for this at the moment; I don't know enough fancy math to do it myself). Of course, you can't map the four points to the four corners of the screen (which is the way the Wiimote Whiteboard works). So you have to perform a second calibration mapping step, whereby you indicate four points on the corners of your screen which are then mapped relative to the 4 IR points, and finally turned into screen coordinates. (Sorry if I'm not being real clear here.)

So, long story short, you put a sensor bar above and below your TV, shoot a point in each corner, and you have 1:1 light-gun tracking with your Wiimote! (Sadly, of course, it's not QUITE that simple... continue reading.)

(Problems to be mitigated)
Problem #1: The Wiimote's camera has a rather limited field of view. What if one or more of the IR points moves out of view?
Solution: This is the reason I mentioned that I'm standing 10 feet from the TV.
- If one point is lost, I think it should be reasonably possible to estimate the location of the missing fourth point using the remaining three (remember, we only need a minimum of three points to define a 3D plane).
- If two points are lost, it should still be possible to estimate the location of the remaining two based on their last known location, although with reduced accuracy. This will occur quite often, when pointing toward the top or sides of the screen while standing too close.
- If three points are lost, it may still be possible to estimate the other three based on their last known locations, but accuracy will start to suffer horribly. This will occur if pointing toward the corners of the screen while standing too close. Fortunately, you will likely point toward the corners the least often (mostly it'll be towards the top or sides), and will most likely not remain pointing toward the corners for extended periods of time while moving around the room.
- If all four points move out of view, there is not much we can do. It's probably safe to say you're not pointing the Wiimote at the TV; unless of course you're like 1 foot away or something (it might be possible to use the accelerometer and/or Wii MotionPlus to determine if this is the case).

Problem #2: Currently the calibration only holds up well if you remain standing in the same position as when you calibrated it. It does work from alternate angles, but accuracy begins to suffer (targeting was off by about 2 inches in the corner of the screen when I calibrated from the front, and then tried shooting from about 45 degrees to the side).
Solution: I believe this is because with four points, it's possible to define a creased plane instead of a perfectly flat plane, and thus you have to position your sensor bars very carefully. I believe this could be mitigated by averaging the normals of both triangles which make up the quad, and using that to define a perfect 3D plane, but I don't know how to do that. You also have to try and position them directly opposite each other vertically, or you get a skewed quad, which throws things off (I think it should be possible to adjust for this as well, but again, I don't know how).

Problem #3: Cursor wiggle.
Solution: The Wiimote's camera is not very high resolution, which, combined with the shakiness of your hand, results in a lot of cursor wiggling (which gets worse the further away you are from the IR points). This can be reduced with a smoothing routine (which I've already implemented), but it may be possible to use the accelerometer and/or MotionPlus to provide improved micro-accuracy. As I mentioned, I can fairly reliably hit a 2 inch target on a 42 inch screen from 10 feet as-is; if micro-accuracy could be improved and wiggle reduced this could probably be increased to 1/2 inch or less.

Problem #4: Lag.
Solution: As anyone who's played Link's Crossbow Training is probably aware, there is a slight, but noticeable delay when you swing the cursor from one side of the screen to the other, requiring you to wait a moment for the cursor to "catch up" before pulling the trigger in order to hit what you're aiming for. This seems to be inherent in the system (the Wii's IR sensitivity calibration screen exhibits the same lag), so it may not be possible to do anything about it. On the other hand, if the delay is caused by the amount of time required by the Wiimote to process the IR image (and not the Bluetooth connection or something), and other sensors (I'm thinking MotionPlus here) send data sooner, it may be possible to use that data to estimate the location of the cursor slightly sooner than the actual IR information is available, thereby reducing perceived lag.

And I'm sure there are more problems that I haven't thought of yet... but I think that covers the major ones.

I look forward to your feedback!

Pages: 1