Hi,
Most probably the Wiimote is detecting random IR sources.
If you are facing the Wiimote outdoors with a significant amount of sun light, this may cause the erratic results like you have.
Some lightings that emit light within the infrared electromagnetic range may also cause the interference.
You can take a look at Smoothboard's Calibration Screen which will let you determine the existence and position of the stray IR dots.

If you did not activate the IR pen's IR light, there should not be any IR dots present. You may need to drop the sensitivity of the software which is found in Settings->Wiimote->Sensitivity.
It would be best if you avoid facing the Wiimote to sun light or other stray IR source.
If the above still does not solve the issue, you may be having a faulty or non-genuine Wiimote which gives unpredictable output.
Hope this helps.
Regards,
Boon Jin