Device misdetected as serial mouse

Question

I'm working on a device which communicates with a PC through a (virtual) serial port. The problem is that the data we are sending occasionally gets incorrectly identified by Windows as a bus mouse, after which the "Microsoft Serial Ballpoint" driver is loaded and the mouse pointer starts jumping around on the screen and randomly clicking on things.

A bit of Googling reveals that is an old and well-known problem with serial devices where the usual work-around is a bit of registry hacking to disable the offending driver. That it is a lot to demand from our users however and I'd rather not have our application messing around with the user's registry. Especially not when the fix is dependent on the Windows version and the user may well be using a bus mouse.

Instead I'd like to avoid the problem by changing our protocol to not send any data which may get us misidentified as a mouse. The only problem is that I'm not quite certain what patterns to avoid. Apparently Microsoft's Mouse protocol consists of packets of four bytes where the MSB of the first is set and that of the last three is clear.

Would sending only 7-bit ASCII suffice? Are there any other devices I need to worry about being detected as?

Rosenberg

I've found in the past that if a device presents itself to Windows as a COM port and then starts transmitting automatically as soon as it is connected to the system that it gets seen as a mouse. This is regardless of the data it sends, and it certainly didn't match your 4 bytes. Can you wait a short time before your device begins transmitting? – Tailback

tinman: Thanks for the hint! I'll try having the PC application poll for updates instead, and wait for a second or two before making the first request. – Rosenberg

I think we managed to create the biggest serial mouse on the Earth... It is a level crossing fit in a pair of about 1 cubic metre racks, and it can move it's entire software in the recycle bin in a flinch of a second! – Gregorygregrory

Answer 1

I just encountered this problem myself on Windows 7 Professional x64, and a solution that worked for me was to go into the registry and edit the following value:

Location: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\sermouse  
Key: Start  
Value: 3

Change Value to 4 and it will stop this problem occurring.

Here is a list of all valid Start values:

0 Boot (loaded by kernel loader). Components of the driver stack for the boot (startup) volume must be loaded by the kernel loader.

1 System (loaded by I/O subsystem). Specifies that the driver is loaded at kernel initialization.

2 Automatic (loaded by Service Control Manager). Specifies that the service is loaded or started automatically.

3 Manual. Specifies that the service does not start until the user starts it manually, such as by using Device Manager.

4 Disabled. Specifies that the service should not be started.

---

A reg edit command would be as follows:

REG ADD "HKLM\SYSTEM\CurrentControlSet\Services\sermouse" /V Start /T REG_DWORD /F /D 4

You then need to restart the computer, which should now start correctly and not attempt to discover a serial mouse.

good luck.

Damaris

Perfect! I needed to restart the OS too. – Prosaic

Forgot to mention that, I'll add it to the answer, Thanks. – Damaris

It drove me mad for 2 Hours! – Irrepealable

Is there any kind of documentation on this registry entry? Right now, 3 and 4 are totally magic numbers to me. Microsoft has documentation on the sermouse node, but not on the actual Start element itself. – Singlehanded

@GünthertheBeautiful Here you go! I've added it to the answer for you. – Damaris

You are my hero! I was ready to throw away my computer I was so frustrated! – Semipostal

Answer 2

It turns out that mouse detection in Windows is normally handled by the serenum.sys filter driver. This driver implements support for legacy serial mice along with serial plug-and-play. Microsoft has even provided the sourcecode as a WDK sample.

During detection the ports switches to 1200-7-N-1 mode while asserting DTR+RTS to which a response is expected within 200 ms, with a couple of retries in case of failure. Unfortunately for a legacy mouse a single M or B character suffices as identification.

In our case the protocol was reworked to avoid these characters and now appears not to be misidentified anymore.

However we were using a virtual USB serial port and for a traditional serial port this approach may be somewhat difficult as anything sent at a different baud rate is liable to look like line noise. In this case I suppose the easiest workaround is probably, as has already been suggested, to avoid making any unsolicited transmissions.

Alternatively with the serial control signals actually hooked up, or intercepted by a USB CDC device, processing the DTR or RTS signals and holding off on output. Actually implementing the plug-and-play protocol would be an even niftier option. Supposedly there are cheap RS232 cables around without a full complement of control signals though so this approach might still fail.

Rosenberg

This comment is so useful to me! Now I understand why my Arduino-clones reboot a couple of time before powering up (Because the CH340's DTR pin through a 100NF cap is connected to the reset pin.) Thanks for the great explanation! – Ministry

Answer 3

I also encountered this problem, fixed it by disabling "serial enumerator" in the advanced properties of the FTDI driver (properties of COM ports in Device Manager). This is described in http://www.ftdichip.com/Support/Documents/AppNotes/AN_107_AdvancedDriverOptions_AN_000073.pdf.

Aretta

This is the best solution in my opinion. On my radio link device I could see the host TX/RX light flashing all the time and that was obviously the serial enumerator scanning it. Once disabled and connecting the device (no reboot necessary which is another plus) there was no activity until a program which uses the device is started. Also I had trouble in the past connecting to the device, hit-and-miss, which has gone so must have been the serial enumerator jumping in-between the application's own attempt to lock/open the serial port. Thanks! – Giffard

Answer 4

I have encountered this Windows bug myself. Here is my own research on the topic:

Microsoft acknowledges this bug: http://support.microsoft.com/kb/819036 Start with downloading their tool and see if it solves the issue.

• Download & install their program.
• Run it from the command prompt from C:\program\Microsoft comdisable\
• Write comdisable /list when executing the program.
• All ports on the computer will be shown.
• Write comdisable /disable COMx where x is the port number.
• Do this for all ports on the computer.
• Reboot.

This should hopefully work as an universal solution.

Alternatively, you can hack in boot.ini but I don't believe this works in Vista/Win 7. I have some app note from Cisco systems describing how to do this. If the above doesn't solve your problem, please let me know.

Sobriety

Thanks for the suggestion. I'd prefer to avoid any user intervention by changing the protocol on our device so as not to confuse Windows, but if that doesn't work out then at least I have an official workaround (and "bug" acknowledgment) to refer to. I've implemented tinman's suggestion along with repackaging our data as 7-bit ASCII now and have yet to see the problem again, though to be honest it never did appear all that frequently. – Rosenberg

@Rosenberg I'm not sure what kind of obscure protocol this "ballpoint" junk uses, but changing to 7-bit ASCII will might not solve anything. Because Windows assumes that this ancient Microsoft mouse has a certain baudrate. I have encountered the problem when using a faster baudrate than 9600, the data can suddenly get interpreted as something coming from the mouse. When that happens... disaster. The mouse will move all over the screen at the speed of light, clicking everywhere! Anything can happen. I wouldn't risk that, I would use the tool Microsoft recommends in that link. – Sobriety

Thankfully we're using a FTDI chip with a virtual serial port driver instead of a real serial port. So there's no risk of the data being mangled through an invalid baud rate setting. – Rosenberg

@Jon: on Windows 8.1 it helped to open devmgmt.msc, plug in the device, select the "Microsoft Serial Ballpoint" device, right click, and click "Uninstall" to get rid of the driver and the problem – Rashid

Answer 5

In my development environment, I've simply disabled Microsoft Serial Mouse from the Device Manager.

This seems to solve the culprit of my problem. Before doing so, the CH340G chip I've used in my design used to lower the DTR five times before initiating the connection, effectively rebooting my Arduino-based board and render it useless.

Ministry

That may work for you personally, but it does not address the problem of the original question, which requires a solution that works for end users and not only developers willing to modify the system configuration. – Class

@ChrisStratton True, and I may not have posted this as an answer but I merely wanted to provide a quick solution for devs like me Googling this issue. I used a 500ms delay after serial connection to prevent Windows detecting my device as a mouse, instead. – Ministry

Worked for me just great! My FTDI adapter was being seen as a Microsoft Ball Pointer. Thx – Yonder

@ChrisStratton This is not a developer fix, this is a fix for anyone. Users should have a basic understanding of what Device Management is. – Yonder

Answer 6

Maybe this helps: We had the same problem with FTDI FT232RL.We found out, that it was a hardware issue of our PCB.

FTDI-Datasheet says about #RESET-Pin: Active low reset pin. This can be used by an external device to reset the FT232R. If not required can be left unconnected, or pulled up to VCC.

RESET-Pin was not required in our application, so we connected it to Vcc via 1k Pull-Up. It seemed that the pull-up of #RESET-Pin caused an undefined start-up of the FT232RL, at least every second converter, we connected to a USB-socket caused a serial-ball-point in the devive manager. We removed the pull-up-resistor at #RESET-Pin, therewith the #RESET-Pin is unconnected. Since then every interface worked proberly and didn't any longer create serial-ball-points in the Windows device manager.

Johannajohannah

Answer 7

If you have a "true" serial port, or an USB dongle (RS-232, RS-485, it does not matter) this problem can be worked around by first opening the serial port in question with a terminal, or whatever application you want to monitor it with, and only then plugging the device in. For your own sake, you should also pay attention to remove the device before terminating the connection.

With FTDI chips soldered on the device itself, you are busted. It took a few rounds for me to explain the management that a device communicating on it's own paired with an FTDI chip soldered on the PCB meeting Windows computers won't likely pass for user-friendliness, no matter how slick an USB socket may look like on the cabinet... (Thankfully, all these conditions coming together are quite rare and unusual)

Gregorygregrory

@Jubation I'm interested to know, what do you personally think is the better alternative to Serial-to-USB chips (instead of FTDI, CH340G, ATmega32u4, etc...) for a more user-friendly interface? – Ministry

@DRS David Soft If your design can include a proper USB interface using some appropriate device class for your device, then most likely that for end-user things (of course this also demands creating drivers). If it is not, then I think it is better to stick with an ordinary DSUB-9 connector (RS-232), that's more robust if you later wanted to use the device combined with some other microcontroller-driven device for a more autonomous system. Use dongles (or RS-232 interface card) to connect with a PC. – Gregorygregrory

Answer 8

I had this problem since 2010 with serial scale heads connected to the pc. Usb to serial converter or not.. I use onkly SILABS device's CP2102 or the like.. I worked around it by simply allowing the driver to install and then in device manager look for the ballpoint driver under mouse/HIDA and then simply DISABLE the driver, DO NOT UNINSTALL IT simply disable it. Then when you reboot even with the driver instaled it seems windows ignores the comport as serial mouse and uses the data from the input. You will also find that if the ballpoint driver is active then that COMport is in use and sometimes returns a COM PORT not accessible... hope this helps some one out there :) Tx Ben

Yolandoyolane

Answer 9

Turns out there is a setting to stop windows trying to enmumerate devices that connect as a COM port.

Make sure that "Serial Enumerator" is unchecked under "Miscellaneous Options"

Ebbarta

Answer 10

Code tot stop GPS from being detected as serial mouse.

Below is the code for a subroutine in C#.net. It checks if the registry key is set to 4 and if not it issues the configuration command to disable sermouse. Embed this subroutine in a program which runs at startup and it will correct the setting after a windows update.

Maybe useful if you get annoyed when this problem happens time and again

private void Stop_sermouse()

    {
        string k =
        "HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\sermouse";

        object v = Microsoft.Win32.Registry.GetValue(k, "Start", null);
        if (v==null)
            {
            MessageBox.Show("No Registry Key for sermouse");
                }
        else
        {
            string sr = v.ToString();
            if (sr == "4")
            {; }
            else
            {
                DialogResult mbox = MessageBox.Show("disable sermouse ? " + v.ToString(), "Found sermouse enabled! ", MessageBoxButtons.YesNo);
                if (mbox == DialogResult.Yes)
                { 
                    // prepare a small job to issue confuguration command
        ProcessStartInfo s = new ProcessStartInfo("cmd.exe", "/c sc config sermouse start=disabled");
                    Process p = new Process();
                    s.Verb = "runas"; // Must run as administrator
                    s.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
                    p.StartInfo = s;
        // and run the command
                    p.Start();
        //check if the registry is modified indeed
                    v = Microsoft.Win32.Registry.GetValue(k, "Start", null);
                    sr = v.ToString();
                    if (sr == "4")
                    {
                        MessageBox.Show("finished ''sc config sermouse start=disabled'' but not succesfull in registry!");
                    }
                    else
                    {
                        MessageBox.Show("sermouse is disabled");
                    }
                }
            }
        }           
    }

Memo