Fred:
...
> The Sun Compact 1 three button mouse is 1200 baud, 8 data bits, no
> parity and sends 5 bytes in Mouse Systems protocol. Byte 0 is button
> info. exactly the same as msc. Byte 1 is 8 bit signed X movement. Byte
> 2 is 8 bit signed Y movement. Bytes 3, 4 are zero.

That fits the Mousesystems protocol as described in man mouse.

...
> The problem seems to be that movement info. is not being sent or is
> incorrect when the middle button is down. The inputattach program
> appears to be decoding the mouse protocol so I think that is where the
> problem is. I am not a C programmer so I can't determine what is wrong.
...

If I look at utils/inputattach.c from
https://sourceforge.net/projects/linuxconsole/files/latest/download

inputattach does
. option handling
. sets the line discipline:
ldisc = N_MOUSE;
if (ioctl(fd, TIOCSETD, &ldisc) < 0) {
. sets the device type (e.g. it's a MSC mouse)
ioctl(fd, SPIOCSTYPE, &devt)
. goes into the background (daemon())
. stays there just emptying the input buffer of the serial port

So, inputattach does not decode the byte stream from the mouse.
Instead it seems that the kernels drivers/input/mouse/sermouse.c
does that. Look in the
static void sermouse_process_msc(struct sermouse *sermouse, signed char data)
function.

So... the "input" way is too buggy. Why not consider the deprecated X11
mouse driver.

///

You should be able to do lsinput from input-utils package to see which
/dev/input/inputX file your mouse's byte stream transformed as events
would appear. And then run input-events X to see the events. E.g.:

# lsinput | tail
open /dev/input/event11: No such device or address

/dev/input/event10
   bustype : BUS_I8042
   vendor  : 0x2
   product : 0xa
   version : 0
   name    : "TPPS/2 IBM TrackPoint"
   phys    : "isa0060/serio1/input0"
   bits ev : (null) (null) (null)

# input-events -t 100 10
/dev/input/event10
   bustype : BUS_I8042
   vendor  : 0x2
   product : 0xa
   version : 0
   name    : "TPPS/2 IBM TrackPoint"
   phys    : "isa0060/serio1/input0"
   bits ev : (null) (null) (null)

waiting for events
11:58:57.217543: (null) ??? (0x110) pressed
11:58:57.217543: (null) code=0 value=0
11:58:57.356137: (null) ??? (0x110) released
11:58:57.356137: (null) code=0 value=0
11:58:59.084678: (null) ??? (0x112) pressed
11:58:59.084678: (null) code=0 value=0
11:58:59.225056: (null) ??? (0x112) released
11:58:59.225056: (null) code=0 value=0
11:58:59.613730: (null) ??? (0x111) pressed
11:58:59.613730: (null) code=0 value=0
11:58:59.689984: (null) ??? (0x111) released
11:58:59.689984: (null) code=0 value=0
^C
#

Where I pressed the left, middle and lastly right button on my laptop.

Regards,
/Karl Hammar