I manage a few remote servers that are running Devuan. Recently (last
week) I noticed a couple updates to some x11 libraries. I went ahead
and updated. But now, I'm wondering if it's okay to remove them
altogether? I do not use any graphical user interface on them. I only
connect via ssh, and issue text-only commands.

I'm a little concerned about just removing these libraries though. The
one in particular that I remembered updating was libx11-6. Apparently
there is a bit of dependencies on it, and I don't want to break anything.

On beowulf:
# apt remove libx11-6
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer
required:
fontconfig-config fonts-dejavu-core libdrm-amdgpu1 libdrm-common
libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libdrm2
libfontconfig1 libfontenc1 libgl1-mesa-dri libglapi-mesa libglvnd0
libice6 libllvm7 libpciaccess0 libsm6 libx11-data libx11-xcb1
libxau6 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0
libxcb-shape0 libxcb-sync1 libxcb1 libxdmcp6 libxshmfence1
x11-common
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
libgl1 libglew2.1 libglu1-mesa libglx-mesa0 libglx0 libx11-6 libxaw7
libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3
libxft2 libxi6 libxinerama1 libxmu6 libxmuu1 libxpm4 libxrandr2
libxrender1 libxt6 libxtst6 libxv1 libxxf86dga1 libxxf86vm1
mesa-utils x11-utils x11-xserver-utils xauth
0 upgraded, 0 newly installed, 29 to remove and 0 not upgraded.
After this operation, 8,081 kB disk space will be freed.
Do you want to continue? [Y/n] n

I don't think I should need any font or drm stuff.
On ascii, I get a much longer list:

# apt remove libx11-6
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer
required:
dconf-gsettings-backend dconf-service fontconfig fontconfig-config
fonts-dejavu-core gconf-service gconf2-common glib-networking
glib-networking-common glib-networking-services
gsettings-desktop-schemas hicolor-icon-theme libatk1.0-0 libatk1.0-data
libavahi-client3 libavahi-common-data libavahi-common3 libblas-common
libblas3 libck-connector0 libcolord2 libcroco3 libcups2
libdatrie1 libdconf1 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2
libdrm-radeon1 libdrm2 libepoxy0 libfontconfig1 libfontenc1
libgbm1 libgck-1-0 libgconf-2-4 libgcr-3-common libgcr-base-3-1
libgdk-pixbuf2.0-common libgfortran3 libgl1-mesa-dri libglapi-mesa
libgnome-keyring-common libgnome-keyring0 libgraphite2-3
libgtk-3-common libgtk2.0-common libgtop-2.0-10 libgtop2-common
libharfbuzz0b libice6 libjbig0 libjpeg62-turbo libjson-glib-1.0-0
libjson-glib-1.0-common liblapack3 liblcms2-2 libllvm3.9
libpam-gnome-keyring libpango-1.0-0 libpangoft2-1.0-0 libpciaccess0
libpixman-1-0 libproxy1v5 libquadmath0 librest-0.7-0
libsecret-1-0 libsecret-common libsm6 libsoup-gnome2.4-1 libsoup2.4-1
libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0
libwayland-client0 libwayland-cursor0 libwayland-server0 libwebp6
libx11-data libx11-xcb1 libxau6 libxcb-dri2-0 libxcb-dri3-0
libxcb-glx0 libxcb-present0 libxcb-render0 libxcb-shape0 libxcb-shm0
libxcb-sync1 libxcb-util0 libxcb-xfixes0 libxcb1 libxdmcp6
libxkbcommon0 libxshmfence1 p11-kit p11-kit-modules python-numpy
x11-common xkb-data
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
elogind libelogind0 libpam-elogind pinentry-curses python-urwid
wicd-curses
Suggested packages:
pinentry-doc
Recommended packages:
policykit-1
The following packages will be REMOVED:
adwaita-icon-theme analog at-spi2-core consolekit dbus-x11 gconf2 gcr
gksu gnome-keyring gtk-update-icon-cache libatk-bridge2.0-0
libatspi2.0-0 libcairo-gobject2 libcairo2 libegl1-mesa libgail-common
libgail18 libgcr-ui-3-1 libgd3 libgdk-pixbuf2.0-0 libgksu2-0
libgl1-mesa-glx libglade2-0 libglew2.0 libglu1-mesa libgtk-3-0
libgtk-3-bin libgtk2.0-0 libgtk2.0-bin libnotify4
libpam-ck-connector libpangocairo-1.0-0
libpolkit-backend-consolekit-1-0 libpolkit-gobject-1-0
libpolkit-gobject-consolekit-1-0
librsvg2-2 librsvg2-common libstartup-notification0
libwayland-egl1-mesa libx11-6 libxaw7 libxcomposite1 libxcursor1 libxdamage1
libxext6 libxfixes3 libxft2 libxi6 libxinerama1 libxmu6 libxmuu1
libxpm4 libxrandr2 libxrender1 libxt6 libxtst6 libxv1 libxxf86dga1
libxxf86vm1 mesa-utils notification-daemon pinentry-gnome3
pinentry-gtk2 python-cairo python-glade2 python-gtk2 python-notify
wicd-gtk x11-utils x11-xserver-utils xauth
The following NEW packages will be installed:
elogind libelogind0 libpam-elogind pinentry-curses python-urwid
wicd-curses
0 upgraded, 6 newly installed, 71 to remove and 0 not upgraded.
Need to get 1,632 kB of archives.
After this operation, 68.8 MB disk space will be freed.
Do you want to continue? [Y/n] n


If it were something local where I could plug in a keyboard and monitor
to log in if something went wrong, I would just go ahead and try it
without too much thought. But since I have no physical access, if
something like ssh (or its authentication) fails for some reason, it'd
take a lot more time to restore things.

Is there any harm or benefit to leaving these things installed? Is this
a case of, "if it's not broke, don't fix it"? Thank you for any response.