GUI on headless Linux server [SOLVED]

I merely wanted to run NrvrCommander to do load testing with Selenium – a combo working quite well – but circumstances forced me to first regain control of a remote computer I had been driving just fine until one morning VNC stopped working. Presented is a documentation of working solutions, and how we got here.

The failure of VNC started with no clearly identifiable single cause. There have been multiple causes:

• Possibly one trigger of events has been a network reconfiguration at the site hosting that remote computer. Cannot be certain.

• Another trigger may have been a reboot behind that site’s KVM switch switched to a different computer and our computer not receiving EDID information – apparently a weakness of that specific KVM switch.

• Another problem has been a known issue with the operating system’s built-in vino-server storing the password differently in some versions under some circumstances, which wasn’t obvious at all, specifically if not seeing the remote screen.

This post may be useful for various configurations, but my adventure has been on: A server running Scientific Linux 6.4 (similar to RHEL 6.4 and CentOS 6.4) on a 2008 board with Intel CPUs and with ATI ES1000 graphics. We had replaced its hard disk RAID with a modest SSD. It is connected to a KVM switch in a server room. Its Ethernet is connected to a LAN into which we can VPN if not on site.

Once upon a time it worked

I was VNC connected from remote over VPN to port 5900 serving display :0.0 via that Linux’s built-in vino-server and happily working.

At some point VNC went dark and didn’t come back.

SSH and restart

Luckily I was able to ssh. Restarts were possible and I did plenty with

sudo shutdown -r now

SYN_RECV and switching off IPv6

With

netstat -anp | grep 5900

I noticed incoming VNC connections were stuck at SYN_RECV. Also I noticed IPv6 even though all IP numbers we specified were IPv4.

Apparently at some time there wasn’t any port 5900 listening on IPv4, i.e. no

tcp   0   0 0.0.0.0:5900   0.0.0.0:*   LISTEN

but there was only port 5900 listening on IPv6

tcp   0   0 :::5900   :::*   LISTEN

By itself use of IPv6 could be a sign of the times, but after some online searches I decided to force IPv4 only. Per one recommendation I added in

/boot/grub/grub.conf

the kernel parameter

ipv6.disable=1

That solved one problem, after a restart, connections made it through to ESTABLISHED.

No EDID behind KVM switch – Kernel Parameter

VNC clients also said they were connected, but none showed any screen.

On X Window Systems if you are in via ssh on a command line, if you want to access X commands you often need to first

export DISPLAY=:0.0

Being a greatly simplified version of a more general concept, that is what is needed many times.

With

xrandr

and

xset q

I found out the graphics system thought no monitor was connected. That was because of rebooting behind that specific kind of KVM switch when the KVM switch was switched to another machine, or to none, not getting EDID information.

Among many possibilities and after experiments, I ended up adding in

/boot/grub/grub.conf

the kernel parameter

drm_kms_helper.edid_firmware=edid/1280x1024.bin

which essentially at a kernel level tells the graphics system to proceed as if a 1280×1024 monitor were connected.

With a screen assumed, still no luck. VNC clients said they were connected, but none showed any screen.

gconftool-2

I checked some more with

gconftool-2 -R /desktop/gnome/remote_access

and maybe made some adjustments with gconftool-2. Obviously for example you’d need

prompt_enabled = false

Else you’d be stuck “If true, remote users accessing the desktop are not allowed access until the user on the host machine approves the connection.”

Still no luck. VNC clients said they were connected, but none showed any screen.

Monitoring vino-server

Similar to a forum post, but notably differently running as regular user, not as root, for a while I manually started an instance of vino-server, logged it and tailed that log. Plenty of

[IPv4] Got connection from client localhost
  other clients:
Client Protocol Version 3.7

but nothing after that. VNC clients said they were connected, but none showed any screen.

Screenshots

If VNC was not showing any screen, if I could see the screen by other means, I might be able to see where it is stuck. Screenshots to the rescue! Install a command line capable screenshot tool with

sudo yum install ImageMagick

and screenshot with

import -window root screenshot1.jpg

then SCP that image file to a regular computer and look at it.

Monitor is Off – DPMS and such

Alas, screenshots I got were mostly black, with the exception of a couple icons in the upper right corner indicating this wasn’t completely off track, it was from the screen, in the operating system’s GUI, almost working.

Another check with

xset q

told me either exactly or something similar to

  DPMS is Enabled
  Monitor is Off

I made an

/etc/X11/xorg.conf.d/10-monitor.conf

and in there I have

Section "ServerLayout"
  Identifier "ServerLayout0"
  Option "BlankTime" "0"
  Option "StandbyTime" "0"
  Option "SuspendTime" "0"
  Option "OffTime" "0"
EndSection

Then, after another restart, I was able to see a normal desktop when taking screenshots.

Also there was no luck putting an xorg.conf line

Option "DPMS" "false"

anywhere, no matter which section, so now l am again seeing

  DPMS is Enabled
  Monitor is Off

but screenshots are still working.

Unfortunate choice of password storage

Via a screenshot I found a dialog box saying

Enter password for default keyring to unlock
The application ‘GNOME Remote Desktop’ (/usr/libexec/vino-server) wants access to the default keyring, but it is locked
Password:
OK

The problem is, that dialog box is on a screen I cannot get to via VNC because that very dialog box is blocking the vino-server VNC server on the server from proceeding. I only was able to extract a screenshot to find out where it is stuck at, but I cannot type into it. This appears to be a known issue of some versions of vino-server.

Installing a second VNC server

To get work done I finally opted for installing a separate VNC server and connecting to a virtual seat 1, instead of real seat 0.

Install

sudo yum install vnc-server

Configure this VNC server

sudo nano /etc/sysconfig/vncservers

to contain (with your user name)

VNCSERVERS="1:yourusername"
VNCSERVERARGS[1]="-geometry 1920x1080"

While the configured user, enter

vncpasswd

Make effective

sudo service vncserver restart
sudo chkconfig vncserver on

In Enterprise Linux, configure firewall

sudo nano /etc/sysconfig/iptables

to contain equivalent to SSH on port 22 or equivalent to VNC seat zero on port 5900, now for port 5901

-A INPUT -m state --state NEW -m tcp -p tcp --dport 5901 -j ACCEPT

Make effective

sudo service iptables restart

Or Ubuntu I think that would be

sudo ufw allow 5901/tcp

Now I can work on that machine again.

Why

I wanted to have a computer to run tests on. While remote, I wanted to see its GUI. And, it was behind a KVM switch. I didn’t want a person required on site, so we can run tests other places too, 24/7.

Always give EDID information

Some of the problems described here could be avoided by making sure the video connector on the computer gets EDID information at all times. Either through a smarter KVM switch or by means of a specialized small device.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: