adding AES encryption to TightVNC

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

adding AES encryption to TightVNC

Ed Pizzi
Hey everyone,

I mentioned that I was working on adding encryption to VNC on this
list over a year ago, but I lost interest when I realized that I'd
done all my work on the old codebase, and merging to the new one would
basically be starting over. However I've become interested in this
again, and I have a more or less working draft of a change to add AES
based encryption to TightVNC.

I have x0vncserver, vncviewer (unix), and the Java viewer working
using encryption currently. I haven't built Xvnc yet, and I'm
struggling a bit to understand the build system. (If we update
our Xvnc code to use a newer codebase, the integration may be
easier, since now uses autoconf as well.)

For the unix programs, I'm using openssl (detected at configure-time;
openssl-disabled builds work fine, with no AES support). For Java, I'm
using the builtin Java 1.4 crypto API when it's detected (through
reflection), otherwise we continue without AES support.

I have no plans to update the windows viewer, however it should be
possible to use openssl from windows as well. This will require
changes to the windows build system to link in openssl, minor code
changes, and possibly changes in how we distribute the windows client.

Here's a recap of the issues with this change, many of which were
discussed previously on this list:

 - Exporting encryption code internationally could conceivably be an
issue. However from the US, this is a non-issue for open-source
software. I'll need to send an email to a US government department
with the URL of the project, right after we release it. I've contacted
them to double-check that this is all that is required. I'm not aware
of any other legal problems.

 - This adds a dependency to OpenSSL libraries for the moment. This is
good and bad. The good part is that users will get updates to the
cryptography code through routine security updates, without having to
update VNC specifically. The bad part is that there isn't a version of
this code included in the VNC source, so we need to do some
configuration to detect it, and some #ifdefs based on whether it is
found. My reading of the OpenSSL license says that we could probably
embed it in our code, if we distribute their license, and maintain the
code we embed.

 - The OpenSSL license requires us to add their brag string to any
"about" dialog describing features. I've added this to the unix client
so far. This is only added when we configure with OpenSSL.

Here are some details about the approach I took:

A new security type is added, using AES encryption (128-bit CFB8).
There's a challenge / response authentication similar to the VncAuth
authentication mode. In the AES mode, since all transmissions are
encrypted, we add new input / output streams that encrypt / decrypt as
data passes through them. Verifying encryption is just done by writing
a challenge (plain) to the client, and checking if you get the
challenge back through the AES-decrypting input stream.

For the unix server, I've added an AESPassword, a password encoding on
disk that isn't trivially decodable to a plaintext password. The
format is an 8-byte random salt, followed by a 20-byte SHA1 hash of
the salt + password string. To authenticate, the client is passed the
salt, and a random challenge. The client recomputes the hash from the
salt and the user's password, and uses this as an encryption key.  The
client then sends an encrypted copy of the challenge as the challenge

The idea here is that if someone compromises a host, they get access
to VNC, but they don't know your plaintext password, which may prevent
them from accessing other hosts that you use the same password for (or
other things like bank accounts). The salt here is useful in a number
of ways, all of which are basically the reason we use salts on modern
unix password files. You can't tell if two users' passwords are the
same, nor can you easily build up a table of pre-computed hashes of
common passwords. This also lets us support >8 character passwords.

This change introduces a nontrivial delta relative to the upstream
RealVNC base, however I think it's a natural step for the TightVNC
project to take. This changes many parts of the codebase, so I'm not
sure where to start the review process. A good place to start to look
at this might be the Java viewer source, since it's fairly easy to
understand. I've attached a subversion diff of the java client for the

I don't have access to a browser with Java1.1. If you do, email me and
I'll give you a link to a java vnc client to test, to make sure the
new java viewer still works.

I'm using authentication protocol number 9 for the time being,
although I don't know if this is in use by some other VNC version. We
can change this pretty easily.

So open questions at this point are:
 - do we want this going into tightvnc in the first place
 - openssl: dynamically linked or included in our source
 - protocol version number
 - whether to update the X.Org 6.8.2-based Xvnc, or to hold off until
we're using
   a newer, autoconf-based X.Org release

Sorry for the long email, let me know what you think.


SF.Net email is sponsored by:
Check out the new Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
VNC-Tight-devel mailing list
[hidden email]

java-diff.txt (24K) Download Attachment