This is VisualNaCro.

DISCLAIMER: recorder.pl is not yet functional.

What does it?

	It is a Perl module meant to remote control a VNC server.
	
	It includes a recorder (written in Perl) to make it easy to
	record a macro, which is just a Perl script, and which you can
	modify to your heart's content.
	
	The most important feature, however, is that you can mark a
	rectangle which the Perl script will try to find again when you
	run it. Thus when you play a game and want to hit a certain button,
	you just hit the Ctrl key twice, mark the button, and from then on,
	all mouse movements will be repeated relative to that button, even
	if the button is somewhere else when you run the script the next
	time.

	If you know Tcl Expect, you will recognize this approach. Only this
	time, it is not text, but an image which is expected.
	
How does it work?

	It acts as a VNC proxy: your Perl script starts its own VNC server.
	The script now can intercept inputs and outputs, and act upon them.
	In order to write a macro, start
	
		recorder.pl --script my-macro.pl --timing host:port
	
	connect with a vncviewer of your choice to <host2>:23, where <host2>
	is the computer	on which recorder.pl was started (not necessarily the
	same as the VNC server!). Now your actions are recorded into
	my_macro.pl, and the images you want to grep for will be saved as
	my_macro-1.pnm,	my_macro-2.pnm, ...

	In order to finish the script, hit Ctrl twice and say "q".

Why did I do it?

	Because I could ;-)

	No really, I needed a way to write automated tests. While there
	exist a lot of OpenSource programs for web testing, I found none
	of them easy to use, and for GUI testing I found xautomation.

	Xautomation has this "visual grep" (or "graphical expect") feature:
	given an image it tries to find it on the desktop and returns the
	coordinates. Unfortunately, there is no easy way to record macros
	with it, and it only works on X11.

	As I know VNC pretty well, and there are VNC servers for every OS
	and gadget, I thought it might be cool to have this feature to
	control a VNC server.

	Actually, it makes it even easier: with plain X11, for example, you
	can not know where on the screen the action is if you don't check
	the whole screen. This complex problem is beautifully addressed
	in Karl Runge's x11vnc.

	My main purpose is to run regression tests on different browsers,
	which I can easily do by starting Xvnc and using VisualNaCro.

How did I do it?

	I wondered long about how to do it. I couldn't take the same approach
	as xautomation: I cannot connect to the VNC server thousand times
	per second. So I decided to create an interface of LibVNCServer/
	LibVNCClient for use in a script language.

	Fortunately, this task is made very, very easy by SWIG. As Perl
	is one of my favorite script languages, I decided to use this.
	But SWIG makes it easy to use the very same interface for other
	popular languages, so you are welcome to port VisualNaCro to
	the language of your choice!
	
Isn't it pronounced "Visual Macro"?

	Yes. But I liked the Visual Na Cro play of acronyms. I'm sorry if
	you don't find it funny.

What's the license?

	GPL. It is based on LibVNCServer/LibVNCClient, so it has to be.
	If you want to port this package to use vncreflector, which has a
	BSD license, go ahead.