Wrapping Manure

Our company has a common time registration system. Every month we are mandated to fill out our time sheets, using its Netscape-era UI.

It’s named Kaba and it looks roughly like this:

Filling out your time sheet is not the most popular of chores, especially given the fact that you would just mindlessly enter information already available elsewhere, such as your primary project code and any vacation days. This information is already maintained using its own proprietary (and rather nice) application named the Absence Manager.

If you prefer an architectural drawing with impeccably aligned boxes, here you go:

The time registration system and the absence manager system

A tale of two web apps

I guess that I should mention that at the corporate level, tax deductions are involved. In other words it’s not a process which is likely to go away anytime soon.

It’s not the biggest of deals but still … could there be an easier way?

Trying Selenium

Our first attempt at automating the process involved Selenium, a technology that we had just started to explore at that point. After some work we got it to run. After half a year, for some reason, bugs started creeping in, some of which were really odd. We tried to debug them but we hit rock bottom; it seemed like a bug in the web driver or maybe even in Chrome. The HTML and Javascript that it was dealing with was pretty weird stuff so maybe we stumbled upon some corner case. After a year or so we decided to abandon the Selenium strategy. This decision was also guided by the experience that we had gathered using Selenium in other projects. To us, at least, it seems surprisingly brittle. Perhaps because every browser update or web driver update was a potential source of jitter.

Either way, Selenium was off the table.

Wrapping the UI

During the time period where the Selenium based solution was slowly derailing itself, a new idea had formed. If we could only avoid the browser and instead communicate directly using HTTP, maybe there would be fewer moving parts? It would require us to reverse engineer how to get information in and out (i.e. how the UI would communicate with the server). But Chrome makes it easy to snoop on web communication. Also, there were really just three requests that we needed to reverse engineer:

  • Logging in to the time registration web UI with a given user
  • Getting the timesheet data for a given week for the logged in user
  • Posting timesheet data for a given week for the logged in user

In terms of architecture, I decided to create a simple, nodeJS based server which would be able to communicate with the time registration system and which would in turn expose its own REST API. I picked nodeJS to get my feet wet with the technology and to see if it would be an easy and painless way of creating a server (it most certainly was!).

The basic idea was to make the nodeJS server an adapter around the time registration system as well as around the absence manager. For instance, it would expose a REST method to obtain the user list (from the absence manager) and another REST method to obtain the time sheet data for a given week (data from the time registration system). The REST API should be convenient and should not convey the origin of the data.

After a bit of thinking I decided to also make it the responsibility of the nodeJS server to transparently log in. The login mechanism of the time registration system is a pretty standard deal in which it sets a session cookie on the response and use that for subsequent request validation. To store sessions I used a plain javascript object mapping from user name to cookie. As far as the time registration system was concerned, it could only assume that a bunch of people decided to log in simultaneously, forming a bizzare kind of time registration flash mob.

Implementing the method for getting time sheet data involved HTML scraping (no AJAX used here). As it turned out, the HTML returned from the time sheet application included a large inlined data structure (a variable declaration within a script tag) and a bunch of Javascript to render it to the DOM. Thus, it was refreshingly easy to get my hands on the data and to transform it to something useful.

Posting data, on the other hand, was a pain, and I almost gave up on reverse engineering it since nothing seemed to work. As a last resort I challenged a very smart colleague of mine to crack it, offering a desirably bounty composed of beer and bragging rights. Within days he had not only managed to reverse engineer the protocol, he had also written the first take on an AngularJS based UI on top of the NodeJS server. These were valuable contributuions.

All of this brought us pretty much to the place where we wanted to be. A place where we didn’t have to interact with the hideous UX and UI presented by the time sheet application. Where we could write a modern UI to present the data and where we could write business rules on the server to make our lives easier. It looked like this:

A new shiny UI

A new shiny UI

The remaining part consisted of writing the business logic for autofilling timesheets based upon the data that the system could pull from various places (users, project codes, vacation days, and public holidays). Also, I took the opportunity to play around with with fun stuff like AngularJS animations, CSS, and the like, although none of this was really required.

The performance also deserves a mention. Once a user has logged in — which is sluggish in itself — the filling out of a timesheet manually for a week takes at least 30 seconds. As a side note, the time sheet data has historically been wildly inaccurate since a lot of people didn’t really know what to fill in or didn’t bother to check the list of holidays.
The Selenium implementation took around 90 seconds to log in a user and fill out a week; this added up to about an hour for our department as the users were filled out in sequence.
The NodeJS based implementation, running users in parallel and steering clear of the sluggish UI, filled out the weekly timesheet for all users for in about one minute!

Conclusions

  • Small side projects are a great way to explore new technologies. It’s much like training, only without pricey consultants and oxygene depleted conference rooms. Also, it’s highly motivating.
  • NodeJS is both fun and productive for creating all kinds of stuff, including web servers and REST APIs. I don’t miss the verbose Java way where you would define value classes at both the back and the front of your system and creating yet other classes with long sequences of dest.setFoo(src.getFoo()) type statements. In fact I seem to be developing an allergy against just that kind of thing these days.
  • Selenium is ingenious but it’s also brittle. So use it as a last resort, whether you are writing automation scripts or automatic tests.
  • If you don’t like the UI of a system (or a public web site for that matter), and if you are able to work out how it communicates with its backend, then you can create a better UI. It’s not hard at all and you don’t have to ask permission from anyone.

When Flash was cool

I created a Flash game once.

spritsprint1

A flash game

It’s more than a decade ago now. It was back in the times when when Flash was cool and the original IT bubble had just blown to smithereens. I was fresh out of university and had little in terms if financial obligations so when I was offered a freelance contract to create a Flash game, I happily accepted.

The game was directed at a Danish audience and point of the game was to campaign against drunk driving and to promote the idea that young females, known to have been endowed with far more common sense than young males, should be assigned “guardian angels” for the dimwits in the latter group.

I was given a fair amount of creative freedom to design both the gameplay and the humour. In reponse to this freedom I pretty much based the gameplay upon an ancient C64 racing game, the name of which I had already forgotten at the time. I based the humour upon puerile drawings and animations which I drew up fairly quickly using a tablet.

spritsprint3

At the auto dealer you could upgrade your car. I tried to pick six makes and models which I considered dubious taste. Interestingly, I’m currently driving a Skoda Felicia in real life.

The game code was implemented in Flash Actionscript. As I recall, the language was more or less an extension to Javascript which is an interesting choice since this is before anyone envisioned Javascript ever joining rank with the cool kids. The editor was kind of shoddy and it was apparent that Flash was mostly designed as a tool to create linear animations; scripting seems to have been added as kind of an afterthought. I can’t really say that I enjoyed the language. Still, I managed to finish the game in about one month of effective work (in reality more like two months with integrated procrastination).

The race courses were all autogenerated with a random seed based upon the name of the place where the course was supposed to be located (I made a conscious effort to pick the silliest sounding place names that I could find). By autogenerating the course but fixing the seed, I was freed from manually creating a large number of courses, yet the course would stay the same at all times. When playing the game it’s apparent that the quality of the courses vary quite a bit.

spritsprint2

The protagonist goes to jail. Admittedly a puerile drawing.

The pay? A handful of glass beads as far as I recall. And a shot a doing something fun, maybe as a career. As it turned out this was pretty much all the Flash I ever did and a few months later I got a regular job as a Java developer. I have no regrets about this. Working with Java has been more fun that working with Actionscript would ever have been; and working on teams has been far more fun than working by myself ever was.

You can test drive the game here (providing that you have Flash installed).

[Quick instructions: At the login screen, enter anything in the bottommost input field. When driving, steer with the cursor keys and use space to brake. Make a full stop close to the guardian angels to pick them up. Try to stay clear of the drinks and by all means, don't hit the girls with your car].

Fun Times with Fiddler

I hacked my colleague’s multiplayer rocket game. I mainly did it for kicks. But also to make an important point about the server never trusting a client.

My colleague is developing a small game in his spare time. It’s based upon the Amiga classic Roketz and uses Google playn to cross compile from Java to various native platforms, such as web browsers, Android, and iOS.

I won’t disclose the URL of the beta since it’s not yet launched but here is a sneak preview of the browser version:

Being a multiplayer game, there is usually more than one rocket. When hit by enemy fire, your rocket will take damage and eventually explode. You use thrust and rotation to stay clear of walls. There are landing pods where you can recharge. This pretty much sums up the gameplay of this charming, retro style game.

Having followed the development on the sideline I had gathered a few pieces of information that made made me conclude that hacking the game would be feasible. I knew that all clients communicated with the server using JSON and web sockets. Also I knew that, by design, each client is responsible for keeping track of the state (such as position and velocity) of its own rocket and all shots fired from this rocket as well as for communicating this to the server. The server, in turn, would pass information about rocket and shot state of all other players. Clients would do hit detection themselves and when a rocket took sufficient damage its client would tell the server about this. I argued that there would be plenty of ways to hack this. My colleague stated that this was largely a hypothetical problem. And so, the challenge was on.

Snooping upon the network traffic it turned out that the browser version was serving gwt compiled javascript, which is not all that surprising from a Google framework which offers Java to javascript compilation. It was bit dense due to the formatting but it was unminified and it wasn’t difficult to feel the Java below the surface.

rocketcode

The linefeeds are escaped but it’s readable. This particular function looks like it might be called to update the position of the rocket.

Next step was figuring out how to replace this benign javascript with evil javascript that I could modify to do my bidding. I unsuccessfully spent some time trying to find a tool for the Mac. I then switched to Windows, downloaded Fiddler and immediately made progress. After googling a bit I found out how to make it work as a proxy server .. and how to make this proxy server replace selected files by acting as a “man in the middle”. 

Fiddler comes with a very powerful scripting language. Using this, I configured Fiddler so that if I hit the rocket game URL at post 8888, it would serve the files as if I had hit port 80 — except for a few select files which I made Fiddler grab from my own domain (I’m sure that I could have stored them locally as well, I just found the other solution first).

As “candidates for evil” I picked two javascript files which are pretty much standard gwt stuff. The *.nocache.js file exists only to detect the browser type and load the correct flavour of the second one. The second file, the one with the autogenerated and highly volatile name, contains all the interesting stuff, i.e. the code that was compiled from Java. The only reason that I’m also shadowing the first file it to make the hack a little bit more robust since the name of the second file is referenced within the first one so they kind of go together.

rocketredirecting

Shadowing with my copies of the javascript files. I’ve beeped out the domains from the screendump. Hope you don’t mind.

Next, I set up Fiddler as reverse proxy as described in the first section of this instructionhttp://fiddler2.com/documentation/Configure-Fiddler/Tasks/UseFiddlerAsReverseProxy

It worked! If I went to the normal rocket game URL but I specified port 8888, I got a fully functional game, but Fiddler substituted my copies of the files (as was easily verified by a console.log() statement):

letthehacking

Ignition!

To be able to really work with the javascript, I had to deal with the escaped linefeeds which were caused by gwt having encoded most of the javascript as an escaped string. I copied the text into Notepad++ and replaced all the \n linefeeds with proper linefeeds. Before uploading the edited file I would do the inverse substitution. This simple approach turned out to work fine.

Here is the malicious piece of code that I added:

hackcode2

The piece of javacode that I added to the update loop. Also a few lines here and there to pass the required arguments. Not the prettiest Javascript in the world but it does its job.

Behold the final result. Distributed from my client to your client through a trustful protocol .. kiss my heat seeking missiles!

Duplicating rolls in Trivial Pursuit

Trivial Pursuit is a game of knowledge but it’s also about dice and probability theory.

Usually it’s obvious how to move. Fields which may award wedges are most desirable. Second are “free roll” fields. Least attractive are the ordinary question fields. However, question fields also come in two kinds.

The following image depicts the possible rolls and moves when placed on the rim, between two “free roll” fields.

trivial3

Note how every dice roll lands nicely on either a spoke or a free roll. This is a good position to roll from.

Adjacent to a spoke

trivial4

Overall, this is a worse position than the one above. In particular, note how fives and twos just lead to another question square.

When does this matter?

It matters in the cases where you cannot avoid moving to a question square, but you can choose the type. One example is the roll of a two in the figure above. Another is the roll of a one or a six when on a “free roll” square:

trivial2

All in all, if you need to pick a question square, pick one in the middle between the spokes if you can. If you don’t go by this principle, you’ll end up answering more questions to make the same progress. Maybe not in the specific game but in the long run.

I’m pretty sure that you brainy people knew this all along. You just chose not to tell me.

Stuff that I learned

  • Trivial Pursuit is (also) a game about dice probabilities
  • Brainy people are good at keeping secrets, apparently