Posts Tagged With: Today’s World

More Comment-Preserving Configuration Parsers

For the past few weeks I've been on the hunt for a configuration file format with the following three properties:

  1. You can use a library to parse the configuration. Most configuration formats allow for this, though some (nginx, haproxy, vim) aren't so easy.

  2. You can manipulate the keys and values, using the same library.

  3. When that library writes the file to disk, any comments that were present in the original config file are preserved.

Why bother? First, allowing programs to read/write configuration files allows for automated cleanup/manipulation. Go ships with a first-class parser/AST, and as a consequence there are many programs that can lint/edit/vet your source code. These wouldn't be possible without that ast package and a set of related tools that make parsing and manipulating the source easy.

You can imagine installers that could automatically make a change to your configuration; for example, certbot from the Let's Encrypt project tries to automatically edit your Apache or Nginx configuration. This is an incredibly difficult task, due to the complexity of the configuration that have piled up over the years, and that those configuration files weren't built with automatic editing in mind.

Backwards incompatible changes are never good, but their downsides can be mitigated by effective tools for parsing and updating configuration.

You want comments in your configuration file because configurations tend to accumulate over the years and it can be incredibly unclear where values came from, or why values were set the way they were. At Twilio, the same HAProxy config got copied from service to service to service, even though the defined timeouts led to bad behavior. Comments allow you to provide more information about why a value is set the way it is, and note values where you weren't sure what they should be, but had to pick something besides "infinity" before deploying.

What problems do you run into when you try to implement a comment-preserving configuration parser? A lot of config parsers try to turn the file into a simple data type like a dictionary or an array, which immediately loses a lot of the fidelity that was present in the original file. The second problem there is that dictionaries in most languages do not preserve ordering so you might write out the configuration in a different order than you read it, which messes up git diffs, and the comment order.

You are going to need to implement something that is a lot closer to an abstract syntax tree than a dictionary; at the very least maps of keys and values should be stored as an array of tuples and not a dictionary type.

The next problem you run into is that syntax trees are great for preserving the fidelity of source code but tend to be unwieldy when all you want to do is index into an array, or get the value for a key, especially when the type of that value may take any number of values - a number, a string, a date, or an array of the above. The good news is configuration files tend to only need a subset of the syntax/fidelity necessary for a programming language (you don't need/want functions, for example) so you can hopefully get away with defining a simpler set of interfaces for manipulating data.

(Incidentally I realized in the course of researching this that I have written two libraries to do this - one is a library for manipulating your /etc/hosts file, and the other is a library for bumping versions in Go source code. Of course those are simpler problems than the one I am trying to solve here).

So let's look at what's out there.

  • JSON is very popular, but it's a non-starter because there's no support for comments, and JSON does not define an ordering for keys and values in a dictionary; they could get written in a different order than they are read. JSON5 is a variant of JSON that allows for code comments. Unfortunately I haven't seen a JSON5 parser that maintains comments in the representation.

  • YAML is another configuration format used by Ansible, Salt, Travis CI, CircleCI and others. As far as I can tell there is exactly one YAML parser that preserves comments, written in Python.

  • XML is not the most popular format for configuration, but the structure makes it pretty easy to preserve comments. For example, the Go standard library parser contains tools for reading and writing comments. XML seems to have the widest set of libraries that preserve comments - I also found libraries in Python and Java and could probably find more if I looked harder.

  • TOML is a newer format that resembles YAML but has a looser grammar. There are no known parsers for TOML that preserve comments.

  • INI files are used by windows programs, and the Python configparser module, among others. I have found one parser in Perl that tries to preserve comments.

  • Avro is another configuration tool that is gaining in popularity for things like database schema definitions. Unfortunately it's backed by JSON, so it's out for the same reasons JSON is out.

  • You can use Go source code for your configuration. Unfortunately the tools for working with Go syntax trees are still pretty forbidding, for tasks beyond extremely simple ones, especially if you want to go past the token representation of a file into actually working with e.g. a struct or an array.

I decided on [a configuration file format called hcl], from Hashicorp. It resembles nginx configuration syntax, but ships with a Go parser and printer. It's still a little rough around the edges to get values out of it, so I wrote a small library for getting and setting keys in a configuration map.

This is difficult - it's much easier to write a parser that just converts to an array or a dictionary, than one that preserves the structure of the underlying file. But I think we've only scratched the surface of the benefits, with tools like Go's auto code rewriter and npm init/npm version patch. Hopefully going forward, new configuration formats will ship with a proper parser from day one.

Liked what you read? I am available for hire.

Cleaning up Parallel Tests in Go 1.7

I have a lot of tests in Go that integrate with Postgres, and test the interactions between Go models and the database.

A lot of these tests can run in parallel. For example, any test that attempts to write a record, but fails with a constraint failure, can run in parallel with all other tests. A test that tries to read a random database ID and expects to not fetch a record can run in parallel with other tests. If you write your tests so they all use random UUID's, or all run inside of transactions, you can run them in parallel. You can use this technique to keep your test suite pretty fast, even if each individual test takes 20-40 milliseconds.

You can mark a test to run in parallel by calling t.Parallel() at the top of the test. Here's an example test from the job queue Rickover:

func TestCreateMissingFields(t *testing.T) {
  t.Parallel()
  test.SetUp(t)
  job := models.Job{
    Name: "email-signup",
  }
  _, err := jobs.Create(job)
  test.AssertError(t, err, "")
  test.AssertEquals(t, err.Error(), "Invalid delivery_strategy: \"\"")
}

This test will run in parallel with other tests marked Parallel and only with other tests marked Parallel; all other tests run sequentially.

The problem comes when you want to clear the database. If you have a t.Parallel() test clean up after it has made its assertions, it might try to clear the database while another Parallel() test is still running! That wouldn't be good at all. Presumably, the sequential tests are expecting the database to be cleared. (They could clear it at the start of the test, but this might lead to unnecessary extra DB writes; it's better for tests that alter the database to clean up after themselves).

(You can also run every test in a transaction, and roll it back at the end. Which is great, and gives you automatic isolation! But you have to pass a *sql.Tx around everywhere, and make two additional roundtrips to the database, which you probably also need to do in your application).

Go 1.7 adds the ability to nest tests. Which means we can run setup once, run every parallel test, then tear down once. Something like this (from the docs):

func TestTeardownParallel(t *testing.T) {
  // This Run will not return until the parallel tests finish.
  t.Run("group", func(t *testing.T) {
    t.Run("Test1", parallelTest1)
    t.Run("Test2", parallelTest2)
    t.Run("Test3", parallelTest3)
  })
  // <tear-down code>
}

Note you have to lowercase the function names for the parallel tests, or they'll run inside of the test block, and then again, individually. I settled on this pattern:

var parallelTests = []func(*testing.T){
  testCreate,
  testCreateEmptyPriority,
  testUniqueFailure,
  testGet,
}
func TestAll(t *testing.T) {
  test.SetUp(t)
  defer test.TearDown(t)
  t.Run("Parallel", func(t *testing.T) {
    for _, parallelTest := range parallelTests {
      test.F(t, parallelTest)
    }
  })
}

The test mentioned there is the set of test helpers from the Let's Encrypt project, plus some of my own. test.F finds the defined function name, capitalizes it, and passes the result to test.Run:

// capitalize the first letter in the string
func capitalize(s string) string {
  r, size := utf8.DecodeRuneInString(s)
  return fmt.Sprintf("%c", unicode.ToTitle(r)) + s[size:]
}
func F(t *testing.T, f func(*testing.T)) {
  longfuncname := runtime.FuncForPC(reflect.ValueOf(f).Pointer()).Name()
  funcnameparts := strings.Split(longfuncname, ".")
  funcname := funcnameparts[len(funcnameparts)-1]
  t.Run(capitalize(funcname), f)
}

The result is a set of parallel tests that run a cleanup action exactly once. The downside is the resulting tests have two levels of nesting; you have to define a second t.Run that waits for the parallel tests to complete.

=== RUN   TestAll
=== RUN   TestAll/Parallel
=== RUN   TestAll/Parallel/TestCreate
=== RUN   TestAll/Parallel/TestCreateEmptyPriority
=== RUN   TestAll/Parallel/TestUniqueFailure
=== RUN   TestAll/Parallel/TestGet
--- PASS: TestAll (0.03s)
    --- PASS: TestAll/Parallel (0.00s)
        --- PASS: TestAll/Parallel/TestCreate (0.01s)
        --- PASS: TestAll/Parallel/TestCreateEmptyPriority (0.01s)
        --- PASS: TestAll/Parallel/TestUniqueFailure (0.01s)
        --- PASS: TestAll/Parallel/TestGet (0.02s)

The other thing that might trip you up: If you add print statements to your tear down lines, they'll appear in the console output before the PASS lines. However, I verified they run after all of your parallel tests are finished running.

Liked what you read? I am available for hire.

Six Years of Hacker News Comments about Twilio

(I'm omitting the many, many, astroturf posts - "Why X is Better than Twilio", "Why I'm Ditching Twilio for X" - and comments from employees at competitors between 2010 and 2014.)

Twilio Raises $12m Series B

"Can something like Twilio really become a $100m+ company? I hope so but my ignorance blinkers me to how this could happen.."

Twilio Launches UK SMS

"I'm not sure what the big deal is. (Competitor) has much better international coverage and costs less."

Twilio Launches in Europe

"Great when it will have SMS support."

Twilio Raises $17m Series C

"When are they going to get SMS enabled numbers in Canada?"

"Their move to the UK was very half-assed, still no SMS support for the UK... :( I hope they fix that with this new funding before they expand elsewhere."

"It's a good service, but unless they bring prices way down, some big provider (Cough Amazon..) is going to come in and eat their lunch. Granted they probably wouldn't offer the level of detailed APIs that Twillio does. We started using them for SMS sending but went with (Competitor) at a fraction of the cost."

"I believe their exit strategy is to be acquired by Amazon or someone of their scale... anyone can do it once they learn how to handle the SMS messages between the web and the carrier gateways, as well as how to handle call flows with Asterisk or Freeswitch." - (ed "Anyone can do it once [the entire value proposition])

Twilio Launches MMS

"Well, all the kudos should go to Bandwidth.com the primary carrier of of Twilio, (Competitor), (free, irrelevant service), etc ..." - (ed. This is completely incorrect)

"MMS died years ago. Terrible technically and never actually worked the four times in someone tried to send one."

Twilio Launches WebRTC Support

"I came to the conclusion that on mobile phones, WebRTC video is not yet usable."

"Out of all the WebRTC products I've tried, (competitor) was the easiest one for me to use."

Twilio Launches International SMS

"Love how Hacker News will post anything Twilio like their pricing, but competitors posts are instantly removed. More proof that HN is a silicon valley whore"

Twilio Launches SIP Support

"( Competitor ) has been supporting SIP for a very long time."

Twilio Acquires Authy

"Authy is one of the worst-designed iOS applications I have ever used."

"Authy was only a marginal improvement in technology."

Twilio Files S-1

"This IPO is an exit for their VCs. They were all the way up to a series E round, and since they grew fast by losing money, the early investors had to pour in a lot of cash."

"I can only imagine that as more and more developers flock to "free" SMS verification services provided by companies like Facebook (Account Kit) and Twitter (Digits), their long term outlook is even more unsure."

"Looking at their escalating losses, I have to wonder if this IPO is a desperation play after failing to raise private money at an acceptable valuation in the current climate."

"Not even cash flow positive. Stay away."

This Post Gets Submitted to HN

"What is the point of this post?"

"It seems like a running [joke] that their primary (ed. struggling) competitor is ahead on features, and has better prices. I think those are completely fair criticisms of any company. It's a little strange that this compilation goes to great lengths to never mention the competitor by name."

Twilio Sells 10m Shares on Open Market, Stock Rises 90% on Opening Day, Hope Restored for Other Tech Company IPO's

(In this thread someone posted links to a spreadsheet with 500 other SMS API's. Oh. Shut down the company)

"Good service but still very overvalued."

"For a company with a yearly gross profit of USD92 millions and net losses of USD35 millions, I cannot imagine who is buying shares at a USD1.8 billion market cap."

"I'm not sure how defensible this is against AWS or another infrastructure provider cross-selling to existing customers."

"I used to work for a company that vaguely competed with them. It was frustrating to have people ask "why does your service cost so much compared to Twilio" and to bite my tongue and say something diplomatic rather than "because we're running an actual business that makes money rather than a VC-funded playpen". Apparently they're not even cash-flow positive at this point?"

Keep doubting! And don't forget to crap on any new thing that gets launched.

Liked what you read? I am available for hire.

The TSA Randomizer iPad App Cost $1.4 Million

You may have seen the TSA Randomizer on your last flight. A TSA agent holds an iPad. The agent taps the iPad, a large arrow points right or left, and you follow it into a given lane.

How much does the TSA pay for an app that a beginner could build in a day? It turns out the TSA paid IBM $1.4 million dollars for it.

It's not hard! I searched on Google for "TSA FOIA" and found this page, which describes exactly how to reach the FOIA team at the TSA. Ignore the part about how they will get back to you in 20 days. I sent them this email:

I request that a copy of documents concerning the following subject matter be provided to me:

The RFP (Request for Proposal) issued by the TSA for designing and implementing the "TSA Randomizer" iPad application, described here and currently in use by TSA agents at security checkpoints at many airports, including Terminal 2 at Oakland International Airport.

Details of any submitted bids from contractors or internal government agencies to design and construct the "TSA Randomizer" iPad app.

The final signed contract between the TSA and a contractor to implement the "TSA Randomizer" iPad app.

I also included a little bit about how I thought this request should be eligible for a fee waiver, and how I wasn't going to profit from knowing this information, but it didn't seem particularly difficult to get the data, so I'm not sure it mattered.

They just got back to me! They sent me two documents. The first is a disclaimer about how they had to black out some of the information. The second is the contract between the TSA and IBM. And there's the payment:

Later today Pratheek Rebala reached out to mention that this data is available publicly, and there were 8 other payments as part of the same award, totaling $1.4 million; the document I have is one part, totaling $336,000. Furthermore, there were 4 bids for the contract and IBM won the bidding.

Unfortunately we don't know everything the TSA got for that $1.4 million. They might have just gotten the iPad app; they might have gotten iPads, or work on multiple different apps, including the TSA Randomizer. We only know it's associated with the TSA Randomizer based on the FOIA request that returned this document.

TSA randomizer payment

I should mention that the Obama Administration, the "most transparent", has set numerous records for delays in turning over files and refusing to fulfill requests for access, and none of the candidates seem likely to reverse that trend. If you think this is important, consider writing your elected officials and asking them to prioritize this, or making decisions in November based on this.

Liked what you read? I am available for hire.

YOUR ASSISTANCE IS NEEDED: STOCK AWARD NOTICE (Ref: LSUK/2031/8161/05)

DEAR SIR,

I HAVE THE BELIEVE YOU ARE A REPUTABLE AND RESPONSIBLE AND TRUSTWORTHY PERSON I CAN DO BUSINESS WITH FROM THE LITTLE INFORMATION SO FAR I GATHERED ABOUT YOU DURING MY SEARCH FOR A PARTNER AND BY MATTER OF TRUST I MUST NOT HESITATE TO CONFIDE IN YOU FOR THIS SIMPLE AND SINCERE BUSINESS.

LET ME START BY INTRODUCING MYSELF PROPERLY , I AM MR. IRVING TRUBE, LOAN OFFICER WITH SILICON VALLEY BANK PLC SAN FRANCISCO BRANCH, I CAME TO KNOW OF YOU IN MY PRIVATE SEARCH FOR A RELIABLE AND REPUTABLE PERSON TO HANDLE THIS CONFIDENTIAL TRANSACTION,WHICH INVOLVES TRANSFERING HUGE SUM OF MONEY TO A FOREIGN ACCOUNT REQUIRING MAXIMUM CONFIDENCE.

THE PREPOSITION:

A FOREIGNER AND AN AMERICAN , MR KEVIN BURKE (SWE) AN ENGINEER WITH A UNICORN STARTUP IN SAN FRANCISCO, UNTIL HIS DEPARTURE WEEKS AGO DUE TO BOREDOM AND A TWENTY FIVE CENT SURCHARGE ON DRINKS FROM THE SODA MACHINE BANKED WITH US AT SILICON VALLEY BANK OF SAN FRANCISCO PLC. A YOUNG WHITE MAN OF MEDIOCRE ABILITY, ENGR BURKE HIRED DUE TO HIS APPARENT WILLINGNESS TO WORK LONG HOURS FOR BEER, JUNK FOOD AND A PINBALL TABLE, WAS IMPRUDENTLY GRANTED A LARGE NUMBER OF SHARES OF COMMON STOCK. THROUGH NO FAULT OF HIS OWN, THE END OF DECEMBER 2015 THE SHARES WERE WORTH $8,432,000USD.

THE MANAGEMENT OF UNICORN STARTUP PLC UNDER THE INFLUENCE OF THEIR CHAIRMAN AND MEMBERS OF THE BOARD OF DIRECTORS ARRANGEMENTS HAVE BEEN MADE FOR THE STOCK TO BE DECLARED "UNCLAIMED" AFTER 90 DAYS AND SUBSEQUENTLY DONATE THE FUNDS TO THE VENTURE CAPITALISTS, AND THIS WILL FUTHER ENHANCE THE CAUSE OF RICH WHITE MEN IN SILICON VALLEY AND SCHOOL TEACHER PENSION FUNDS IN GENERAL . THE COMPANY NOW EXPECTS BURKE TO EXERCISE HIS OPTIONS WITHIN 90 DAYS OR LOSE THE SHARES.

POOR ENGR BURKE EXPLAINED TO ME THAT HE HAS NO CASH ON HAND TO PURCHASE THE SHARES, ( HE PUT HIS LIFE SAVINGS IN AN ONLINE BITCOIN ACCOUNT THAT WAS FOUNDED AS A TRADING SITE FOR MAGIC THE GATHERING CARDS ).

HE SAID I SHOULD SEEK FOR A GOD FEARING PARTNER IN A BAY AREA COUNTY OF MY CHOICE WHERE I WILL TRANSFER THIS STOCK AND USE IT FOR INVESTMENT PURPOSE, (SUCH AS FURTHER INCREASING THE ALREADY RIDICULOUS PRICES OF BAY AREA REAL ESTATE). SIR, WE ARE HONOURABLY SEEKING YOUR ASSISTANCE IN THE FOLLOWING WAYS.

IN OTHER TO AVERT THIS NEGATIVE DEVELOPMENT SOME OF MY TRUSTED COLLEAGUES AND I NOW SEEK YOUR PERMISSION TO LOAN MONEY TO THE DESPONDENT MR. KEVIN BURKE SO THAT THE STOCK WILL BE RELEASED AND A CERTIFICATE WILL BE ISSUED AS THE STARTUP'S THIRD EMPLOYEE . WE HAVE BEEN MANDATED BY THE BANK TO OFFICIALY PROVIDE THE FUNDS WITHIN THE SHORTEST POSSIBLE TIME (90 DAYS), THAT IS WHY WE HAVE DECIDED TO CONTACT YOU AND MORE SO WE ARE ASSURING YOU THAT YOU THAT THE BUSINESS IS 100% RISK FREE INVOLVEMENT .

MOREOVER SIR, WE ARE WILLING TO OFFER YOU 15% OF THE SUM AS COMPENSATION FOR EFFORT INPUT AFTER THE SUCCESSFUL LOAN OF THIS MONEY TO MR. KEVIN BURKE'S ACCOUNT, UNTIL UNICORN STARTUP RIDES OUT ADVERSE MARKET CONDITIONS, UNDERGOES SEVERAL POINTLESS BRANDING EXERCISES, HIRES ITS OLD CEO BACK AND TENDERS AN INITIAL PUBLIC OFFERING. PLEASE FEEL FREE TO CONTACT ,ME VIA THIS GPG PUBLIC KEY: 8F A0 A0 2F 1A D1 C0 2F

THANK YOU IN ADVANCED FOR YOUR ANTICIPATED CO-OPERATION

BEST REGARDS

MR IRVING TRUBE

Liked what you read? I am available for hire.

Ready Player One and a Dystopian View of Future Oil Prices

I read Ready Player One recently and I enjoyed it; it was a pretty fast read and I finished it in a day. It presents a view of the future where the possibilities allowed by a virtual reality world surpass those of the real world, so most people spend as much time as possible connected to a system called OASIS.

One part of the book's prediction of the future bothered me. The author describes the price of oil skyrocketing, causing the decay of American roads, and cities with cars stacked around their exteriors, as the unaffordable price of oil made them too expensive to drive.

This bothered me and my economics degree. People hugely value the ability to move around and travel for many reasons:

  • For reasons we cannot yet explain in-classroom teaching produces better results than online teaching, meaning the teacher and students need to travel to the same place (There's a study showing this result but I don't know where it is).

  • People migrate for work, to bring their skills to an area of the world where there's more demand for them, see for example the millions of Filipinos who work overseas. People will still need food delivered and haircuts and their plumbing fixed and their cars driven even if they spend twelve hours a day with a headset strapped to their face.

  • The same is mostly true of relationships and friendship bonding; the face to face time is a costly signal that helps show the other person you care and are committed to the relationship.

  • People enjoy traveling for tourism, to see beaches, mountains, etc. Sure virtual reality could put you on a beach and it might be "good enough" but a device that can replicate the feeling of the sand beneath your toes is still a ways away.

And many, many more; sure, virtual reality may chip away at the margins here but there still will be a vast demand for people to fly around the planet.

The vast demand means that there's a huge incentive for folks to figure out cost-effective ways to get around. If gas gets too expensive, we see people figure out ways to create gas from other minerals, as the large expansion of activity in North Dakota shows. Or there will be more of an incentive to figure out how to store solar energy to use at night, or more of an incentive to use electric power to get around, and we'd see more trains and less planes.

How can we turn this into a testable prediction? One obvious answer is to look at the price of oil. The author writes that the price will be sky-high, causing people to abandon their cars. Well another group of people bet on the price of oil every day. I tend to trust them more, because if they are wrong, they will lose tons of money.

Data from the CME Group indicates that the best prediction for the price of oil in December 2022 is $86.20, lower than it is today. Now, there are some caveats that would affect this. There are reasons to believe the futures price might be lower than the true price in 2026, because a buyer today is assuming risk and the seller is shedding it. On the flip side at least one source states that oil futures with long maturities tend to be priced higher than the actual price.

So that's the best prediction of the price of oil 8 years from now - about what it is today, not drastically higher. I would encourage anyone who thinks oil will be very expensive (including the author) to bet on that outcome, and profit from all of the people who currently think it will be cheap.

In general we look to science fiction authors for a vision of what the future will look like. In some places they may be better than average at predicting what the future will look like. But clearly in others, they're not.

Liked what you read? I am available for hire.

Hacking Roller Coaster Tycoon with Genetic Algorithms

I used to play a ton of Roller Coaster Tycoon when I was a kid. I loved the game but I was never very good at making the roller coasters. They always felt too spread out, or too unnatural looking. As a ten year old I idly wondered about writing a computer program that was really good at playing the game. What sort of parks would it make? How would a computer approach the freedoms inherent in an empty park? What would we learn about the game engine from doing so?

A cool coaster

Sadly, this one wasn't generated by a computer

In case you're not familiar, Roller Coaster Tycoon is a amusement park simulation game most notable because the entire game was written in x86 assembler by Chris Sawyer.

Finally a few months ago, I had the tools and the free time available to work on this, and I made some progress toward writing a program that would generate cool looking roller coasters. Let's examine the parts of this program in turn.

Interacting with the Game

So let's say you have a program that can generate roller coasters. How do you actually put them in the game, or integrate them into your parks?

Fortunately, Roller Coaster Tycoon has a format for saving track layouts to disk. Even more amazingly, this format has been documented.

4B: departure control flags
4C number of trains
4D number of cars per train
4E: minimum wait time in seconds
4F: maximum wait time in seconds

Once you decode the ride data, it follows a format. Byte 0 stores the ride type - 00000010 is a suspended steel roller coaster, for example. Some bytes indicate the presence of flags - the 17th bit tells you whether the coaster can have a vertical loop. And so on, and so on.

To compress space, RCT used a cheap run-length encoding algorithm that would compress duplicates of the same byte. But once you encoded/decoded the file, it was super easy to get it running in the game.

So, great! I could write my coaster generator in any language I wanted, write out the file to disk, then load it from any of the parks.

Getting Track Data

There are a lot of track pieces in the game, and I needed to get a lot of data about each of them to be able to make assertions about generated roller coasters. As an example, if the track is currently banked left, which pieces are even possible to construct next?

A steep upward slope track piece increases the car height 4 units. A sharp left turn would advance the car 3 squares forward and 3 squares left, and also rotate the car's direction by 90 degrees. I had less than zero interest in doing this all by hand, and would probably make a mistake doing it. So I went looking for the source of truth in the game..

OpenRCT2

Literally the same week that I started looking at this, Ted John started decompiling the Roller Coaster Tycoon 2 source from x86 into C, and posting the results on Github. More importantly (for me), Ted and the source code actually showed how to read and decompile the source of the game.

The repository shipped with an EXE that would load the C sources before the x86 game code. From there, the C code could (and did, often) use assembler calls to jump back into the game source, for parts that hadn't been decompiled yet.

This also introduced me to the tools you use to decompile x86 into C. We used the reverse engineering tool IDA Pro to read the raw assembly, with a shared database that had information on subroutines that had been decompiled. Using IDA is probably as close as I will come to a profession in code-breaking and/or reverse engineering.

Most of the time with IDA involved reading, annotating the code, and then double checking your results against other parts of the code, the same way you might annotate a crossword puzzle. Other times I used guess and check - change a value in the code, then re-run the game and see what specifically had changed, or use debugging statements to see what went on.

So I started looking for the track data in the game. This turned out to be really, really difficult. You would have a hunch, or use the limited search capability in the game to search for something you thought should be there. Ultimately, I figured out where the strings "Too high!" and "Too low!" were being called in the engine, figuring that track height data would have been computed or used near those points.

This was only part of the solution - it turns out that track data is not stored in one big map but in several maps all around the code base. Some places store information about track bank, some store information about heights and it's tricky to compile it all together. Ultimately, I was able to figure it out by spending enough time with the code and testing different addresses to see if the values there lined up with the pre-determined track order.

Visualizing rides

With a genetic algorithm you are going to be generating a lot of roller coasters. I wanted a quick way to see whether those roller coasters were getting better or not by plotting them. So I used Go's image package to draw roller coasters. To start I didn't try for an isometric view, although that would be fun to draw. Instead I just plotted height change in one image and x/y changes in another image. Running this against existing roller coasters also revealed some flaws in my track data.

A fitness function

A good fitness function will have penalties/rewards for various pieces of behavior.

  • Is the ride complete?

  • Does the ride intersect itself at any points?

  • Does the ride respect gravity, e.g. will a car make it all the way around the track?

  • How exciting is the ride, per the in-game excitement meter?

  • How nauseating is the ride, per the in-game excitement meter?

The first two points on that list are easy; the last three are much more difficult. Finding the excitement data was very tricky. I eventually found it by getting the excitement for a "static" ride with no moving parts (the Crooked House) and searching for the actual numbers used in the game. Here's the function that computes excitement, nausea and intensity for a Crooked House ride.

sub_65C4D4 proc near
or      dword ptr [edi+1D0h], 2
or      dword ptr [edi+1D0h], 8
mov     byte ptr [edi+198h], 5
call    sub_655FD6
mov     ebx, 0D7h ; ''
mov     ecx, 3Eh ; '>'
mov     ebp, 22h ; '"'
call    sub_65E7A3
call    sub_65E7FB
mov     [edi+140h], bx
mov     [edi+142h], cx
mov     [edi+144h], bp
xor     ecx, ecx
call    sub_65E621
mov     dl, 7
shl     dl, 5
and     byte ptr [edi+114h], 1Fh
or      [edi+114h], dl
retn
sub_65C4D4 endp

Got that? In this case 0xD7 in hex is 215 in decimal, which is the ride's excitement rating. I got lucky that this value is static and not changed by anything, which meant I could search for it from outside the binary. This is then stored in the ride's location in memory (register edi), at the offset 0x140. In between there are a few subroutine calls, which shows that nothing is ever really easy when you are reading x86, as well as calls to functions that I have nothing besides hunches about.

Anyway, when you turn this into C, you get something like this:

void crooked_house_excitement(rct_ride *ride)
{
    // Set lifecycle bits
    ride->lifecycle_flags |= RIDE_LIFECYCLE_TESTED;
    ride->lifecycle_flags |= RIDE_LIFECYCLE_NO_RAW_STATS;
    ride->var_198 = 5;
    sub_655FD6(ride);

    ride_rating excitement  = RIDE_RATING(2,15);
    ride_rating intensity   = RIDE_RATING(0,62);
    ride_rating nausea      = RIDE_RATING(0,34);

    excitement = apply_intensity_penalty(excitement, intensity);
    rating_tuple tup = per_ride_rating_adjustments(ride, excitement, intensity, nausea);

    ride->excitement = tup.excitement;
    ride->intensity = tup.intensity;
    ride->nausea = tup.nausea;

    ride->upkeep_cost = compute_upkeep(ride);
    // Upkeep flag? or a dirtiness flag
    ride->var_14D |= 2;

    // clear all bits except lowest 5
    ride->var_114 &= 0x1f;
    // set 6th,7th,8th bits
    ride->var_114 |= 0xE0;
}

And we're lucky in this case that the function is relatively contained; many places in the code feature jumps and constructs that make following the code pretty tricky.

So this one wasn't too bad, but I got bogged down trying to compute excitement for a ride that had a track. The function gets orders of magnitude more complex than this. One positive is, as far as I can tell, excitement and nausea ratings are wholly functions of overall ride statistics like the vertical and lateral G-forces, and there's no accumulator per track segment.

Most of the computation involves multiplying a ride statistic by a constant, then bit shifting the value so it can't be too high/influence the final number by too much.

And sadly, this is where the project stalled. It was impossible to test the C code, because the track computation functions were buried four subroutines deep, and each of those subroutines had at least 500 lines of code. Decompiling each of these correctly, just to get to the code I wanted, was going to be a massive pain. There are ways around this, but ultimately I got back from vacation and had to focus on more pressing issues.

Conclusion

You can hack Roller Coaster Tycoon! There are a bunch of people doing interesting stuff with the game, including improving the peep UI, working on cross compilation (you can play it on Macs!), adding intrigues like the possibility of a worker strike, removing limitations based on the number of bytes (you can only have 255 rides, for example), and more.

It's been really fun having an utterly useless side project. I learned a lot about registers, calling conventions, bit shifting tricks, and other things that probably won't be useful at all, for anything.

I will definitely revisit this project at some point, hopefully when more of the game has been decompiled, or I might try to dig back into the x86/C more on my own.

Liked what you read? I am available for hire.

Levelers

I read a great blog post in college, and sadly I can't find it now, so I'll summarize. It showed a picture of Hearst Castle, and a photo of an average middle-class home, and the text of the post went something like:

  • One of these can travel across the country in 6 hours.

  • One of these can communicate with friends and family on the other side of the country instantaneously.

  • One of these can receive news from the other side of the globe instantaneously.

  • One of these can watch movies in full color and sound in their own home.

  • One of these can expect to live to age 80, and give birth to a child without serious risk of complications.

  • One of these can wash an entire day's worth of dishes in 40 minutes with no human effort.

Hearst Castle. Photo credit Beedie Savage. Creative Commons BY-NC 2.0

The point is, even though Hearst was one of the richest men on the planet, in many ways the average middle class person today enjoys a better quality of life, due to advancements in technology and medical science since the 1940's.

It's really exciting when technologies (air travel, the Internet, smartphones, appliances) become accessible and cheap enough that everyone can enjoy them. But it's even more exciting when the high end of the market comes within reach of most people.

Let's say Bill Gates wants a better smartphone - say, with a longer battery, or with a UI that does exactly what he wants. He's basically out of luck - he can't throw enough money at the problem to get a better phone than what's currently on the market. A phone you can afford is the best one that Bill Gates can buy. I call products like these levelers, because they level the playing field for everyone. It's easy to think of other examples - most software/apps/websites, beers, movies, and more.

So I got excited yesterday when I read about a product that might level the playing field for another experience that's currently only for rich people.

To get a sense of what JauntVR has built, their technology is described as “an end-to-end solution for creating cinematic VR experiences.” Their camera films real world footage from everything surrounding it, and their software stitches those clips together to create a seamless experience. The wrap-around film capture means that you’re able to swivel in any direction to maintain that feeling of “actually being there.”

Rarely is the window into the future as stunningly clear as was for me for the next few minutes inside that Oculus Rift. So what was it like inside? (Keep in mind: writing about the mindshock of live action VR, is quite like trying to share a photograph of your favorite song. Words simply cannot do justice.)

I settled into a swivel chair at the center of a dark surround-sound equipped room, slid on the Rift, and instantly I was no longer in Palo Alto but sitting inches away from a string quartet performing inside a luxurious concert hall. Stunned by the realism (it is real after all!) I began turning in every direction, to discover that behind me I could see the full venue where an audience would sit. Then I understood where I was – right where the conductor would stand. Not a bad seat and with the surround sound so clear, I felt as if I was actually there.

Just like that - millions of people can suddenly all sit courtside for the NBA Finals, or even hover above the court, and suddenly, my view of the game isn't from section 400, it's as good as Jack Nicholson's.

This got me really excited for the future of virtual reality, and the new things that people are going to build on top of it. I can't wait to see what people build.

Liked what you read? I am available for hire.

It begins

Email from my alma mater:

Dear Members of the Claremont McKenna College Community,

I am writing to update you on an important action taken by the Board of Trustees at its meeting on March 9, 2013. In particular, the Board acted to end the College’s “No Packaged Loan” financial aid policy. Beginning with the fall 2014 entering freshman class, the College will reinstate its former practice of including reasonable loan amounts of up to $4,000 per year in the financial aid package for need-based students. This policy change will not affect any current students during their remaining time at CMC, nor will it affect new students enrolling in fall 2013.

This decision was not taken lightly, as we know that there are challenges and pressures that some families face regarding the affordability of a college education. However, our current situation is best understood in view of the College’s long-standing commitment to need-blind admission and to meeting the financially demonstrated need of all admitted, domestic freshman students. The College’s Strategic Plan identifies need-blind admission as one of our most important values, and highlights the importance of insuring our need-blind policy is financially sustainable over the long term. Therefore, I wanted to take this opportunity to briefly discuss the background of the No Packaged Loan policy, and the reasons why the Board determined it was necessary to end the policy at this time.

The College adopted the No Packaged Loan policy in the spring of 2008, just prior to the global financial crisis, at a time when a number of CMC’s peer colleges and universities were implementing various forms of no-loan or reduced-loan policies. At that time, the College completed an extensive financial analysis of the cost of a no packaged loan policy. The financial projections indicated that the College could replace loans with grants within its existing financial aid resources through a combination of actions, including reductions in the amount of merit aid and, most significantly, a reallocation of unrestricted endowment funds that were then being used to support a 0% interest institutional loan fund.

The financial collapse that caused the recent economic recession soon followed. As with most colleges and universities, the economic conditions of the past several years have placed significant pressure on the College’s operating budget. The College has worked to navigate through this period, which has included increasing our commitment to institutional financial aid at almost twice the rate of tuition increases. But we have had concerns about the sustainability of the No Packaged Loan policy, as we have focused on doing everything we can to ensure CMC remains accessible and affordable to all qualified students, regardless of need.

It is within this context that the Board has been engaged this year in a number of important discussions related to the costs and funding of a CMC education, and about the No Packaged Loan policy in particular. This discussion has also included valuable insight and analysis from the faculty, particularly from the faculty’s Admission and Financial Aid Committee (AFAC), who examined the effects of the No Packaged Loan policy on lower-income and minority applicants since the program’s inception five years ago. The Board weighed the AFAC’s findings in their decision and is appreciative of this research.

Through these discussions and careful analysis, the Board decided that, although the No Packaged Loan policy was important to preserve, if feasible, the College’s overarching priority should be to preserve and protect the College’s need-blind admission policies.

In making the decision to eliminate the No Packaged Loan policy, the Board reaffirmed several important commitments:

That the College is committed to providing access to all qualified students based on academic talent and not on financial need;

That the College is committed to securing and strengthening its need-blind admission and to meeting full-need policies by making fundraising for financial aid a priority;

That the College is committed to ensuring that packaged loan levels are reasonable and affordable;

That the College’s financial aid budget will not be reduced by this decision, and ongoing evaluation of the financial aid budget should take place during the next five years.

To help meet these goals, the Board authorized the administration to develop a plan for a targeted fundraising initiative that will focus on securing additional support for financial aid.

It seems probable that many colleges and universities across the country will soon be conducting similar evaluations of their financial aid policies and making changes. It is important for each institution to develop a strategy that assists students and their families to afford higher education with a program that is financially sustainable for the institution. I believe that is what we have done here.

Sincerely,

Pamela Gann
President

Liked what you read? I am available for hire.