Posts Tagged With: Code

Tools I use to get the job done

Posted on

Every day I look for new tools that'll help me get my work done faster. Here's some of the stuff I use to get the job done.

My [only] machine is a 13' MacBook Pro. I have a 120GB solid state drive, which is the single best upgrade you can make to any machine - it makes your laptop so much faster. I also have 8GB of RAM. Don't buy RAM or an SSD from Apple, buy the parts from Newegg and then install them yourself using the guides on iFixit. With the new unibody laptops you'll need a Phillips #00 and a Torx T5 screwdriver but you can get these from your local hardware store for less than $10.

I also have a 2TB external drive with backups and my music library, and a Samsung scanner. I use AudioTechnica noise-canceling headphones to stay focused in busy environments.

I have a pretty elaborate set of dotfiles that I use to configure the terminal, vim, and other applications. I keep these in a Mercurial repository so I can get my configuration set up really quickly on a new machine.

MacVim

iTerm 2 and bash are fantastic for a terminal, and MacVim is a great GUI editor for vim - I use it to write blog posts and code. I use Adium for chat and Quicksilver to open and close applications and control iTunes. Skitch is an excellent tool for screenshots, because you can share photos with one click - I use it at least once a day. I use Colloquy for IRC (it's chat for programming people). Jumpcut helps keep track of things I've pasted to the Clipboard and TextExpander expands snippets I use often, like my email address and our home wireless password.

I'm obsessed with maximizing screen space. I keep the Dock minimized, (a) because I really never use it and (b) because it takes up about 50-100 pixels at the bottom of the screen. Divvy resizes/moves windows (to quickly make windows full screen, or left half/right half) and Witch switch between open windows (instead of applications) quickly. At home I have two ASUS 24' monitors; one connects through the DisplayPort and the other one connects with a Diamond BVU adapter.

I use Chrome for casual web browsing and Firefox + Firebug for web development. I use 1Password to generate strong passwords. I use Webfaction for web hosting; it's the best combination of price and features (full SSH, multiple websites, etc) I've been able to find. I might upgrade to Linode soon however, for faster sites and more space on the server. I use rsync to copy my 1Password Anywhere file to the Webfaction server, giving me access to my passwords from any web browser. I also use Shell in a Box to get SSH access to Webfaction from any browser.

I really wish there were better single site browser apps for Mac. Apparently Chrome and Firefox are both working on tools for this. I have single site apps for Google Voice, Calendar, and Gmail, configured with Prism. This helps me pull them up quickly, especially when I have 40+ tabs open in Chrome.

Some stuff I do that's atypical: I type using a DVORAK keyboard, to the endless annoyance of people trying to borrow my laptop. I also use Scroll Reverser to have the scroll direction mirror that of mobile devices; you move your fingers up to scroll down (apparently this is the default in Mac OS Lion).

I have bank accounts with ING Direct and Ally Bank. In my investment account on Sharebuilder I'm invested 50% in VT (a world market fund), 30% in PCRDX (a commodity futures index fund), and 20% in RWO (a global real estate fund).

Liked what you read? I am available for hire.

Test if your server is up, using Ruby

Posted on
How do you know if your website is up, right now? Maybe you are browsing it and got an error, or a friend texted to tell you it was down. Otherwise your website might be down and you'd have no idea. Here is a simple script that will help you learn faster whether your website is up or down. I wrote it in Ruby to start learning Ruby syntax. 

require 'uri'
require 'net/http'
require 'net/smtp'

def send_email(to, opts={})
    # Send an email. In Ruby, the ||= operator means 'only assign the
    # new value if the value has not been set yet'
    opts[:server]      ||= 'localhost'
    opts[:port]        ||= 25
    opts[:ehlo_domain] ||= 'localhost'
    opts[:username]    ||= 'username'
    opts[:password]    ||= 'password'
    opts[:from]        ||= 'youremail@domain.com'
    opts[:from_alias]  ||= 'First Last'
    opts[:subject]     ||= 'Test email'
    opts[:body]        ||= 'This is a test!'

    # the following says 'create a new string called msg containing everything
    # from here to the END_OF_MESSAGE marker'
    # then, we fill in the email with all of the values we filled in above. Note
    # that there has to be a newline between the subject and the body of the
    # message.
    # This is a plaintext message - to send HTML emails, we would need a more
    # complex template.
    msg = <<END_OF_MESSAGE
From: #{opts[:from_alias]} <#{opts[:from]}>
To: <#{to}>
Subject: #{opts[:subject]}

#{opts[:body]}
END_OF_MESSAGE
    Net::SMTP.start(opts[:server], opts[:port], opts[:ehlo_domain],
                    opts[:username], opts[:password], :plain) do |smtp|
        smtp.send_message msg, opts[:from], to
    end
end

# test for main method
if __FILE__ == $0
    # the domain we want to check
    url = 'http://testdomain.com'
    my_email = "myemail@gmail.com"
    r = Net::HTTP.get_response(URI.parse(url))
    if r.code != "200"
        # we have a big problem, server is down or other
        opts = {
            :server     => 'smtp.yoursmtpserver.com',
            :port       => 587,   # should be 25 or 587
            :ehlo_domain => 'localhost',      #this doesn't matter too much
            :username   => 'yoursmtpusername',
            :password   => 'yoursmtppassword',
            :from_alias => 'testdomain.com Admin',
            :subject    => 'The website is down',
            :body       => "Hey,\nan attempt to reach the homepage returned " + 
            "the status code " + r.code.to_s() + ". \n\nYou should check it out."
        }
        send_email my_email, opts
    end
end
That will send an HTTP request to your website. If the request was successful, you'll get an HTTP Header Response that says 'Status Code: 200 OK'. If we don't get a 200, this means the site is definitely wrong. We may get a 200 and still have problems with the database or other, but this is meant to be a simple test. Other well known codes are 404 (File Not Found), 500 (Internal Server Error), and 418 (I'm a teapot). Once you've got the script in place, save it as "is_server_up.rb" somewhere on a computer which has lots of uptime (your school's CS server, a remote SSH server, or otherwise). Your laptop is a bad choice because when you close your laptop, or are not connected to the Internet, the test will not be able to run. The computer hosting your website is also a bad choice because if that computer goes down, your website will be down but the script will not run. On a Linux or Mac machine, type 
$ crontab -e
which should open up a Terminal text editor. Then type: 
0-59/2 * * * * /usr/local/bin/ruby /path/to/your/file/is_server_up.rb
where 
/usr/local/bin/ruby
is the path to the Ruby executable on your laptop. If you don't know where it is, try typing "which ruby" in the Terminal. If nothing comes up, you probably need to download Ruby. The 
0-59/2
bit says to run the script every 2 minutes. You might be worried that fetching your homepage every 2 minutes will generate an extra 720 hits to your site every day and affect your Analytics stats. However, Analytics will only measure visits from browsers that can execute Javascript, and this Ruby script will not execute any Javascript, so you are safe. Note that if you have an important site, or one which generates revenue, you probably should not use this tool - there are better tools like Pingdom that will let you know right away whether your site is down.

Liked what you read? I am available for hire.

How to find a Pearson’s correlation and ordinary least squares in Python

Posted on
I'm going to try to blog more technical stuff, as well as simple recipes, here more. I'm working out the best way to present it - whether to bundle everything together or separate out the technical posts. Today we're going to use Python to find a simple correlation, and then fit a straight line to the curve. First you want to install the SciPy and NumPy libraries - they have a lot of cool functions for Python. On Mac, if you have MacPorts installed, this is trivial: 
$ sudo port install py27-numpy py27-scipy
Then you find a Pearson's correlation as follows: 
# The dependent variable
>>> x = [1, -2, 2, 3, 1]

# The independent variable
>>> y = [7.5, -3.5, 14.5, 19, 6.6]

#Find a correlation
>>> from scipy.stats.stats import pearsonr

#First value is the r-value, 2nd is the p-value
>>> pearsonr(x,y)
(0.98139984935586166, 0.0030366388199721478)

# To find the best-fit line, use the numpy directory
>>> from numpy.linalg import lstsq

# Put the x variable in the correct format.
>>> A = numpy.vstack([x, numpy.ones(len(x))]).T

>>> A
array([[ 1.,  1.],
       [-2.,  1.],
       [ 2.,  1.],
       [ 3.,  1.],
       [ 1.,  1.]])

>>> lstsq(A, y)
(array([ 4.5 ,  4.32]), array([ 10.848]), 2, array([ 4.53897844,  1.84327826]))
# 4.5 is the slope, 4.32 is the y-intercept. 10.848 is the sum of the residuals. 
# 2 is the rank. The last array are the singular values.
For more details, including how to plot the correlation, see the NumPy documentation here.

Liked what you read? I am available for hire.

Presentation on JIT compilers (and the unfortunate sex life of the banana)

Posted on
Posting's been kind of light, so I'll link to two presentations I've done for class recently. I'm currently taking a course on compilers, which is the best course I've taken in college. Very quickly, your computer can only understand a very limited set of instructions: add 01110011 to 11001011, load eight zeroes and ones from memory (the cache, or random-access memory (RAM)), overwrite a set of zeroes and ones, etc. However, if you had to write entire programs using only this instruction set, it would be hard for programmers to keep track of variable state, to write error-free code and to find bugs when they are made (Notable exception: Roller Coaster Tycoon was written entirely in x86 assembly language, which explains (a) why there has never been a Mac version released, and (b) why RCT3 was so much worse - it was written in a higher-level language by a different company). So what programmers did was invent abstractions - control flow tools like if-then-else branching, while and for loops, as well as data structures like arrays and dictionaries. These allow programmers to write in a more natural language, but programs must be translated - "compiled" - down to machine language. Because this is done automatically, and many people use the same compiler (so everyone benefits from errors caught by one person), generated machine code is usually free of errors. I gave a talk in class on a special type of compiler, called just-in-time (JIT). Normally you write your program, compile it, and then run the machine language. But a JIT compiler is writing and compiling new machine language as the program is running. Here are the slides: There are a few reasons this can be quicker than compiling beforehand ("static" compilation). First, you have more information about the values of variables at runtime that you can use to eliminate redundancy in your code. Second, in the case of loops, where you are following the same code paths hundreds or thousands of times in a row, you can see which paths are being used the most often, and rewrite your code so that the most common code path is executed in a straight line. This is important because modern processors will try to pre-fetch and execute several instructions at a time, and if you keep jumping from place to place in your code, the processor can't execute instructions in advance. I also wrote a paper (and gave a talk) on some of the threats facing the Cavendish banana, which will probably go extinct soon. I've been fortunate to write papers this year on some fun topics; last semester I was able to cite Robin Hanson, Tyler Cowen, Eliezer Yudkowsky and Alan Turing in one philosophy paper, which was pretty fun. That paper's here. The Unfortunate Sex Life of the Banana

Liked what you read? I am available for hire.

Everything I know about debugging

Posted on

If you write programs, a large portion of your time is spent testing your code, and finding and removing bugs in your program. Not being able to understand a bug is one of the more frustrating experiences you can have as a programmer. I’ve been on the lookout for ways to reduce the amount of time I spend debugging, but I really haven’t seen any good articles on the theory of debugging. Here’s everything I know about debugging. If you have additions, feel free to leave a comment.

Write better code

The best defense is a good offense – if you write your code in a clear and concise way, and write by making incremental changes which are immediately tested, you are less likely to run into time-sucking bugs later on. Some people recommend test-driven development and code review as ways to cut down on the number of bugs that make their way into programs. Remember, the earlier you can catch bugs, the less expensive they are to fix.

Try simple fixes for 2-3 minutes

Some bugs are simple and can be fixed within minutes. I would recommend spending 2 or 3 minutes trying a few simple fixes before bringing out the big guns. Yes, this is “cargo cult” programming but maybe 50% of the bugs you find have trivial fixes, and it might not be cost-effective to bring out the big guns right at the start.

Set up a reduced test case

Try to set up the simplest possible version of your code that produces the error. In HTML and CSS, this is known as a reduced test case. This often involves deleting large swathes of your code, so I would recommend either copy/pasting to another page, or committing the current version of your code, then making changes, and reverting to the original version once you’ve identified the problem. If you have a unit test suite, this would be a good time to add a new test case to the suite.

Use print statements or a step-through debugger

If setting up a reduced test case is not feasible (multiple dependencies or complex code), use print statements to determine the values of variables before and after all method calls in your program that could be causing the bug. This is tedious, but systematic, and preferred to cargo cult programming, where you make changes at random and hope that they solve your problem. Another approach is to use a step through debugger – stop the program at a point where you know the output is correct, and then step through the program line by line, monitoring the information in the program. In python, the pdb module does this. In Java, the DrJava program is looked down upon for being mostly for students, but I’ve found it extremely useful for viewing the values of variables at points in the program. Most programming languages should have a step-through debugger.

Use assert statements

Alternately, you can use “assert” statements to assert that a variable should have a specific value at a point in your code. This will stop code from executing immediately at the point where a variable has an incorrect value. This has the added benefit of making sure you understand your code.

Concede defeat

If you’re still at a loss, consider ways in which you can “go around” the bug, by rewriting sections of your code in a way you understand, or trying something else entirely to solve your goals. Note: I struggle with this because not knowing the cause of a bug drives me up a wall, but you might have some success with it. Also, stepping away from your computer and thinking about the problem in more abstract terms can help – I’ve had some luck with just taking a shower and thinking through the problem.

Ask the Internet for help

If you’re still at a loss, try copying and pasting your error message into Google, or posting on StackOverflow. Be sure to be explicit about what you expected to see and what actually happened, and post the code of any error messages you might have. You can also try the IRC channel for the programming language, but I’ve had less success with this – there might be someone nice who’s willing to help you out, but more often than not your message will be ignored.

About me
My name is Kevin Burke and I’m a senior at Claremont McKenna College. I started programming in January of 2010. Read more about me, or follow me on Twitter here.

Liked what you read? I am available for hire.

Disable ESPN Autoplay

Posted on
I wrote a Google Chrome extension that stops videos and ads from playing automatically on ESPN.com. This is another example of scratching my own itch; most people can enable this feature by clicking Autostart Off on any ESPN video, but I clear out my cookies every time I close Chrome, so that tool doesn't work for me. Also, this is more important for me than other people because I'll click to open three stories at once, and if the videos all begin playing at the same time, it gets extremely annoying. Anyway, here's the extension, all six lines of it in Javascript: 
jQuery("#videoPlayer").ready(function(){
    var script = document.createElement("script");
    script.type = "text/javascript";
    script.text = "function check_if_ready(){if (espn.video.player.adPlaying){espn.video.pause();} else{setTimeout(check_if_ready, 100);}}check_if_ready();"
    document.body.appendChild(script);
});
Unfortunately getting to that point took a while; I tried a few other things before I hit on that solution. It's not perfect but it gets the job done. In the future it would be nice to skip the ads entirely, or auto-play only the video in the tab I'm currently watching. You can download the extension here or improve the source code here.

Liked what you read? I am available for hire.

Auto-lightbox images in WordPress

Posted on

I’m starting a new side project, where I’ll have to insert lots of screenshots for each blogpost. I’d like to allow users to enlarge images without having to leave the current page. The best way to do this is with a “lightbox,” which darkens the background and displays the image in the center of the screen.

What I would like is just to post an image to the blog and have the lightboxing and image resizing occur automatically. Unfortunately none of the WordPress plugins let you do this; most of them require you to add an extra link with a special tag. So I wrote this short bit of Javascript which extends a popular WordPress lightbox plugin, Lightbox 2. The script takes an image and wraps an “a” tag around it with the rel=”lightbox” attribute set, which triggers the Lightbox code. It also resizes the image to a maximum of 600 pixels wide or high. Here’s the code:

//get all images inside entry content
jQuery(document).ready(function(){
    jQuery(".entry-content img").each(function(){
//get the URL of the image
        var the_url = jQuery(this).attr('src');
//insert a href before image where rel=lightbox
        var a = jQuery("<a></a>").attr({"href": the_url, "rel": "lightbox"});
        jQuery(this).wrap(a);
        jQuery(this).load(function(){
            var the_width = jQuery(this).width();
            var the_height = jQuery(this).height();
            var max_dimension = 600;
            if (the_width && the_height){
                var ratio = the_width / the_height;
                if (ratio >= 1 && the_width > max_dimension){
                    jQuery(this).css('width', max_dimension);
                    jQuery(this).css('height', the_height * max_dimension / the_width);
                }
                else if (ratio > 1 && the_height > max_dimension){
                    jQuery(this).css('height', max_dimension);
                    jQuery(this).css('width', the_width * max_dimension / the_height);
                }
            }
        });
    });
});

You can also download the file here. To install, edit the header.php file of your WordPress directory and add the following lines above the

<?php wp_head(); ?>

line:

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js" type="text/javascript"></script>
<script src="/path/to/img.js file" type="text/javascript"></script>

Then you’re done!

Liked what you read? I am available for hire.

The ACM ICPC: a supposedly fun thing I’ll never do again

Posted on
Yesterday I was part of CMC's first ever entry into the ACM ICPC, an international programming competition. Last year solving three problems would have been good enough for 5th place, but this year solving three problems was only good enough for 29th out of 75 or so. The easiest three problems were easy to pick out and we started work on them. The first problem was, given an expected distribution of T-shirt sizes and a number of participants, write some code to determine how many T-shirts to order. The code for this problem was more or less written in the problem description and I was able to solve it in the first twenty minutes or so. While I was doing that my teammates were working on a cipher problem - given a codeword, create a simple code alphabet and use it to decrypt a message. While they were doing that going on I started working on the third problem, which was a map labeling algorithm. It was like a Rand McNally map where if you go off the top of the map you have to go to another page on the map. You were given a 'map' divided up into a series of blocks, and had to number each block, and then for each block label the block's neighbors. I solved this one pretty quickly too, however I forgot to remove a print statement from my code, so it was printing an empty line before the output, which cost two incorrect submissions and 40 penalty minutes. The cipher problem turned out to be fairly difficult. There was a problem with the program that took about an hour to solve, partly because we kept printing out code that we thought was showing one thing when in fact it was showing something else. Also, we had an array of characters that was supposed to hold all of the letters of the alphabet from A to Z, but in fact was missing W. Once we solved those we submitted the code and got the correct answer on the first submission, but by this point we had an hour and a half left. I started working on the fifth problem, which I thought was the next easiest. The problem was, given a triangle and a thousand points inside that triangle, split the points into three even groups by finding another point inside the triangle. triangle I found the algorithm right away, which was to do a binary search and always move toward the region that contained the most points. However, this turned out to be tricky to implement in practice. Ultimately this problem was really difficult and only a few groups solved it correctly (the winning Harvey Mudd team solved seven of eight problems and this was the one problem they didn't solve). Trying it was a tactical mistake on my part, and we might have been able to solve a fourth if we'd attempted a different problem. Ultimately it was a long day and as my teammate Jacob noted, "the nerdiest thing I've ever done." It was good I guess, considering I was the only person on our team who'd practiced before the contest and we were the first team ever from CMC to go to the competition. I'd wanted to join the competition because I show my resume to people and they say "Great you're an economics major, so, marketing?" and I wanted to show that I could code a little bit. It would have been great to do better in the competition but them's the breaks. Among other things this shows how wide the differences between programmers can be; it's estimated that the best programmers are 10 times as efficient as average programmers, and looking at the times to completion it's easy to see how this is true. By the time I solved Problem 1 the Harvey Mudd team had already solved three problems, and there were some teams there that didn't even solve a single problem. These are hard to measure, which is why it's so important to actually look at someone's code to see how they do things, or ask them to write sample code. All of my practice code can be found here; unfortunately you can't keep the code you wrote during the competition.

Liked what you read? I am available for hire.

Google Finance killed my weekend project

Posted on

One annoying thing about Google Finance is that there’s no way to see how your portfolio’s value has changed over time. You can see your cost basis and the current value but none of the intermediate values. Way back when I used to log into MSN Money and record the value in a spreadsheet every day. That didn’t last very long.

Over the weekend I built a tool that keeps track of how much your portfolio’s worth. Every night at midnight it would grab the current value of all your portfolios and store it in my database, and then when you visited my site again, it would display a chart with all of the daily values of your portfolios. I got the entire backend working, and got to the point where the app was displaying all of the daily values on the page in plain text. Here’s my cost basis and a week’s worth of daily values, with some numbers blurred.

This weekend I was going to turn those numbers into a pretty Annotated Time Line using the Google Chart API. Unfortunately (or fortunately), I checked Google Finance today and it turns out that they just built the portfolio value chart into their main product.

Google Finance portfolio view

It’s pretty cool that they put the chart in there; obviously it would have been nice to have visitors to my own site, and I built it so that anyone can log in and check the values, but also a little deflating, that my project won’t be useful for people. Guess I’ll have more time to work on my other projects. In the meantime though, I got better at XML parsing, learned a ton about OAuth, implemented an app with user-specific data for the first time, and got better at programming.

You can grab the code I was using here, or check your own portfolio here.

To learn how you can outperform the Dow by 10%, or to learn more about the app, you should email me here.

Liked what you read? I am available for hire.

Escaping from your email

Posted on

I only download my email once every four hours. Otherwise I check it too much. This works really well and my "need to check" email addiction has totally shut off.

For this to work on your own email, you want to set up a Gmail account with an address that no one knows, as well as access to a cron task scheduler. Cron is an application that runs tasks on a set schedule, so every minute or every year or every Monday at 5pm, etc. If you have Mac or Linux, and you're the type of person that keeps their computer on 24/7, you can run "cron jobs" from your local machine (you want to edit the /etc/crontab file - here are some instructions). I use my web hosting provider's cron service, because it will run even when my laptop is shut down.

Here's the script:

 
import imaplib
from datetime import datetime
import email
import smtplib
import sys
import os
import random
import base64

def main():
    #check user
    if os.isatty(sys.stdin.fileno()):
        #i'm trying to get my email by cheating, not a cron
        print "Are you sure you want to do this?"
        ans = raw_input()
        if ans != "yes":
            return
        no_of_mults = 3
        for i in range(no_of_mults):
            a = random.randint(1000,9999)
            b = random.randint(1000,9999)
            print "What is " + str(a) + " * " + str(b) + "?"
            ans = float(raw_input())
            if ans != a*b:
                print "Sorry, incorrect answer."
                return

    imap_domain = "imap.gmail.com"
    imap_port = 993
    imap_username =  base64.b64decode('my_gmail_username_encoded')
    imap_password = base64.b64decode('my_gmail_pass_encoded')

    #smtp settings
    smtp_domain = "my_smtp_host.com"
    smtp_port = 587
    smtp_username = "my_smtp_user"
    smtp_password = "my_smtp_pass"

    email_recipients = ['kburkeorg@gmail.com']

    imap_server = imaplib.IMAP4_SSL(imap_domain, imap_port)
    imap_server.login(imap_username, imap_password)

    server = smtplib.SMTP(smtp_domain, smtp_port)
    server.ehlo()
    server.starttls()
    server.ehlo()
    server.login(smtp_username, smtp_password)

    imap_server.select('INBOX')

    status, email_ids = imap_server.search(None, '(UNSEEN)')

    recipient = email_recipients

    count = 0
    for e in email_ids[0].split(' '):
        if e is not '':
            try:
                raw_msg = imap_server.fetch(e, '(RFC822)')
                msg = email.message_from_string(raw_msg[1][0][1])

                #modify reply-to so we preserve email address
                if not msg['Reply-To']:
                    msg['Reply-To'] = msg['From']

                result = server.sendmail(msg['From'], recipient, msg.as_string())
                count += 1

            except smtplib.SMTPRecipientsRefused, e:
                print "SMTPRecipientRefused" + str(e)
                continue
            except Exception, err:
                print err
                continue

    # so I can see how many messages were printed
    print "Sent " + str(count) + " messages."
    filename = "/home/kevinburke/webapps/b/sent.txt"
    with open(filename, 'w') as f:
        f.write(str(count))
    print datetime.now()

    server.quit()

if __name__ == "__main__":
    main()

Then in your cron file, you want to add the following line:

0 */4 * * * /usr/bin/python /path/to/email/file.py

where 

/path/to/email/file
is the filepath to wherever you saved the script above. The 0 means run the script at the top of the hour, and the */4 means run it once every 4 hours.

Liked what you read? I am available for hire.