Recently I had a very weird problem with iTerm where new login shells were being created with environment variables already present. Restarting my machine made the issue go away, and I wasn't able to reproduce it again.

But I got curious about how long ZSH spends in various parts of the startup process. A new /bin/bash login shell loads instantaneously and I was wondering why my ZSH startup was so slow, and if there was anything I could do to make it faster. My goal was to get to a command prompt in under 100 milliseconds.

What files run when you start a ZSH login shell?

In order, your machine will load/execute the following files when ZSH starts:

/etc/zshenv
~/.zshenv
/etc/zprofile
~/.zprofile
/etc/zshrc
~/.zshrc
/etc/zlogin
~/.zlogin

On my machine only /etc/zshenv (which adds paths in /etc/paths* to the $PATH variable) and ~/.zshrc actually existed, which simplified the problem somewhat.

I used the following two scripts to check execution time. First, this snippet logged the execution time of every command run by my startup script in ~/tmp/startlog.<pid>.

    PROFILE_STARTUP=false
    if [[ "$PROFILE_STARTUP" == true ]]; then
        # http://zsh.sourceforge.net/Doc/Release/Prompt-Expansion.html
        PS4=$'%D{%M%S%.} %N:%i> '
        exec 3>&2 2>$HOME/tmp/startlog.$$
        setopt xtrace prompt_subst
    fi
    # Entirety of my startup file... then
    if [[ "$PROFILE_STARTUP" == true ]]; then
        unsetopt xtrace
        exec 2>&3 3>&-
    fi

Essentially this prints the command that zsh is running before you run it. The PS4 variable controls the output of this print statement, and allows us to insert a timestamp each time we run it.

I then wrote a short python script to print all lines above a certain threshold. This is okay, but can show a lot of detail and won't show you if one line in your .zshrc is causing thousands of lines of execution, which take a lot of time in total.

Observations

At first I used the GNU date program in the PS4 prompt to get millisecond information about the time, however this program consistently used 3 milliseconds to run, so it got costly to run it thousands of times during system profiling.

The first thing you find is that shelling out is very expensive, especially to high-level languages. Running something like brew --prefix takes 50 milliseconds which is half of the budget. I found that Autojump's shell loader ran brew --prefix twice so I submitted this pull request to cache the output of that and run it again.

Another timing method

Ultimately what I want is to time specific lines/blocks of my zshrc, instead of get profiling information for specific lines. I did this by wrapping them in time commands, like this:

    { time (
        # linux
        (( $+commands[gvim] )) && {
            alias vi=gvim;
            alias svi='sudo gvim'
        }
        # set up macvim, if it exists
        (( $+commands[mvim] )) && {
            alias vi=mvim;
            alias svi='sudo mvim'
        }
    ) }

This will time commands in a sub-shell, which means that any environment variables set in the sub-shell won't be set in the environment. However the purpose is to get timing information and it's good enough for that.

Lazy loading

By far the worst offenders were the various "Add this to your .zshrc" scripts that I had added in the past - virtualenvwrapper, travis, git-completion, autojump, pyenv, and more. I wanted to see if there was a way to load these only when I needed them (I don't, frequently). Turns out there is! Most of these set functions in zsh, so I can shadow them with my own functions in a zshrc. Once the file with the actual function definition is sourced, it'll replace the shim and I'll be fine. Here's an example for autojump:

    function j() {
        (( $+commands[brew] )) && {
            local pfx=$(brew --prefix)
            [[ -f "$pfx/etc/autojump.sh" ]] && . "$pfx/etc/autojump.sh"
            j "$@"
        }
    }

Or for pyenv:

pyenv() {
    eval "$( command pyenv init - )"
    pyenv "$@"
}

Essentially on the first invocation, these functions source the actual definition and then immediately call it with the arguments passed in.

Conclusion

Ultimately I was able to shave my zsh startup time from 670 milliseconds to about 390 milliseconds and I have ideas on how to shave it further (rewriting my weirdfortune program in Go for example, to avoid the Python/PyPy startup cost). You can probably get similar gains from examining your own zshrc.

Liked what you read? I am available for hire.

You have photos on your computer. You would probably be really sad if you lost them. Let's discuss some strategies for ensuring that doesn't happen.

What you are doing now is probably not enough.

Your current strategy is to have photos and critical files stored on your most current laptop and maybe some things in Dropbox. This means you will probably lose your data sometime in the next three years.

Hard drives fail, often, and are unrecoverable, or expensive to recover.

The software that manages your photos is written by people who are bad at their jobs, or didn't anticipate the software running on (insert hardware/software condition 10 years from now here) and it breaks, destroying your photos in the process.

Backup services fail, often enough to make you worry. Apple can't be trusted to reliably deliver messages with iMessage, I don't trust Time Machine at all.

Apartments and cars get broken into and laptops/external drives are good candidates for theft. You can buy renters insurance, but you can't get your photos back.

What you should do

I'm going to focus specifically on keeping photos for a long time. For files and folders which change more often, I use git and source code tools like Github or Bitbucket. For work environments that take a while to set up, I try to automate installation with shell scripts instead of trying to store the entire environment via Time Machine or similar.

I'm also going to pick a thirty year time horizon, just for fun. If you want something to last for thirty years, you can't just store it on your local machine because your hard drive will probably fail between now and then, and then you lose your photos. So you want to store it on your machine and then also somewhere offsite (e.g. not in the same 5-mile radius as your machine).

Storage Tool

Which raises the question, which storage/backup companies will be around in 30 years? Any small enough candidate, even Dropbox, could get acquired or shut down in that time frame. My guess right now is that Amazon Web Services will be around the longest, because it is profitable, already part of a large, growing company, and the service is growing rapidly.

Specifically, I am putting a bet on Amazon Glacier, which has extremely low storage costs - $0.01 per GB - and is one of the most reliable services Amazon runs. Glacier is a subset of Simple Storage Service (S3) which has had extraordinarily good availability in the 7-8 years it's been available. In addition Amazon regularly publishes information on the technology underpinning S3, see "Amazon Dynamo paper" for example.

I use Arq to back up photos to Glacier. It seems fairly stable, has software preferences for the right things, and I am encouraged that the author charges for the software, which means he/she has an incentive to continue developing the product and making sure that it works. This is $30 but this is still much, much cheaper than any other tool (iCloud for example would be $100 per year).

I have 52 GB of files, which means I'll pay roughly $7 per year to have 11 years of photos stored safely in the cloud. Not bad.

I would consider Tarsnap as well, which encrypts your data, but it's currently 2.5x the price of Glacier. I expect this price to decline soon.

Photo Management Software

The second piece is you need to choose a stable piece of software for viewing and managing your photos. This is orders of magnitude more risky than Glacier. The ideal piece of software would have:

• An easy to understand file format, published online

• Source code available online

• Some support for grouping photos into albums and importing them from my phone/camera/whatever

• Some sort of tool for auto-adjusting the layers on a photo, cropping it, editing the brightness/contrast. Not Lightroom, but enough to make a photo better than it was.

• Supports hundreds of gigabytes of photos without falling over.

I haven't done as much research into this as with backup solutions, but there are a few tools. Picasa is supported by Google and relies on Google's charity to stay running and supported. Lightroom is very nice, but overshoots my needs, is very expensive, and Adobe may run out of money and fold within the next 30 years. This leaves iPhoto, which isn't well documented, or open source, but mostly works, some of the time, can crop/edit photos while saving the originals, and is a core component of Apple's Mac product, which may also die, when we all have tablets. At least iPhoto stores the files on disk.

If I was a better person, I'd manually arrange photos in files on my filesystem and use that. But I am not.

Problems

In this game it's a question of how paranoid you want to be, and how much your photos are worth to you. If they're worth a lot it's worth investing significant time and resources into redundant backup systems. Specifically, the 3-2-1 rule suggests storing backups in 3 different places, 2 different file systems (or photo management tools), with at least one offsite. For photos, backing up to CD's or tapes is not a terrible option though it's not very efficient and CD's also die. Using a 2nd photo management software solution, storing the data for that one on an external drive, and backing one of them up to Glacier every night is not a terrible idea.

My photos are worth a lot to me, but I'm not going to go insanely overboard; if my hard drive dies, and Glacier loses my data, that would suck, but I'm willing to gamble on the one-in-a-million odds of them not both happening simultaneously.

I am most worried about the management software; recently iPhoto decided I had no photos in my library and deleted all of the metadata for those photos (albums, rotation information, etc), leaving only the raw copies in my library. This was very sad, though I didn't lose the photos themselves, and might be able to recover the metadata if I am lucky. So yeah, I am not too happy with this. I now store important iPhoto databases in git, backed up to Bitbucket, so I can pull them back down in the event of another catastrophic failure.

Conclusion

This is too long already. Ensuring your photos aren't lost is difficult, and probably requires professional experience as a computer programmer to get right, or for you to part with a significant amount of money, or achieve a large amount of good luck, to avoid the bad things that happen to most people who just hope their photos stick around. I wish you the best of luck.

Liked what you read? I am available for hire.

You kick off a long running job - maybe a data migration script that operates on a large data set, or you're copying a large file from one disk to another, or from the Internet to your local computer.

Then a few minutes in, you realize the job is going to take longer than you thought, and you'd like to trigger some action when it's done - notify you, or remove a temp directory, or something.

Like this: wget reallybigfile.com/bigfile.mp3 && say "file done downloading"

But you can't queue an action without hitting Ctrl+C and restarting the job, setting you back minutes or hours. Or can you?

With most modern shells on Unix, you can suspend the running process, and the Unix machine will freeze the state of the process for you. Simply hit Ctrl+Z while any process is running and you will get a message like this:

$ sleep 10
^Z
[1]  + 72277 suspended  sleep 10

You can then resume it with the fg command, which tells Unix to resume operations with the suspended process. You can then combine fg with the notification command of your choice. So let's say you've suspended the process with Ctrl+Z, you can bring it back to the foreground and attach actions afterwards like so:

fg; say -vzarvox "Job complete."

Of course, you can do whatever you want instead of using the say command; trigger another long running operation or whatever.

I use this probably about once a day, it never fails and it's always useful. Hope it helps you too!

Liked what you read? I am available for hire.

It's important to recognize the people that have contributed to your project, but it can be annoying to keep your project's AUTHORS file up to date, and annoying to ask everyone to add themselves in the correct format.

So I did what any good engineer should do, and automated the process! I added a simple make target that will update the authors file based on the current Git history:

    # Run this command to update AUTHORS.md with the latest contributors.
    authors:
        echo "Authors\n=======\nWe'd like to thank the following people for their contributions.\n\n" > AUTHORS.md
        git log --raw | grep "^Author: " | sort | uniq | cut -d ' ' -f2- | sed 's/^/- /' >> AUTHORS.md

Essentially, that snippet gets every author from the git history, sorts and unique-ifies the list, strips the word "Author: " from the line, and outputs the new list to the AUTHORS.md file. Now updating the authors list is as easy as running make authors from the command line.

Liked what you read? I am available for hire.