C++, JAVASCRIPT, GIS, AND OTHER TECH STUFF

JavaScript Development with jsdom When Offline

I've been working on a project to create a Shakespeare twitter-bot. While parsing all of Shakespeare's work from the MIT hosted web pages I used the jsdom package to crawl through the html. Recently while working on the project offline, I came into some problems with jsdom. The below sample from jsdom's github wouldn't work while I was offline:

// Count all of the links from the Node.js build page
var jsdom = require("jsdom");

jsdom.env(  
  "http://nodejs.org/dist/",
  ["http://code.jquery.com/jquery.js"],
  function (errors, window) {
    console.log("there have been", window.$("a").length, "nodejs releases!");
  }
);

In order to load jquery from a local source the devs for jsdom suggest using the following format for grabbing a local source files:

// Print all of the news items on Hacker News
var jsdom = require("jsdom");  
var fs = require("fs");  
var jquery = fs.readFileSync("./jquery.js", "utf-8");

jsdom.env({  
  url: "http://news.ycombinator.com/",
  src: [jquery],
  done: function (errors, window) {
    var $ = window.$;
    console.log("HN Links");
    $("td.title:not(:last) a").each(function () {
      console.log(" -", $(this).text());
    });
  }
});

I don't know what the reason was, but I couldn't get this to perform correctly. So instead I'm running a local python bottle server in order to serve myself the jquery file when offline:

from bottle import Bottle, run, static_file

app = Bottle()

@app.route('/static/<filename>')
def server_static(filename):  
    return static_file(filename, root='./static')

run(app, host='localhost', port=8080)

Now I update the original jsdom JavaScript, connect with the localhost on the Python Bottle server port, and everything is happiness:

// Count all of the links from the Node.js build page
var jsdom = require("jsdom");

jsdom.env(  
  "http://nodejs.org/dist/",
  ["http://localhost:8080/static/jquery.js"],
  function (errors, window) {
    console.log("there have been", window.$("a").length, "nodejs releases!");
  }
);

Repairing a WAV File with Java

wav file icon

A friend of mine produces podcasts for a living and she needed help with a corrupted WAV file. What I think happened is that the recording device malfunctioned or it ran out of disk space, so it never completed the WAV file header information. Without the correct header info it couldn't be read properly.

After inspecting the file she discovered that the incomplete parts of the header were the file size and the data size (it blew me away how much she'd learned about the WAV header format in such a short period). She had recovered all the data from her flash drive and all I needed to do was search for the beginning of the header information and rebuild the file from that point onwards with correct header info.

WAV files have a mix of big and little endian encoding, so there were a few moments of confusion. But eventually I was able to salvage over an hour of her interview. Feels great to use my coding skills to help out a friend!

I had to use a lot of different resources to really understand the header information. Here are a few of the most helpful:

I also used a few different tools to help convert hexadecimal to decimal and to look at the file's hexadecimal encoding:

I probably didn't need to write a program to do this and could have used hex fiend's editing options, but sometimes life is more fun if you just throw yourself into some code!!

Here's the Java I used to clean up the file:

import java.io.*;  
import java.io.FileOutputStream;  
import java.nio.ByteBuffer;  
import java.nio.ByteOrder;


public class Main {

  public static void main(String[] args) throws java.io.IOException {
    BufferedInputStream in = null;
    BufferedOutputStream out = null;
    File file = new File(args[0]);
    int fileSize = (int)file.length();

    try {
      in = new BufferedInputStream(new FileInputStream(args[0]));
      out = new BufferedOutputStream(new FileOutputStream(args[1]));
      byte[] ioBuf = new byte[4];
      int bytesRead;
      boolean bFoundHeader = false;
      while ((bytesRead = in.read(ioBuf)) != -1) {

        // int representation of "RIFF", faster than converting 
        // the value to string every time
        int num = ioBuf[0] << 24 |
                (ioBuf[1] & 0xff) << 16 |
                (ioBuf[2] & 0xff) << 8 |
                (ioBuf[3] & 0xff);

        // if RIFF found
        if (1380533830 == num)
        {
          // write RIFF, bytes 1-4
          out.write(ioBuf, 0, bytesRead);

          // write file size, bytes 5-8
          ByteBuffer bb = ByteBuffer.allocate(4);
          bb.order(ByteOrder.LITTLE_ENDIAN);
          bb.putInt(fileSize - 8);
          bb.flip();
          in.read(ioBuf, 0, 4);
          out.write(bb.array(), 0, 4);

          // write header contents, bytes 9-40
          byte[] ioBufHeader = new byte[40-8];
          in.read(ioBufHeader, 0, ioBufHeader.length);
          out.write(ioBufHeader, 0, ioBufHeader.length);

          // write data size, bytes 41-44
          bb = ByteBuffer.allocate(4);
          bb.order(ByteOrder.LITTLE_ENDIAN);
          bb.putInt(fileSize - 44);
          bb.flip();
          out.write(bb.array(), 0, 4);

          bFoundHeader = true;
        }
        else if (bFoundHeader)
        {
          out.write(ioBuf, 0, bytesRead);
        }

        fileSize -= bytesRead;
      }
    } catch (FileNotFoundException f) {
      System.out.println(f.getLocalizedMessage());
    } finally {
      if (in != null) {
        in.close();
      }
      if (out != null) {
        out.flush();
        out.close();
      }
    }
    System.out.println(args[0]);
  }
}

Getting Started with the ZeroMQ (ØMQ) C++ Bindings

Why ØMQ?

At a previous job some friends were working on a team creating a satellite communication system for cube-sat swarms. They started with the Node.js platform running on all the satellites, and for their communication protocol they decided on ZeroMQ.

Recently I became curious about running a service that would use High Performance Computing to solve a geopotential calculation. At first I thought it might be interesting to create a native add-on for Node.js that would provid access to the GPGPU processing power of one of Amazon's compute instances. But writing a native add-on isn't a trivial task and it seems more valuable to learn ZeroMQ and have node communicate directly with a C++ ZeroMQ enabled server.

This post is only about the process of getting the ZeroMQ hello world application up and running.

Installing ZeroMQ on a Macbook

I'm going to use Apple's XCode (you must install XCode before proceeding further through this tutorial) to write my C++ code in, so I need to setup a project that references the zeromqlib. This requires installing a bunch of build tools. I'll be using homebrew for my instalation supporting libraries. The requirements for building from source are listed on the ZeroMQ doc page. Depending on your setup a lot of these libraries are already installed.

zeromq-4.1.0-rc1.tar.gz is the POSIX tarball I downloaded from http://zeromq.org/area:download.

$ brew update
$ brew install libtool --universal
$ brew update libtool
$ brew install pkg-config --universal
$ brew update pkg-config
$ brew install autoconf --universal
$ brew update autoconf
$ brew install automake --universal
$ brew update automake
$ tar -xvzf zeromq-4.1.0-rc1.tar.gz
$ cd zeromq-4.1.0
$ ./configure
$ make
$ make install

Side note: build-essential is specific to Ubuntu therefore and is therefore excluded from the above bash commands for my mac install.

You can do all the above installing and making from scratch OR you can run the infinitely easier brew install call:

$ brew install --universal zeromq

ZeroMQ C++ Bindings

Next you'll need to download the C++ binding file from github. Then you'll copy the binding header file zmq.hpp into the include directory for the zeromq library you just built. In my case I copied it into /usr/local/include. That was easy, right?

XCode Project Creation and Setup

Now we need to create the client and server "Hello World" XCode projects from the ZeroMQ tutorial. In XCode, create a new command line application project for both the client "Hello World" application and the server "Hello World" application.

create xcode command line project

Once you click next you'll name the projects hello-world-client and hello-world-server and from the languages dropdown you'll select C++. For the client application you'll delete the main.cpp file and copy in the hwclient.cpp from the zguide github. And for the server application you'll delete the main.cpp file and copy in the hwserver.cpp from the zguide github.

If you build now you will get a build error "Lexical or Preprocessor Issue 'zmq.hpp' file not found". In order to get rid of this we need to adjust our Header Search Paths to include the location of our zeromq C++ binding header file and our other zeromq library headers. Add /usr/local/include to the Header Search Paths in the Search Paths section of the Build Settings:
header search paths

Now if we build we'll probably get a lot of "Apple Mach-O Linker(id) Error" errors. Add -lzmq to the Other Linker Flags list in the Linking section of the Build Settings:
lzmq other linker flags

Now if we build we'll have one last "Apple Mach-O Linker(id) Error" error. This is because we haven't indicated the location of our zeromq libraries. Add /usr/local/lib to the Library Search Paths in the Search Paths section of the Build Settings:
Library Search Paths

Our build should be successful! After you've updated the Header Search Paths, the Other Linker Flags, and the Library Search Paths for both projects we should be building without any problems. Now if we run both projects we should have successful communication between the client and server!!!!

Successful communications!!

C++, Unit Tests, CxxTest and XCode

Cross Platform C++ Unit Testing

If you're curious about using an easy cross platform unit testing suite for C++, checkout my fork of CxxTest on github(LGPL2.1). Message me on Github if any of the instructions aren't clear.

Writing cross platform C++ is a liberating experience for anyone who has previously had their code tangled up in a particular operating system's helper functions. While working on ESRI's cross platform geometry library, our brilliant lead developer decided we should all be on board with test driven development. As a result he installed CppUnit as a part of our project solution.

After I left ESRI I found myself curious about keeping up my cross platform skills. So I started writing C++ libraries for iOS games with the idea that I would use JNI on Android to resuse my code (this was how ESRI's geometry library was to be used on their mobile platforms when I left). So having a cross platform unit testing solution made sense. I struggled to get CppUnit to work, and my impatience drove me to give CxxTest(LGPL3) a try.

The CxxTest landing page gives a nice summary of why someone might want to use it for testing:

CxxTest is easy to use because it does not require precompiling a CxxTest testing library, it employs no advanced features of C++ (e.g. RTTI) and it supports a very flexible form of test discovery.

So what's the catch? You need to have python installed on your testing machine in order for the cxxtestgen script to scrape relevant info out of you test header files and place it in the main cpp file. But other than python, it's so absurdly simple to setup CxxTest for unit testing that I've been committed to it for a few years now.

In the past I was using a messy pre-build shell script in XCode to execute cxxtestgen unit test prep. But as of yesterday I have fixed up my fork of CxxTest to use a python script as a prebuild step for preparing the unit tests. The same python script works for a Visual Studio setup. Checkout my CxxTest on github for instructions on how to setup your project to use CxxTest with XCode or Visual Studio.

That's all for CxxTest.

Now for a picture of Voltron. He was sort of like a cross platform robot, right? No? Well...whatever, Voltron is a badass. Picture of Voltron

Geometric Spatial Relationships

Topological Spatial Relationships

alt

Our thesis team at Hack Reactor have begun work on automated air traffic control software. It is an ambitious undertaking, and we would be doomed to underacheive if it wasn't for Postgres and the spatial relationship operators provided with the PostGIS extension.

While PostGIS provides some spatial relationships for 3D geometries, I'm only going to be covering the 2D (and 2.5D) relationships. The geometries covered by most 2D spatial relationship operators are points, polylines, and polygons (and maybe the 'multi' versions of those geometries). While the spatial relationships are 2D and only operate in the x-y domain, the geometries themselves can have z-values and those z's will be ingnored (although beware of vertical lines as those can be tricky).

What can't I do with spatial relationships

If a spatial relationship operator doesn't support testing geometries of certain type, then you may get unexpected behavior. You may think that you can test for 2 polygons for cross, but that relationship is better described by overlaps. If you use a crosses test on 2 polygons, then you may get a false when in fact the 2 polygons do overlap. Beware of false results from spatial operators on unsupported geometries.

  • test point cross point
  • test polygon cross polygon
  • test overlap between geometries of a different types
  • test touches between points
  • there are others I'm probably forgetting or that different libraries don't support

The operators typically available in a geometry library are intersects, disjoint, touches, overlaps, crosses, within and contain. Next Blog I'll focus on the intersect and disjoint operators.

Solving git Merge Conflicts with a GUI Tool

This image is huuuuuge!

Merging in git is almost as scary as merging in Dallas:Photo Credit: Dallas Observer

Wrecking a merge can feel as bad as breaking the build. The best way to avoid the process of a manual merge is to be proactive about commits and pull requests. But there will be times when you and another developer are in lightspeed-productivity-mode on the same file, and when it comes to spinning down the hyperdrives, one of you may be met with the heinous task of joining your histories together.

Integrating a GUI merge tool into your git workflow is a breeze. If you're a terminal pro you might flinch at the idea of working outside of your text editor, but I assure you it's more fun to get your merges done quickly and save your brain's computing power for more engaging programming problems. If you're using Github's fancy desktop client, you'll have to touch your terminal a little, but I swear it won't hurt. This tutorial is all Mac, but it applies for Windows as well.

In order to visualize the merge you'll need your GUI. If you've got money to burn($25) you can buy yourself the fancy Sublimerge. If you're frugal like me, you'll need to download the DiffMerge application from SourceGear (DiffMerge is Mac only, but similar softwares exist for Windows). For DiffMerge select the OS X 10.6+ Installer (Intel) package installer (not the DMG). The reason to not install from the DMG is that the installation won't do all the fancy symlinks that the git client will require in order to call DiffMerge from the terminal.

DiffMerge GUI experienceDiffMerge example

Sublimerge GUI experienceSublimerge example

After you've downloaded your GUI of choice with all it's fancy symlinks you're ready to prep your git to interact with GUI goodness. You'll need to add a few lines to your .gitconfig. If you're not a terminal lover I suggest editing your .gitconfig with nano by using the following command at your terminal: nano ~/.gitconfig.

If you're following the DiffMerge path you'll need to add the following lines to your .gitconfig:

[diff]
  tool = diffmerge
[merge]
  tool = diffmerge

[mergetool "diffmerge"]
  cmd = /usr/bin/diffmerge --merge --result=\"$MERGED\" \"$LOCAL\" \"$BASE\" \"$REMOTE\"
  trustExitCode = true

[difftool "diffmerge"]
  cmd = /usr/bin/diffmerge \"$LOCAL\" \"$REMOTE\"

If you're choosing the slick Sublimerge gui you'll need to add the following lines to your .gitconfig:

[diff]
  tool = sublimerge
[merge]
  tool = sublimerge

[mergetool "sublimerge"]
  cmd = subl -n --wait \"$REMOTE\" \"$BASE\" \"$LOCAL\" \"$MERGED\" --command \"sublimerge_diff_views\"
trustExitCode = true

[difftool "sublimerge"]
  cmd = subl -n --wait \"$REMOTE\" \"$LOCAL\" --command \"sublimerge_diff_views {\\\"left_read_only\\\": true, \\\"right_read_only\\\": true}\"

Using the above .gitconfig files means you'll be preserving originals as backups with a .orig extension type. Add *.orig to your project .gitignore to avoid the annoyance of backup files sneaking into your repository.

So now you're all set to merge conflicts. After you've rebased and you've received the dreaded Failed to merge in the changes you type in git mergetool at the terminal and you'll receive a prompt like below (to call git mergetool you must be in the directory of your git repository):

Merging:  
client/js/jadegrizzly.js  
client/templates/create.html

Normal merge conflict for 'client/js/jadegrizzly.js':  
  {local}: modified file
  {remote}: modified file
Hit return to start merge resolution tool (diffmerge):  

Hitting enter will pull up your awesome new merge tool and allow you to modify your files side by side (BIG WARNING!!! LOCAL and REMOTE in the GUI may be confusing identifiers. For the rebase scenario in DiffMerge the LOCAL side should represent the data that you've rebased to your local repository. That means the LOCAL side is the content you retrieved from master when you git pull --rebase upstream master).

Once you've saved and exited your gui merge tool, you need to take the next steps in git to complete your merge. But the hard part should be complete!

This post was completed as a part of an extreme blogging session, so it may contain errors or technical innaccuracies. Git with caution!!

Don't code like this guy bus drives.Photo Credit: Scoop Empire