Thomas Stiegler

Declarative Web Crawling and Data Extraction

2017-01-15T16:56:53-05:00

A few little projects I've been working on had involved ripping information from sites with multiple pages into a database or JSON file. I've done this kind of stuff before and it usually ended up in the same kind of code, something that iterates over a queue of URLs, pulls the data down and parses it into a data structure and sends it off to where it needs to go.

I made a little script for managing all of the logic common to scraping information from websites, intended to be reused over different projects whenever I needed it. In the process of doing that I got to learn some new stuff like publishing an npm package to npmjs and doing automated testing on git commit with TravisCI.

So the package I've made for this is httprip. The module exposes a factory method for creating a ripper instance. The instance allows you to attach implementations for "processing" and "yielding". Processor implementations allow you to define the parsing and extraction of html into the elements of data you want. These elements are to be "yielded" to the data collectors of the ripper instance. Multiple processors and data collectors can be specified. Check out the most basic example of usage below.

var httprip = require("httprip");

var ripper = httprip()
    .processor(function(error, res, body, resolve) {
        // Perform parsing on body here.

        // Yield each item parse from body.
        ripper.yield("item" + Math.floor(Math.random() * 999));
        ripper.yield("item" + Math.floor(Math.random() * 999));

        // Resolve after we've finished processing.
        resolve();
    })
    .data(function(output) {
        console.log("Retrieved item:", output);
    });


// Queue requests.
ripper.enqueueRip({url: "http://google.com"});
ripper.enqueueRip({url: "http://yahoo.com"});
ripper.enqueueRip({url: "http://bing.com"});

// Wait for finish.
ripper.lastQueued().then(function() {
    console.log("done");
})

So as you can see, this queues up three URLs to be ripped. The processor function disregards the output however and yields 2 random items for each request. The data collector method simply console logs the items. Also, a method is attached to the Promise of the last queued element so that some logic can be performed after all queued entries have been completed. There is a more detailed documentation over at the readme for the httprip project.

Creating NPM packages is easy

I was kind of expecting this whole process to be a lot more involved and complicated than it actually was. but it was seriously as easy as this:

Create an account at npmjs.copm
Open up a command prompt and navigate to your code's folder
Run npm init and follow the prompts
Run npm login and enter your username and password
Run npm publish

I mean really, that was it. Every time you want to update your package, just increment your version number in package.json and run npm publish again. NPM publish will automatically ignore anything in .gitignore so don't worry about all the bloat you might have in your package, it wont get pushed to npm.

Automated testing with TravisCI is easy too

I mean I only really did this to get that little "passing" badge on the github page, but it was pretty interesting to see how easy Travis is to set up. By connecting travis to a github account, it will already see all available projects. Setting it up for NodeJS is as simple as committing in a .tavis.yml file and activating the repository in Travis.

The .travis.yml file is just a config YAML file, which is how you let Travis know that you are testing a NodeJS project as well as other things like what Node version to use for testing.

language: node_js
node_js:
  - "6"

Not a hell of a lot. By default, Travis will run npm test, meaning your package.json file needs to have a test script. httprip just uses mocha, nothing special. Once Travis detects a push to github, it will grab the latest code and run the tests. Travis gives you a cool image link to get that "build passing" badge, provided your builds are passing.

Integer Indexable Spiral Algorithm

2017-01-10T15:01:11-05:00

This is a little algorithm I wrote for generating multiple coordinates in a pseudo-spiral fashion using a single incrementing integer index. This function, given a single integer index, will generate an X and Y coordinate that represents a position on a two-dimensional graph, moving around in a spiral as the given index increments. This might sound a little confusing at first, but consider the diagram below. The cell values represent the index given to the function and shows the resulting X and Y coordinates.

	-2	-1	0	1	2
3
2	24	9	10	11	12
1	23	8	1	2	13
0	22	7	0	3	14
-1	21	6	5	4	15
-2	20	19	18	17	16
-3

As you can see the following indexes match with the given coordinates.

0 -> {x: 0, y: 0}
1 -> {x: 0, y: 1}
2 -> {x: 1, y: 1}
3 -> {x: 1, y: 0}
4 -> {x: 1, y: -1}
5 -> {x: 0, y: -1}
6 -> {x: -1, y: -1}
7 -> {x: -1, y: 0}
8 -> {x: -1, y: 1}
9 -> {x: -1, y: 2}

Code

/**
 * Convert spiral location idex to x/y coordinates.
 * 
 * @param {number} idx  Spiral location index.
 * @param {coordinates} origin X/Y origin to start spiral at.
 * 
 * @returns {coordinates} Spiral location coordinates.
 */
function spiralIndexToCoord(idx, origin) {
    origin = origin || {x: 0, y: 0};

    var swSteps = 1;
    var offset = {
        x: 0,
        y: 0
    };

    while(idx > 0) {
        // Calculate number of steps to navigate southwest 1 block.
        var stepsEndpoint = swSteps + 1;
        var movements = ((stepsEndpoint - 1) * 2) + stepsEndpoint * 2;        

        // If we have more indexes left to cover the southwest movement,
        // move the offset.
        if(movements <= idx) {
            idx -= movements;
            
            offset.x--;
            offset.y--;
        } else {        
            // If not, then we need to move it manually to hit the destination.
            // For the given directions N to W, 0 to 3, list out the amount of steps needed.
            var direction = 0;
            var stepArr = [
                swSteps,
                swSteps,
                swSteps + 1,
                swSteps + 1
            ];
            
            // Loop for the remainder of idx.
            while(idx > 0) {
                // Decrement the direction step.
                stepArr[direction]--;

                // Move in the given direction.
                if(direction == 0) { offset.y++; }
                if(direction == 1) { offset.x++; } 
                if(direction == 2) { offset.y--; }
                if(direction == 3) { offset.x--; }

                // Move to next direction if this directions steps have run out.
                if(stepArr[direction] == 0)
                    direction++;

                idx--;
            }
        }

        // Move to next sw count for next iteration.
        swSteps += 2;
    }

    return {
        x: origin.x + offset.x,
        y: origin.y + offset.y
    };
}

Origin parameter

By default, the 0th index of the spiral will start at (0, 0). However you manually set the origin of the spiral through the origin parameter.

spiralIndexToCoord(0, {x: 5, y: 12});

Performance Issues

Due to the nature of the algorithm, higher indexes take longer time to calculate than lower indexes. I have tested the performance hit for generating the first million indexes as well as Number.MAX_SAFE_INTEGER (Which in NodeJS is 9,007,199,254,740,991).

The testing script implementation and result is as follows:

// Show the first ten indexes.
for(var i = 0; i < 10; i++) {
    console.log("Index " + i + ":", spiralIndexToCoord(i, {x: 0, y: 0}))
}

// Measure time taken for 1 million iterations of spiral index to coordinate conversions.
console.time("First million indexes");
for(var i = 0; i < 1000000; i++) {
    var coords = spiralIndexToCoord(i, {x: 0, y: 0});
}
console.timeEnd("First million indexes");

// Measure time taken for a large index.
console.time("MAX_SAFE_INTEGER");
console.log("Index " + Number.MAX_SAFE_INTEGER + ":", spiralIndexToCoord(Number.MAX_SAFE_INTEGER, {x: 0, y: 0}));
console.timeEnd("MAX_SAFE_INTEGER");

Index 0: { x: 0, y: 0 }
Index 1: { x: 0, y: 1 }
Index 2: { x: 1, y: 1 }
Index 3: { x: 1, y: 0 }
Index 4: { x: 1, y: -1 }
Index 5: { x: 0, y: -1 }
Index 6: { x: -1, y: -1 }
Index 7: { x: -1, y: 0 }
Index 8: { x: -1, y: 1 }
Index 9: { x: -1, y: 2 }
First million indexes: 6925ms
Index 9007199254740991: { x: 47453133, y: 23868632 }
MAX_SAFE_INTEGER: 1886ms

FastPokeMap API scraping - Spawn notifications by Email

2016-09-18T22:37:35-04:00

(Edit: FPM has long since been dead, was awesome while it worked!)

Skip to the next section if you want to get straight to the good stuff. This is just backstory.

Anyone who knows me, knows that I am massively into Pokemon GO. I was into Pokemon as a kid during the late 90s when the gameboy games and trading cards were huge. If you're reading this and don't understand the new mobile game, you might want to read up on it otherwise the below might not make too much sense.

Back when Pokemon GO came out, there was a cool feature that let you "track" pokemon. A portion of the UI would show you how far away you were from the nearby pokemon, letting you track the rare pokemon that you don't have yet. This was later removed before I could get the chance to really use it. There is a lot of speculation as to why Niantic removed it. Some say it was to lessen server load, some say they wanted to implement something different. I don't know.

The removal of this feature sparked a bit of innovation in the community, and some people have made really cool web apps that made use of the Pokemon go backend API to discover pokemon spawns at any given location, essentially letting people see what pokemon were nearby and exactly where they were. This was (and still is) frowned upon and is actively being fought against by Niantic.

I don't doubt the ethical grey area that using these apps is in. But given how much of an advantage anyone near the Santa Monica Pier has over anyone else, I almost feel like using these maps is just getting me up to speed with anyone who has the luck to be able to play over there.

Anyway, I digress. The biggest and best map/scraper for Pokemon spawns out there (in my opinion) is FastPokeMap. I've used it a few times and had the idea of writing an automated service that would make use of their API to fetch Pokemon spawns at your current location and send you emails when something rare is nearby.

Using the FastPokeMap API

Just using Chrome Developer Tools, I checked out what kind of ajax requests were being made to fetch data from the FPM servers. Straight away I could see a ton of requests to https://api.fastpokemap.se/, all with the same query parameters showing the latitude and longitude for the request as well as what I can only assume is a string for filtering responses to only contain specific pokemon.

This struck me as weird at first as it was a lot of calls for the same data. However I could see the result of the call was a small bit of JSON representing an "overload" error. It seems that the FPM servers cant quite handle the traffic they're receiving. The web client just tries over and over with the server throwing back this JSON message until it finally works.

I tried for a while to recreate this API call in Postman, but for some reason couldn't get it just right so I opted to use the Chrome Developer Tools to copy the ajax request as a CURL command. This is what I had used in my scraper script. A bit of a hacky workaround, but it works.

As of late i've been writing code for small little projects in NodeJS since it's so simple to get small little proof of concepts up and running really quickly. In a short while I had the API call up and running, letting you supply your own GPS coordinates to it and getting back spawn locations in the area.

Getting current GPS coordinates

So i wanted this thing to grab my phones current GPS coordinates and send them over to FPM's API to get the nearby spawns. Rather than rolling my own android app, I looked around for something that could do what I wanted already. I found an app called Self-Hosted GPS Tracker (Google Play Link).

This was pretty cool, because it would let me use my own script, hosted wherever I wanted and it would just hit the URL with the latitude and longitude appended onto the end as query string parameters. So I wrote a quick PHP script that would dump the incoming GPS coordinates out as a JSON file that I could grab with the scraper script.

The only thing left was emails, which was pretty easy with the nodemailer package.

Wrapping things up

This was only really the beginning of what I wanted to do, the final version of the script sends all spawns that it finds to an external script, which saves the data into an SQL database so I could do some research on it later (which I plan to write about very soon).

At the moment, the script does the following:

Specify a range of Pokemon IDs to send email notifications for (excluding the rest).
Specify a URL to get GPS data from for spawn scraping.
Specify stationary GPS coordinates to scrape spawns from.
Specify a URL to post all scraped spawns to.

Github Project

Below is a link to the Github repository for the project. All that's required to run it, is some changes in config.js, making sure CURL is accessible (via a path variable on windows, or just dropping the library in the folder) and an "npm install". Then all you need to do is run:

node main.js

Setting up AngularJS in Visual Studio 2013 with TypeScript

2016-01-29T18:30:54-05:00

EDIT (18th September 2016): The code samples for this post have ceased to exist. This post probably won't be helpful to anyone.

In this post I'm going to show you the way that I've come to develop AngularJS applications in Visual Studio in conjunction with TypeScript and ASP.NET MVC. When I searched around for how to do this, I wanted a way to have VS automatically stitch together all of the javascript without having to use something like RequireJS. Which as it turns out, is not hard to do. This meant having my TypeScript files all split out logically into different folders for controllers, directives, services and filters. As well as having one file per controller and etc (While not necessary, eases the part of me that likes to have a place for everything and have everything in its place).

So first off, what you're going to need to do is create a new ASP.NET MVC project using the "Basic" template. Create a HomeController and view for the Index method and make sure that all works nicely.

Using NuGet, go ahead and grab the AngularJS and angularjs.TypeScript.DefinitelyTyped packages. This will dump a bunch of scripts into the Scripts folder, not all of which are necessary... I just tend to add them in whenever I need them. Add in the AngularJS reference to BundleConfig.cs as needed and get it included in _Layout.html. Add the ng-app directive anywhere you want the AngularJS application to be scoped to. For me, I just use the root HTML tag.

In your Scripts folder, create a new TypeScript file. Name it whatever you want, this will be where AngularJS gets initiated from. For this tutorial i'll be using the name AngularDemo, so I'll be creating AngularDemo.ts. You'll need to add in the reference comment to load in the AngularJS typings and then go about creating the Angular application instance.

So at this point you really don't have much. Basic MVC, AngularJS being included and ... Well that's about it. This next step will get VS ready to stitch together all your TypeScript files so our entire application runs under a single JavaScript file despite having our TypeScript logically split up into different files.

Go to your project properties page (Project > "ProjectName" Properties...) and load up the "TypeScript Build" section. Under "Output", check "Combine JavaScript output into file", set the textbox to "Scripts/AngularDemo.js" (Or whatever you want... I'm not picky) and hit Ctrl+S to save it.

Obviously you wont see the effects of this so far since we only have the one TypeScript file, so lets add in a new TypeScript file for a basic controller. Like everything else, you can add these wherever you want, but I'll be using "Scripts/ng/controllers/TestController.ts". Using the code below, I've created a basic controller. The $scope gets passed in as a ITestControllerScope which extends ng.IScope, this allows our scopes to be type safe and fully defined.

Now due to the TypeScript build settings we added previously, the AngularDemo.js file now contains the JavaScript output for both the main TypeScript file as well as the controller we've created. All of our TypeScript is being automatically concatenated into a single output file. Just add in some HTML to make use of the controller and now you can see everything coming to life in the browser.

That's really all there is to the actual setup of AngularJS in Visual Studio using TypeScript. This is really just scratching the surface of what is required to build full applications with these technologies. In the next post I'll be writing about AngularJS routing and serving templates for AngularJS to use via ASP.NET MVC.