Skip to the next section if you want to get straight to the good stuff. This is just backstory.
Anyone who knows me, knows that I am massively into Pokemon GO. I was into Pokemon as a kid during the late 90s when the gameboy games and trading cards were huge. If you're reading this and don't understand the new mobile game, you might want to read up on it otherwise the below might not make too much sense.
Back when Pokemon GO came out, there was a cool feature that let you "track" pokemon. A portion of the UI would show you how far away you were from the nearby pokemon, letting you track the rare pokemon that you don't have yet. This was later removed before I could get the chance to really use it. There is a lot of speculation as to why Niantic removed it. Some say it was to lessen server load, some say they wanted to implement something different. I don't know.
The removal of this feature sparked a bit of innovation in the community, and some people have made really cool web apps that made use of the Pokemon go backend API to discover pokemon spawns at any given location, essentially letting people see what pokemon were nearby and exactly where they were. This was (and still is) frowned upon and is actively being fought against by Niantic.
I don't doubt the ethical grey area that using these apps is in. But given how much of an advantage anyone near the Santa Monica Pier has over anyone else, I almost feel like using these maps is just getting me up to speed with anyone who has the luck to be able to play over there.
Anyway, I digress. The biggest and best map/scraper for Pokemon spawns out there (in my opinion) is FastPokeMap. I've used it a few times and had the idea of writing an automated service that would make use of their API to fetch Pokemon spawns at your current location and send you emails when something rare is nearby.
Using the FastPokeMap API
Just using Chrome Developer Tools, I checked out what kind of ajax requests were being made to fetch data from the FPM servers. Straight away I could see a ton of requests to https://api.fastpokemap.se/, all with the same query parameters showing the latitude and longitude for the request as well as what I can only assume is a string for filtering responses to only contain specific pokemon.
This struck me as weird at first as it was a lot of calls for the same data. However I could see the result of the call was a small bit of JSON representing an "overload" error. It seems that the FPM servers cant quite handle the traffic they're receiving. The web client just tries over and over with the server throwing back this JSON message until it finally works.
I tried for a while to recreate this API call in Postman, but for some reason couldn't get it just right so I opted to use the Chrome Developer Tools to copy the ajax request as a CURL command. This is what I had used in my scraper script. A bit of a hacky workaround, but it works.
As of late i've been writing code for small little projects in NodeJS since it's so simple to get small little proof of concepts up and running really quickly. In a short while I had the API call up and running, letting you supply your own GPS coordinates to it and getting back spawn locations in the area.
Getting current GPS coordinates
So i wanted this thing to grab my phones current GPS coordinates and send them over to FPM's API to get the nearby spawns. Rather than rolling my own android app, I looked around for something that could do what I wanted already. I found an app called Self-Hosted GPS Tracker (https://play.google.com/store/apps/details?id=fr.herverenault.selfhostedgpstracker&hl=en).
This was pretty cool, because it would let me use my own script, hosted wherever I wanted and it would just hit the URL with the latitude and longitude appended onto the end as query string parameters. So I wrote a quick PHP script that would dump the incoming GPS coordinates out as a JSON file that I could grab with the scraper script.
The only thing left was emails, which was pretty easy with the nodemailer package.
Wrapping things up
This was only really the beginning of what I wanted to do, the final version of the script sends all spawns that it finds to an external script, which saves the data into an SQL database so I could do some research on it later (which I plan to write about very soon).
At the moment, the script does the following:
- Specify a range of Pokemon IDs to send email notifications for (excluding the rest).
- Specify a URL to get GPS data from for spawn scraping.
- Specify stationary GPS coordinates to scrape spawns from.
- Specify a URL to post all scraped spawns to.
Below is a link to the Github repository for the project. All that's required to run it, is some changes in config.js, making sure CURL is accessible (via a path variable on windows, or just dropping the library in the folder) and an "npm install". Then all you need to do is run: