route3
Joined 16 years ago
Joined 16 years ago
I clearly remember when I finally let my initial prototype run on a laptop for a day and ignored it, an old HP laptop just chugging away parsing flight plan data, plugged in and running too hot in my office. By the next day I had > 20 diversions, with no notable weather, and assumed I had a bug in my logic.
Lots more to do here, including adding some weather details and someday predictive analytics. Thanks again for checking it out and the feedback.
The data processing is handled by two EC2's running Java to parse the FAA's flight plan feed. Peak midday that's running about ~65msgs/sec, a reduced and manageable number because I'm filtering for major airlines only (no GA, no bizjet, etc). The cleaned-up and filtered flight plan data heads to both Postgres for storage as well as SQS.
Consuming the SQS messages is a Python app that's crunching flight plans and confirming each flight is still heading to it's original destination (thanks redis!). If the destination airport changes, the flight gets flagged and some add'l business logic runs.
Ultimately you get the frontend (Django) presented here with a sprinkling of terrible JS to get the map and some timeline data.
I've been hacking with the FAA's SWIM data feeds for a few years now and hope to start sharing more of the technical details. Happy to answer any questions or hear some feedback!
The FAA offers a sample client. Depending on your level of skill, the hardest part might be the onboarding with the FAA which includes establishing a site-to-site VPN between your network and theirs. Once that is setup and some other networking is configured, you can use their Java demo client with the credentials they provide to get the hang of it.
The source of data is an event queue, powered by Solace, using Java's JMS standard. You can write a little bit of a Java to parse the incoming messages to get to the XML payload. Honestly I found that part to be the easiest and the Linux networking parts (as part of the VPN) to be the most difficult.
I wrote up a bit of a guide here, it's a little rough and high-level but I hope it helps clarify things a bit if you feel like working with these feeds/data.
http://www.route3software.com/faa-swim-getting-started-guide...
They do show taxi times for arriving/departing aircraft and I wonder if they are stitching that data together using the STDDS feed.
To give you an example, for an arriving aircraft you would get an alert of "wheels down", which is as literal as it sounds: a timestamped event when the aircraft made contact with the ground. In addition to ground position events (allowing the tracking you see here), you would also receive an alert when the aircraft left the tracked movement surface, i.e. transitioned to the gate/ramp area and no longer on an active airport surface.
(Pretty sure I have that correct, it's been awhile since I got back to my STDDS-powered projects. If anybody familiar with STDDS wants to chime in, please do!)
For the curious, all these "alerts" are JMS messages with XML payloads. Pretty much non-stop stream of data, incredibly fun if you're a developer+av geek. For FlightAware-style tracking without using any ADS-B you'll want to subscribe to the SFDPS feed. You'll get position updates every 60 sec for all airborne flights in the US, which last week amounted to ~40GB in my Postgres DB. You can request higher resolution reporting which is every 12s.
I'm using the STDDS feed to build a rough "go-around detector" in AWS for approaching aircraft by monitoring each approaching aircraft glide slope and dispatching a SNS notification "if currAlt > prevAlt".
If anyone has any questions about this stuff feel free to reach out, my email is in my profile. The easy part (IMHO) is slurping in the data which only requires a couple small EC2's and RDS, ~$125/mo. The hard part is often the presentation and making demos like these fun and shareable.
There's quite a bit to the SWIM program. The onboarding can be a bit tricky though and you have to know your way around site-to-site VPNs. I recently completed the onboarding process and am receiving live flight plan data -- proposed flights, en route, completed etc.
Would be curious to know if there are others on HN that are working with SWIM data or if there are companies that are hiring in this space.
YC has invested[0] in a company called InstallMonitizer[1] that appears to help developers and advertisers connect in the pay-per-install marketplace.
[0] http://www.techdirt.com/articles/20130115/17343321692/why-ar...
[1] http://www.installmonetizer.com/
Sadly it does't seem like a practice that will go away any time soon. I'd like to do some digging around on developer forums and see if any folks have shared their experience and would be able to comment on the amount of extra revenue that they see from such programs.
Migrations[0] are being included in Django 1.7 which is currently under development. For those unaware, Andrew Godwin, the author of South, ran a successful Kickstarter campaign[1] the fund this effort to incorporate migrations in Django core.
[0] https://docs.djangoproject.com/en/dev/topics/migrations/
[1] http://www.kickstarter.com/projects/andrewgodwin/schema-migr...
I wonder if someone has created a pricing matrix/table that updates and compares the SMS/voice API providers. For example, I know Plivo is slightly cheaper for SMS, domestically (USA). A price comparison table might make a good holiday project :)
Although I understand this wouldn't help your case. As you pointed out, the SMS provider would have already increased rates by the time you could be notified of increases.
paul at domain