{
"@context": "http://web.archive.org./web/20190226010534/http://schema.org",
"@graph": [{
"@type": "WebSite",
"name": "iRail Blog",
"url": "http://web.archive.org./web/20190226010534/http://hello.irail.be"
}, {
"@type": "WebPage",
"name": "iRail Blog",
"url": "http://web.archive.org./web/20190226010534/http://hello.irail.be"
}, {
"@context": "http://web.archive.org./web/20190226010534/http://schema.org",
"@type": "Organization",
"url": "http://web.archive.org./web/20190226010534/http://hello.irail.be",
"logo": "http://web.archive.org./web/20190226010534/http://hello.irail.be/wp-content/uploads/2017/03/cropped-apple-touch-icon-precomposed-192x192.png"
"sameAs": [
"http://web.archive.org./web/20190226010534/http://twitter.com/irail",
"http://web.archive.org./web/20190226010534/http://irail.be"
]
}]
}
iRail Blog – Promoting digital creativity concerning mobility in Belgium
window._wpemojiSettings = {"baseUrl":"http:\/\/web.archive.org.\/web\/20190226010534\/https:\/\/s.w.org\/images\/core\/emoji\/11\/72x72\/","ext":".png","svgUrl":"http:\/\/web.archive.org.\/web\/20190226010534\/https:\/\/s.w.org\/images\/core\/emoji\/11\/svg\/","svgExt":".svg","source":{"concatemoji":"http:\/\/web.archive.org.\/web\/20190226010534\/https:\/\/hello.irail.be\/wp-includes\/js\/wp-emoji-release.min.js?ver=4.9.9"}};
!function(a,b,c){function d(a,b){var c=String.fromCharCode;l.clearRect(0,0,k.width,k.height),l.fillText(c.apply(this,a),0,0);var d=k.toDataURL();l.clearRect(0,0,k.width,k.height),l.fillText(c.apply(this,b),0,0);var e=k.toDataURL();return d===e}function e(a){var b;if(!l||!l.fillText)return!1;switch(l.textBaseline="top",l.font="600 32px Arial",a){case"flag":return!(b=d([55356,56826,55356,56819],[55356,56826,8203,55356,56819]))&&(b=d([55356,57332,56128,56423,56128,56418,56128,56421,56128,56430,56128,56423,56128,56447],[55356,57332,8203,56128,56423,8203,56128,56418,8203,56128,56421,8203,56128,56430,8203,56128,56423,8203,56128,56447]),!b);case"emoji":return b=d([55358,56760,9792,65039],[55358,56760,8203,9792,65039]),!b}return!1}function f(a){var c=b.createElement("script");c.src=a,c.defer=c.type="text/javascript",b.getElementsByTagName("head")[0].appendChild(c)}var g,h,i,j,k=b.createElement("canvas"),l=k.getContext&&k.getContext("2d");for(j=Array("flag","emoji"),c.supports={everything:!0,everythingExceptFlag:!0},i=0;i
img.wp-smiley,
img.emoji {
display: inline !important;
border: none !important;
box-shadow: none !important;
height: 1em !important;
width: 1em !important;
margin: 0 .07em !important;
vertical-align: -0.1em !important;
background: none !important;
padding: 0 !important;
}
#site-navigation {background:##ffffff;}
a,
ul.sidebar li.widget_text a,
#content h2 a:hover,
ul.sidebar a:hover,
.comment-meta a:hover,
p.logged-in-as a:hover,
p.meta a:hover,
a.post-edit-link:hover,
#footer a:hover
{ color: #ba2a2a;}
body, #content h2 a { color: #000000; }
.broken_link, a.broken_link {
text-decoration: line-through;
}
.logo h1 a {display: none !important;}
var analyticsFileTypes = [''];
var analyticsSnippet = 'enabled';
var analyticsEventTracking = 'enabled';
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-37999668-3']);
_gaq.push(['_addDevId', 'i9k95']); // Google Analyticator App ID with Google
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'http://web.archive.org./web/20190226010534/https://ssl' : 'http://web.archive.org./web/20190226010534/http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
Toggle navigation
iRail
Routeplanner
Spitsgids
API documentation
iRail BlogPromoting digital creativity concerning mobility in Belgium
Search
One million daily requests: where do they come from, and how we’ll cope with them
Since improving iRail server performance back in October, the iRail servers have been serving everyone without any major issues. The servers handled around 300.000 requests every day, which left quite some margin. Back then, we estimated the server capacity at 1 million request per day.
Today is that day. Our server monitoring alerted us of increasing CPU usage, and we found the origin: we’re now serving over 900.000 requests every day. Due to the fast increase in the number of requests, where half of the requests are coming from only one IP address, we temporary started enforcing stricter rate limiting, meaning that instead of 5 requests per second, only 3 will be allowed. This shouldn’t affect you, but if it does, please get in touch with us to work out a solution.
In order to know where requests are coming from, and how this evolved over time, we analyzed log files from the past 74 days. After analyzing the 18gb of data, I decided to created an infographic. This way everyone can see the current state and how we evolved. It’s interactive, so you can enable or disable any client by clicking it in the graph legend.
!function(e,t,n,s){var i="InfogramEmbeds",o=e.getElementsByTagName(t)[0],d=/^http:/.test(e.location)?"http:":"https:";if(/^\/{2}/.test(s)&&(s=d+s),window[i]&&window[i].initialized)window[i].process&&window[i].process();else if(!e.getElementById(n)){var a=e.createElement(t);a.async=1,a.id=n,a.src=s,o.parentNode.insertBefore(a,o)}}(document,"script","infogram-async","http://web.archive.org./web/20190226010534/https://e.infogram.com/js/dist/embed-loader-min.js");
Infogram
Why clients use iRail
Most of the clients can be grouped into one of three categories:
Routeplanning applications for users
Digital signage, applications meant to be shown on TVs to inform people passing by of nearby public transport departures
Gathering information on delayed trains
The first one is a smaller category, which need an API like iRail. The second category also needs an API, but is more specific: they usually just need the departing trains for one or more nearby stations the next hour or two. The last category is different compared to the previous two: they need data which isn’t available through an API requests. In most cases they need to resort to loading the departures for all stations over and over again. This means they need a lot of requests (about 600) to update their information, which takes 2 minutes when obeying the old rate limiting system. Using the stricter rate limiting rules, this will take three to four minutes.
Linked Connections
It’s clear we need to do better. therefore, we can proudly announce that we’re currently beta testing Linked Connections, a brand new way of publishing transport data. Our new linked connections API will publish a list of all departing trains in Belgium, sorted by departure time.
What does this mean for you?
Offering users a routeplanning app? You will be able to implement your own route planning and support offline searches. As always, we’ll publish some open-source examples so you don’t have to figure it all out on your own. Does this sound scary to you? We’ll keep providing a traditional API.
Wanting to show the next departures? Just load the list for the time period you want to get the departures for, and filter out the departures at the station(s) you’re interested in. Again, we’ll provide some open-source examples and we’ll keep providing a traditional API.
Want to retrieve all delays? This category is the big winner. Just load the list for an hour or two, which is about 12 requests. That’s 50 times less requests!
The new API will serve the same list to everyone, meaning that with 1 million daily requests, you’ll have between 80% and 90% chance to hit a cache. Caching means we can serve more requests and we can server them faster – this also means rate limiting will be quite loose (around 10 requests per second, or more), so you can refresh your data often enough.
A few last words
Creating this infographic was only possible thanks to the user-agents included in the requests. However, still 5 of the top-8 clients don’t include a clear user agent. This means that we could not inform them of the new stricter rate limiting, or other updates to the iRail API (We only included the application name in the infographics above).
If possible, set the user-agent header string. Include the name of your application, and a way to contact you in case something would go wrong. An example user agent string format is <application name>/<application version> (<website>; <mail>)
, which could result in the following string: irail/1.2.0 (irail.be; hello@irail.be). The use of a user agent string like this isn’t obligated or enforced, but definitely allows for better communication.
Do you have a great concept?
If you have a great idea which needs railway data, you can instantly get started with our free to use API. You can find all information on our API at docs.irail.be. Want to analyze data yourself? We’re publishing API logs and occupancy logs at gtfs.irail.be.
Stay up to date by following us on twitter, joining our gitter channel, or make sure to follow this blog.
16. March 2018 by Bert
Categories: iRail |
Tags: api, irail, statistics |
2 comments
Open data for stations and accessibility
While the iRail API has been getting quite some updates, we remained silent about our open data on stations. Mostly because there wasn’t really anything to talk about. Today is different.
What we had before
Before today, we provided a list of stations, with their ID, translations, coordinates and ‘importance’ (e.g., Brussels West would be less important than Brussels South). While this enabled a lot of people to create nice things, it isn’t enough. In the past, we’ve received questions about station addresses, and at UGent, research is being done to plan routes keeping accessibility and disabilities of travelers in mind. We didn’t have this data, while it’s clearly valuable.
And what we offer now
We have created a scraper to scrape every station page there is, based on our existing stations.csv file. The results are stored in facilities.csv. The following data is now publicly available:
Address of the station
If ticket vending machines are present
If luggage lockers are present
If there is free parking, taxi, bicycle parking, taxi, bluebike, bus, tram, metro
If there are wheelchairs available,
If there is a ramp to roll wheelchairs on a train
If there are parking spots reserved for people with a disability
Whether or not the platform is elevated, if escalators are present up and/or down, if there is an elevator on the platform
If there is a hearing aid signal
Opening hours for ticket sales
All code and data is open source in the irail/stations repository.
Now go create something awesome with it!
22. November 2017 by Bert
Categories: iRail, NMBS, open data |
Tags: irail, open data, open source, public transport |
Leave a comment
More API data in less network data
Days are getting shorter, and iRail API responses are getting longer. We’ve added some more data to various API endpoints, so you can build even better applications. We’ve also improved support for HTTP caching, and are closing in on a release for LinkedConnections. Let’s check it out!
Route data
For a while, we weren’t able to provide the status for trains in a connection. We didn’t have that data, and we were at peace with it. But everything changed when the NMBS decided to delete a file from their site. Our status monitoring told us we were in trouble a minute or two later. It was the endpoint we were using to query connections, and now that it’s gone, we had to get our data somewhere else. With tens of thousands daily API requests, we’d also better fix it fast. I continued a previous investigation into the NMBS mobile app API. Quickly I was able to figure out how it worked, and we were able to get the API back online within 4 hours and 19 minutes. Within a few days the new code was mostly bug-free, and any bugs in the last weeks were quickly fixed, thanks to the help of some other developers who were happy to assist.
With this bad news also came some good news. While the old endpoint didn’t provide data about trains arriving and leaving the next station in your route, the new one does. Therefore, we are now able to provide you with the left and arrived status of trains in a connection. Make something beautiful with it.
Example usage of the left and arrived attributes in the open-source Hyperrail application
Example usage of the walking parameter
We also have alert data, pin-pointed to specific parts of the route. This way end-users can be even better informed about where in their journey they can come across some hinder. In order to demonstrate what’s possible with this new data, I’ve taken some screenshots using the Hyperrail for Android application which is already update to use these parameters.
Example display of alerts
Last but not least, there can now be walking parts in a route. This is rare, but in some cases it’s necessary, for example when travelers get off at Haren but need a next train at Haren-zuid, which is within walking distance, without any trains connecting those two stations.For the techies: a new attribute ‘isWalking’ has been added to every departure/arrival of a connections object. This way you can easily distinguish those routes. Just don’t try to parse a train or destination for those walking parts, but you can read all that in our documentation.
Vehicle data
I’ll keep it short here: You can now see if a stop is part of the ‘normal’ route of this train, or if it’s an extra stop which was added last minute by traffic control.
Log and feedback data
All iRail API log data, and spitsgids feedback, is now automatically published on gtfs.irail.be every morning at 3:00. This way everyone can instantly make use of the most recent spitsgids data. Feel free to scrape this site if you need data for multiple days. The formats are documented in the API documentation.
Conditional get requests
A 304 Not Modified response signals you can just keep using the last data you have
The API supports caching and conditional get requests. Conditional get requests allow you to tell the server which version of the data you have. If there is newer data available, the server will return an HTTP 200 status code along with the new data. If no new data is available, the server will return an HTTP 304 Not Modified status code, along with an empty body. We only support the if-none-match header for now, with more details in the docs.
What’s next
Having all these new fields doesn’t mean we’re done here. We’ve almost gotten LinkedConnections ready for production, and we’ll be opening up a lot of data about stations and their accessibility. Make sure to follow @iRail on twitter to stay up to date!
You can read all of this over in the documentation, view the iRail API code on github, or view my Hyperrail source code on github. View our data dumps at gtfs.irail.be.
10. November 2017 by Bert
Categories: iRail, NMBS,