HTTP Tutorial for Mobile Developers – Part 3

So far in this tutorial series, we’ve imitated a basic web client, and then mimicked a barebones web server. This was all done from command line tools, but it’s time to write some code. We’re going to write a web server using Node.js. If you haven’t heard of it before, Node.js is Javascript, run server-side. Visit nodejs.org to download the installer.

With node installed, open your favourite text editor and paste in the following:

var http = require("http");
var server = http.createServer( handleRequest );
server.listen( 8765, "127.0.0.1" );
console.log( "server started" );
function handleRequest(req, res)
{
    res.writeHead( 200, { "Content-Type":"text/plain"} )
    res.end( "Richard Penner is really knocking it out of the park with this tutorial series.\n" );
}

Save that file as ‘index.js’, and run it from terminal by issuing:

node index.js


You should now be able to hit your server in your web browser at http://localhost:8765. While I won’t make this a Node.js tutorial, let’s walk through this line by line.

var http = require("http");

Here we’re telling node to include the HTTP library, or “package” in node parlance. The HTTP library let’s us easily create, well, an HTTP server, which we do in the next line:

var server = http.createServer( handleRequest );

Note that we pass in a function as an argument. Node.js is very fast, single-threaded, and non-blocking. Non-blocking means we’re going to be passing in callback functions anytime we’re doing something potentially slow. Handling a request might be slow: if our first user uploads a large file, we don’t want the second user waiting on that file upload to complete before we serve them. So we pass in a callback which will be invoked whenever a request comes in to be served. The next line is easy:

server.listen( 8765, "127.0.0.1" );

We’re going to listen on port 8765 of our local IP address. Continuing on, we do some logging to the console, and then we get to the good stuff:

function handleRequest(req, res)
{
    res.writeHead( 200, { "Content-Type":"text/plain"} );
    res.end( "Richard Penner is really knocking it out of the park with this tutorial series.\n" );
}

This is our callback function, which is invoked by our server object when we receive a request. It’s passed two things: the incoming request (req), and the outgoing response (res). Remember those HTTP headers we saw the browser sending along, identifying the browser and such? You can peel those out of the req.headers array.

We start by writing the HTTP status code: 200, which you’ll recall means “OK”, as in, this request worked. If req.url was something invalid like “/your-non-existent-page/”, we may write a 404 status code instead.

We also specify a “Content-Type” header of “text/plain”, letting the client know what format we’re returning our response in. Finally, we call the response object’s end() function which lets us pass in a string of data to send back to the client, in this case, another generous complement – you’re really too much.

So we now have a web server running locally. But it’s a very simple server. In part 4, we continue our look at Node.js by beefing up our server to reflect a little closer what a real world API might look like.

HTTP Tutorial for Mobile Developers – Part 2

In the first part of this series, we learned a bit about networking and HTTP communication. We pretended to be a web browser, and spoke to some web servers from the command line.

We’re now going to view the conversation from the other side, and pretend to be a web server. To do this, we’re going to use a linux command called netcat. For the uninitiated, netcat is the swiss army knife of networking. We want to setup a server, which means we’ll open up a port and listen for incoming connections. This is done with a simple netcat command:

nc -l 8765

We’ve told netcat to start listening on port 8765. As you know, the default web port is 80: typing http://google.ca is the same thing as typing http://google.ca:80. We’re using a non-standard port, so open a new tab in your browser and type in

http://localhost:8765

After you press enter, take a look at your console output. I used the latest version of Safari, but any browser should work:

GET / HTTP/1.1
Host: localhost:8765
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: keep-alive


This should look familiar by now. The browser here has sent the same command we saw using curl in part 1: GET / HTTP/1.1, or in plain-english: use the GET HTTP method to retrieve the root document, and speak using HTTP 1.1. What follows are some HTTP headers sent by the browser to let our server know what it’s capable of. For example, we can see the language is set to ‘en-us’, and that our browser allows large web pages to be compressed with gzip.

You’ll notice that your browser is still not finished loading the page, and your console command is still running. Well, like I said, we’re pretending to be a web server, so let’s serve some content. Go ahead and type in anything you like, making sure to hit enter twice afterward:

this is the best tutorial i've ever read in my life

Well that’s quite generous of you, but that’s not valid HTML. No matter, when you’re finished paying me your respects, hit Ctrl-C to kill netcat and close the connection. Now if you look at your browser, you should see your message.

So far in the tutorial, we’ve pretended to be both a web client and a web server. What does this have to do with mobile, and why haven’t I mentioned anything about APIs yet? Well, most modern APIs today communicate using these same HTTP commands used by your web browser talking to web servers. And as a mobile developer, sooner or later you’ll need to talk to these APIs.

In part 3, we take what we’ve learned so far and implement a real HTTP API server using Node.js.

HTTP Tutorial for Mobile Developers – Part 1

This post is the first in a series of articles intended to teach some networking and HTTP basics to mobile developers. While any developer looking to learn more about REST APIs or HTTP can benefit from reading this, the focus will be on consuming HTTP APIs from an iOS app.

To begin with, let’s go over some brief terms you are hopefully quite familiar with:

• IP Address: A numerical address identifying you on a network. If you are reading this on your phone over a cellular network for example, your IP address is public and uniquely identifies your phone on the internet. Reading this from home, you may have a cable modem with a public IP address, and a router which maps private IP addresses from your devices through the public IP address of your cable modem. IP addresses are of the form aaa.bbb.ccc.ddd where each set is an 8 bit number (0-255). Out of a possible 2^32 IP addresses available, some IP addresses are designated as private, for use on an internal network (behind your cable modem). If 2^32 (4,294,967,296) doesn’t sound like a lot of IP addresses, you’re right; we’re running out of them. Thankfully, a more modern IP address format called IPv6, has gained adoption and some of your devices may already be using them.
• Port: If you imagine an IP address as a street address, then every house on that street can receive letters in a number of different mailboxes. 65,536 different mailboxes, to be exact. Every web page you download, every email you receive, all network connections go to and from one of these ports. Web browsers know to communicate on port 80. If you visit an SSL-encrypted web page, your browser knows to hit 443.
• HTTP methods: When we pull down a web page, our browser is issuing an HTTP GET request along with a specific URL. This is telling the server that we want to receive the contents of whatever is at that location. For example, if you’re browsing the New York Times, your browser likely issued a GET request for “/”, where “/” is the root document. Click a link and your browser issues a GET for /pages/travel/index.html, for example. The verbs you’ll concern yourself with when consuming an HTTP API are GET, POST, DEL, and PUT. While the implementation of each of these verbs is up to the API developer, these methods typically map to CRUD (create, read, update delete) of server resources. We’ll dig more into that later on.
• HTTP status code: HTTP status codes are a way for the server to tell our browser whether things are working as we intended. For example, if we request the root document “/” and receive an HTTP status code of 200 (which represents “OK”), our browser knows everything worked fine. A status code of 301 means the web page has moved permanently, and you should update your link or bookmark.
• REST: Representational State Transfer. In short, this is how the web is organized. The resource you want to consume, say a web page, lives at a certain address, like http://example.com/webpage.html. You use the HTTP methods listed above to interact with those resources.

OK, so we know about IP addresses, ports, and a tiny bit about HTTP. Now what? Let’s pretend to be a web browser. This will give us some insight into how browsers work, and more importantly, what HTTP communication looks like. We’re going to use cURL, which is an HTTP command-line utility usually bundled with linux and Mac OS/X and available for most other platforms. Open a Terminal window and type the following:

curl --verbose http://google.ca

You may want to substitute google.com if you’re reading this from the U.S. Let’s walk through the output:

* About to connect() to google.ca port 80 (#0)
* Trying 173.194.43.120...
* connected
* Connected to google.ca (173.194.43.120) port 80 (#0)

In the first line we can see that curl is attempting to hit google.ca on port 80, which we know is the web port. The second line shows the IP address that google.ca maps to. Alright, one additional note about curl: output denoted with “>” means we’re sending it, and “<” means we’ve received it. Continuing through the output:

> GET / HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: google.ca
> Accept: */*


Here we can see that curl issued the HTTP GET method, and is requesting the root document, “/”. We’re communicating with HTTP 1.1, which is better than 1.0 in a few ways, one being that we can keep our network connection open when requesting several items. Next, curl sends some HTTP headers. Firstly, curl is telling google what our User-Agent string is. This is not required but is a nice way of telling the server a bit about us, in case the server wants to customize it’s output (send us a mobile-friendly page, etc.). Following that is the Host header, which as part of HTTP 1.1 specifies which host (server) we want the document from; keep in mind that a web page may contain links to images and scripts from many different hosts. Finally, the Accept header informs the server what kind of documents we are willing to receive. Here we tell the server that we’ll accept anything (*/*). Next:

< HTTP/1.1 301 Moved Permanently
< Location: http://www.google.ca/
…

For the scope of this tutorial I’ve trimmed out some of the response, but let’s note the 301 status code. You’ll remember that 301 means this document was moved permanently. In the next line, we see it’s new location: http://www.google.ca. Alright, so when our browser hits http://google.ca, the google server tells us to go grab http://www.google.ca instead. The response ends with some HTML for the browser to display:

<HTML><HEAD><meta http-equiv=”content-type” content=”text/html;charset=utf-8″>
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF=”http://www.google.ca/”>here</A&gt;.
</BODY></HTML>

* Connection #0 to host google.ca left intact
* Closing connection #0


Note that you won’t see this HTML in most browsers; your browser follows that 301’s location automatically. Alright, so google told us to add the www, let’s try the new location:

curl --verbose http://www.google.ca

The output begins with the same GET request and HTTP headers passed along, but this time, success:

< HTTP/1.1 200 OK
< Date: Wed, 17 Apr 2013 18:43:38 GMT
< Set-Cookie: PREF=ID=14d0f143b176675f:FF=0:TM=1366224218:LM=1366224218:S=FctofHxDhvyJlJBS; expires=Fri, 17-Apr-2015 18:43:38 GMT; path=/; domain=.google.ca
< Set-Cookie: NID=67=jHYpBlAGKIEMsPyfnCJ3Z-0McLdcjVZPrtHOPnKPKCt_Zqcqq5Cl1Q30R6nNv5QkLcS2qzeVmJNxkRVu3u1_Ob1fmPGK3fMdLuCNC3tWsQBwfdRnRtVxFTb1KFOr2L0G; expires=Thu, 17-Oct-2013 18:43:38 GMT; path=/; domain=.google.ca; HttpOnly
…


And what follows is the HTML source of Google’s home page.

We now know the basics of HTTP communication, and how to talk to a web server. In the next part of this tutorial, we pretend to be a web server and view the HTTP conversation from the other side.