Why bother with HTTP?

There are a lot of reasons you might want to get two computers talking to each other, and HTTP is a good way to make this happen. It's used to share web pages of course (the name "hypertext transfer protocol" describes this use case directly) but it can be used to share all kinds of other data too—all varieties of text, image, audio, and video formats. Just imagine all the shenanigans you can orchestrate with this power.

In the world of computer networking getting computers talking the same language (a.k.a. protocol) is the first step. Some protocols are very niche, intended for tasks like reading oxygen sensors in combustion engines or controlling the environment in office buildings. In contrast, HTTP is a general purpose networking protocol. It pays the price of longer messages and a more complex standard in order to fill lots of different needs. It's a Swiss Army knife rather than a scalpel. It is said that Python is the second-best programming language for every task. The same could be said about HTTP for networking.

This post is an attempt to give some guide posts through the terrain, but the world of HTTP is wide and has a lot of history and hidden corners. Everything written here is oversimplified, and some of it is probably wrong, but I hope you find it useful.

Clients and servers,
requests and responses

When two computers talk using HTTP, one is always the client and the other is the server. The client initiates the conversation by sending a specific message with a specific format called a request. The server takes that request, does whatever it is that the server needs to do, and replies with another specific message format called a response. (If you're looking for a post on how to write an HTTP server, check here.)

Request messages have three parts

  1. a request to do something
  2. some headers
  3. a body, also called a payload

Response messages also have three parts

  1. a status report
  2. some headers
  3. a body (payload)

If you are writing a client, you'll need to take care of composing requests and interpreting responses. The opposite is true if you are writing a server.

Some Python

Here's an example that will showcase the building blocks. It's an HTTP client that calls a server that performs addition.

from http.client import HTTPConnection
import json

host = "192.168.1.42"
port = 8000
conn = HTTPConnection(host, port)

def call_addition_server(a, b):
    body_dict = {"val_a": a, "val_b": b}
    body = json.dumps(body_dict).encode('utf-8')
    headers = {
        'Content-Type': 'application/json',
        'Content-Length': len(body),
    }
    conn.request("GET", "/", body, headers)

    response = conn.getresponse()
    if response.status == 200:
        content_length = int(response.getheader('Content-Length'))
        return json.loads(response.read(content_length).decode('utf-8'))


if __name__ == "__main__":
    a = 25
    b = 13
    print(call_addition_server(a, b))

Starting from the imports

from http.client import HTTPConnection

the HTTPConnection class from the http.client library is immensely helpful for creating requests and interpreting responses. In the client it's only directly called when creating the connection to the server.

host = "192.168.1.20"
port = 8000
conn = HTTPConnection(host, port)

HTTPConnection expects a host name, like en.wikipedia.org or 192.168.0.1 and an optional port number, although it will default to port 80, the default http port of the internet, if no port is supplied. The server needs to be running and reachable or else an attempt to connect with it will generate a ConnectionRefusedError.

Request body

HTTP connections can carry a bewildering array of data types, but if you have to choose just one, Python dictionaries are an excellent choice. They are good vehicles for carrying collections of input and output arguments.

The challenge with using dictionaries is getting them transformed into a format that can be transferred over the connection. There are several steps to the process

dict -> json -> string -> bytes

In code, this can be done in a one-liner:

body = json.dumps(body_dict).encode('utf-8')

Dictionary to JSON and JSON to string conversion is trivial, thanks to the json library. json.dumps() is short for "take this Python dictionary, parse it and convert it to JSON, then dump that JSON out to a stringified version."

Gotchas:

String to bytes conversion can be done with the .encode('utf-8') appended on the end of the line. There are several ways to encode string data as bytes but utf-8 has become far and away the most common, particularly in HTTP use cases. Unless you have a good reason to do otherwise, you're safe to use this throughout your clients and servers.

Request headers

In Python, we're used to playing fast and loose with data types and memory allocation, but in HTTP we can't get away with that any more. You can't even assume that you know what language the server on the other side of the connection is written in, let alone what types or message size it is expecting. As with all relationships, the answer to this challenge is good communication. HTTP gives clients a way to signal ahead what type of data they will be sending and how much of it. That way there are no misunderstandings.

Headers are the pre-message that give this meta-information. They are a short instruction manual, a packing manifest. In the example code above it includes two fields, one declaring the type of data and one declaring the number of bytes being transferred.

headers = {
    'Content-Type': 'application/json',
    'Content-Length': len(body),
}

Content-Type and Content-Length are a nice miminal set of headers, but there are lots others. The full list of headers shows just how varied the set of instructions and information passed to the server can be. Header field names are not case sensitive, so you may run across a variety of capitalization conventions here.

Content type

JSON is a good multi-purpose message format, but except for the times when you are writing your own server, you'll have to provide a content type that the server expects.

Another extremely common content type is text/html, used for sharing web pages, arguably the main use case for which the hypertext transfer protocol was developed.

In addition to these there are a lot of other types. So many. Literally hundreds of them. Looking at just some of the text formats, there are specific types for

and besides these there are many others for varieties of image, audio, video, email message, and application content.

Luckily you don't have to guess. If you're writing a client for a server, you can look at the API specification for it to find out what content type it's expecting.

The action (The VERB)

Requests are also sent with an action attached, such as GET or POST, clearly inspired by common operations in serving web pages.

GET can be interpreted very broadly as any operation where the client is hoping to get something back, like a web page, the output of a computation, the result of a database query, or the output of a model. But GET requests are not just one-way flows of information. They can be accompanied by hefty payloads like input arguments, feature values, and database queries.

Similarly POST operations are typically upload-heavy, but they can have generouts response payloads too.

Any public API worth its salt will have detailed documentation of which actions it accepts, what the payloads look like, and if you're really lucky, a few examples.

The path

The final piece of the puzzle is the path. The path argument gets top billing, its own reserved seat on the front row, rather than being tucked away in an argument in the body. This illustrates the original purpose of HTTP. With web serving, most requests are for an html file, and the path argument specifies which file that should be.

Whenever a request is intented to retrieve, upload, modify, or otherwise touch a particular file, the path argument is the path to that file, relative to the directory from which the server is running. If I wanted to send a request to the webserver for brandonrohrer.com to retrieve the blog.html file which sits at the top level of the webserver's directory, then the path argument would be /blog.html.

If the server doesn't need path information, it's fine to leave the path argument as a benign /, as in the code example above.

Again, a public API will usually document what the path should be for any given content, resource, or action.

Putting it all together

Now that the header, the body, the action, and the path are sorted, they can be sent to the server using the request() function, which kicks off the request processing

conn.request("GET", "/", body, headers)

and the response can be retrived with getresponse()

response = conn.getresponse()

And this completes the journey of formulating and submitting the request. Huzzah!

Interpreting the response from the server

Unpacking the server response looks a lot like composing the request, in reverse. Responses come in the form of a HTTPResponse object which saves us the trouble of fiddly parsing and has some helpful fields and methods for pulling out the information we need: a status report, response headers, and the response body. These three parts all show up in the example code.

if response.status == 200:
    content_length = int(response.getheader('Content-Length'))
    return json.loads(response.read(content_length).decode('utf-8'))

Response status

Every HTTP response comes with a numerical code telling how it went. Status codes are 3 digits long with the first digit being 1, 2, 3, 4, or 5. From the Wikipedia page:

Or abbreviated:

and I cannot recommend strongly enough the HTTP status dogs who illustrate every status code for pup-loving visual learners.

Code 200 is the signal that everything went well, and the client code above checks for that before trying to process the response. A more robust client would check for other codes, especially 5xx codes, and have fallback strategies and informative error messages.

Response headers

Just like request headers, response headers carry the information needed to handle the message. The list of common response headers is extensive. Again Content-Type and Content-Length are particularly useful. They tell what type of message body is expected and how large it is.

The getheader() method pulls out the value of a given header field. There is also a getheaders() method which lists all the available headers. This one is extra useful for exploring undocumented API behavior.

Reponse body

Not every response has a body, but if it does, the read() method is the way to pull those payload bytes out and start working with them. read(content_length) reads a fixed number of bytes and will wait, listening on the line until the specified number of bytes have been received. If you trust your sever to provide one, you can call read() without a number of bytes and it will keep reading until it finds an EOF character.

Almost every time, the first step is to decode the content with decode('utf-8') to get a string. And then, if it happens to be content type of application/json you can finish backing out the response content by calling json.loads(). This loads the string as a JSON object, then converts that JSON to a dictionary. The content of the dictionary (or whatever form the response happens to be) depends entirely on the server.

A quick note on levels of abstraction

There are several different tools for interacting with HTTP servers in Python. They are at different levels of abstraction, that is, they make different assumptions about what the person writing the code is supposed to know; they're at different levels of down-in-the-weeds. Typically the higher the level of abstraction, the more assumptions are made behind the scenes and the fewer lines of code necessary to get the job done. Very roughly from highest to lowest level: I've chosen to run with http.client here because I'm interested in having (and giving) a lot of control and covering a lot of potential networking use cases. Also, http.client has a lot of symmetry with http.server, which will feature in a companion post.

This concludes the tour of a bare-bones HTTP Python client. This has only scratched the surface of a huge topic, but hopefully it will get you unstuck as you are coding up your first few clients in Python.

Stay tuned for the sequel, the tour of a bare-bones HTTP Python server.