Why bother with HTTP?
There are a lot of reasons you might want to get two computers talking to each other, and HTTP is a good way to make this happen. It's used to share web pages of course (the name "hypertext transfer protocol" describes this use case directly) but it can be used to share all kinds of other data too—all varieties of text, image, audio, and video formats. Just imagine all the shenanigans you can orchestrate with this power.
In the world of computer networking getting computers talking the same language (a.k.a. protocol) is the first step. Some protocols are very niche, intended for tasks like reading oxygen sensors in combustion engines or controlling the environment in office buildings. In contrast, HTTP is a general purpose networking protocol. It pays the price of longer messages and a more complex standard in order to fill lots of different needs. It's a Swiss Army knife rather than a scalpel. It is said that Python is the second-best programming language for every task. The same could be said about HTTP for networking.
This post is an attempt to give some guide posts through the terrain, but the world of HTTP is wide and has a lot of history and hidden corners. Everything written here is oversimplified, and some of it is probably wrong, but I hope you find it useful.
Clients and servers,
requests and responses
When two computers talk using HTTP, one is always the client and the other is the server. The client initiates the conversation by sending a specific message with a specific format called a request. The server takes that request, does whatever it is that the server needs to do, and replies with another specific message format called a response. (If you're looking for a post on how to write an HTTP server, check here.)
Request messages have three parts
- a request to do something
- some headers
- a body, also called a payload
Response messages also have three parts
- a status report
- some headers
- a body (payload)
If you are writing a client, you'll need to take care of composing requests and interpreting responses. The opposite is true if you are writing a server.
Some Python
Here's an example that will showcase the building blocks. It's an HTTP client that calls a server that performs addition.
from http.client import HTTPConnection
import json
host = "192.168.1.42"
port = 8000
conn = HTTPConnection(host, port)
def call_addition_server(a, b):
body_dict = {"val_a": a, "val_b": b}
body = json.dumps(body_dict).encode('utf-8')
headers = {
'Content-Type': 'application/json',
'Content-Length': len(body),
}
conn.request("GET", "/", body, headers)
response = conn.getresponse()
if response.status == 200:
content_length = int(response.getheader('Content-Length'))
return json.loads(response.read(content_length).decode('utf-8'))
if __name__ == "__main__":
a = 25
b = 13
print(call_addition_server(a, b))
Starting from the imports
from http.client import HTTPConnection
the HTTPConnection
class from the
http.client library is immensely helpful for creating
requests and interpreting responses. In the client it's only
directly called when creating the connection to the server.
host = "192.168.1.20"
port = 8000
conn = HTTPConnection(host, port)
HTTPConnection
expects a host name, like en.wikipedia.org
or 192.168.0.1
and an optional port number,
although it will default to port 80, the default http port
of the internet, if no port is supplied. The server needs to be
running and reachable or else an attempt to connect with it will
generate a ConnectionRefusedError
.
Request body
HTTP connections can carry a bewildering array of data types, but if you have to choose just one, Python dictionaries are an excellent choice. They are good vehicles for carrying collections of input and output arguments.
The challenge with using dictionaries is getting them transformed into a format that can be transferred over the connection. There are several steps to the process
dict -> json -> string -> bytes
In code, this can be done in a one-liner:
body = json.dumps(body_dict).encode('utf-8')
Dictionary to JSON and JSON to string conversion is trivial,
thanks to the json
library. json.dumps()
is short for "take this Python dictionary, parse it and
convert it to JSON, then dump that JSON out to
a stringified version."
Gotchas:
- This is not the same as
str(body_dict)
. If you try this, you will get confusing errors. (Ask me how I know.) - Numpy arrays don't mix well with JSON.
They also give confusing errors.
If you want to include arrays in your dict,
first convert them to lists with
. You can convert them back on the other end with.tolist() np.array(
.- )
String to bytes conversion can be done with the
.encode('utf-8')
appended on the end of the line.
There are several ways to encode string data as bytes but
utf-8
has become far and away the most common, particularly in HTTP use cases.
Unless you have a good reason to do otherwise,
you're safe to use this throughout your clients and servers.
Request headers
In Python, we're used to playing fast and loose with data types and memory allocation, but in HTTP we can't get away with that any more. You can't even assume that you know what language the server on the other side of the connection is written in, let alone what types or message size it is expecting. As with all relationships, the answer to this challenge is good communication. HTTP gives clients a way to signal ahead what type of data they will be sending and how much of it. That way there are no misunderstandings.
Headers are the pre-message that give this meta-information. They are a short instruction manual, a packing manifest. In the example code above it includes two fields, one declaring the type of data and one declaring the number of bytes being transferred.
headers = {
'Content-Type': 'application/json',
'Content-Length': len(body),
}
Content-Type
and Content-Length
are
a nice miminal set of headers, but there are lots others.
The full list of headers shows just how varied the set
of instructions and information passed to the server can be.
Header field names are not case sensitive, so you may run
across a variety of capitalization conventions here.
Content type
JSON is a good multi-purpose message format, but except for the times when you are writing your own server, you'll have to provide a content type that the server expects.
Another extremely common content type is text/html
,
used for sharing web pages, arguably the main use case
for which the hypertext transfer protocol was developed.
In addition to these there are a lot of other types. So many. Literally hundreds of them. Looking at just some of the text formats, there are specific types for
text/css
text/csv
text/javascript
text/plain
text/xml
and besides these there are many others for varieties of image, audio, video, email message, and application content.
Luckily you don't have to guess. If you're writing a client for a server, you can look at the API specification for it to find out what content type it's expecting.
The action (The VERB)
Requests are also sent with
an action
attached, such as GET
or POST
,
clearly inspired by common operations in serving web pages.
GET can be interpreted very broadly as any operation where the client is hoping to get something back, like a web page, the output of a computation, the result of a database query, or the output of a model. But GET requests are not just one-way flows of information. They can be accompanied by hefty payloads like input arguments, feature values, and database queries.
Similarly POST operations are typically upload-heavy, but they can have generouts response payloads too.
Any public API worth its salt will have detailed documentation of which actions it accepts, what the payloads look like, and if you're really lucky, a few examples.
The path
The final piece of the puzzle is the path. The path argument gets top billing, its own reserved seat on the front row, rather than being tucked away in an argument in the body. This illustrates the original purpose of HTTP. With web serving, most requests are for an html file, and the path argument specifies which file that should be.
Whenever a request is intented to retrieve, upload, modify,
or otherwise touch a particular file, the path argument is the path
to that file, relative to the directory from which the server is running.
If I wanted to send a request to the webserver for
brandonrohrer.com
to retrieve the blog.html
file
which sits at the top level of the webserver's directory,
then the path argument would be /blog.html
.
If the server doesn't need path information, it's fine to leave
the path argument as a benign /
,
as in the code example above.
Again, a public API will usually document what the path should be for any given content, resource, or action.
Putting it all together
Now that the header, the body, the action, and the path are sorted,
they can be sent to the server using the
request()
function, which kicks off
the request processing
conn.request("GET", "/", body, headers)
and the response can be retrived with
getresponse()
response = conn.getresponse()
And this completes the journey of formulating and submitting the request. Huzzah!
Interpreting the response from the server
Unpacking the server response looks a lot like composing the request, in reverse. Responses come in the form of a HTTPResponse object which saves us the trouble of fiddly parsing and has some helpful fields and methods for pulling out the information we need: a status report, response headers, and the response body. These three parts all show up in the example code.
if response.status == 200:
content_length = int(response.getheader('Content-Length'))
return json.loads(response.read(content_length).decode('utf-8'))
Response status
Every HTTP response comes with a numerical code telling how it went. Status codes are 3 digits long with the first digit being 1, 2, 3, 4, or 5. From the Wikipedia page:
- 1xx informational response – the request was received, continuing process
- 2xx successful – the request was successfully received, understood, and accepted
- 3xx redirection – further action needs to be taken in order to complete the request
- 4xx client error – the request contains bad syntax or cannot be fulfilled
- 5xx server error – the server failed to fulfil an apparently valid request
Or abbreviated:
- 1xx: FYI
- 2xx: A-OK
- 3xx: We moved
- 4xx: You messed up
- 5xx: We messed up
and I cannot recommend strongly enough the HTTP status dogs who illustrate every status code for pup-loving visual learners.
Code 200 is the signal that everything went well, and the client code above checks for that before trying to process the response. A more robust client would check for other codes, especially 5xx codes, and have fallback strategies and informative error messages.
Response headers
Just like request headers, response headers carry
the information needed to handle the message. The list of common
response headers is extensive. Again Content-Type
and Content-Length
are particularly useful.
They tell what type of message body is expected and how large it is.
The getheader()
method pulls out the value
of a given header field. There is also a getheaders()
method which lists all the available headers.
This one is extra useful for exploring undocumented API behavior.
Reponse body
Not every response has a body, but if it does, the
read()
method is the way to pull those payload bytes
out and start working with them. read(content_length)
reads a fixed number of bytes and will wait, listening on the line
until the specified number of bytes have been received.
If you trust your sever to provide one, you can call read()
without a number of bytes and it will keep reading until it finds
an EOF
character.
Almost every time, the first step is to decode the content with
decode('utf-8')
to get a string. And then, if it happens
to be content type of application/json
you can
finish backing out the response content by calling
json.loads()
. This loads the string as a JSON object,
then converts that JSON to a dictionary. The content of
the dictionary (or whatever form the response happens to be)
depends entirely on the server.
A quick note on levels of abstraction
There are several different tools for interacting with HTTP servers in Python. They are at different levels of abstraction, that is, they make different assumptions about what the person writing the code is supposed to know; they're at different levels of down-in-the-weeds. Typically the higher the level of abstraction, the more assumptions are made behind the scenes and the fewer lines of code necessary to get the job done. Very roughly from highest to lowest level: I've chosen to run withhttp.client
here because I'm interested in
having (and giving) a lot of control and covering a lot of potential
networking use cases. Also, http.client
has a lot of symmetry with
http.server
, which will feature in a companion post.
This concludes the tour of a bare-bones HTTP Python client. This has only scratched the surface of a huge topic, but hopefully it will get you unstuck as you are coding up your first few clients in Python.
Stay tuned for the sequel, the tour of a bare-bones HTTP Python server.