Designing better Python interfaces with keyword-only arguments
Keyword-only arguments are one of my favorite features introduced in Python 3. But while they’ve been available since Python 3 was released in 2008, they still seem rarely used today. Probably because their true value is not always fully understood. This is hardly surprising though, since the only rationale provided in PEP 3102, which introduces the feature, is to offer more flexibility in the ordering of the arguments. However, keyword-only arguments can help you design better interfaces.
Positional and Keyword arguments
Let’s start by setting the scene and have a look at the two kinds of arguments Python supports in function signatures: positional and keyword arguments. We’ll discuss them using the copytree example, which we’ll develop throughout this blogpost. copytree recursively copies an entire directory tree from the src to the dest directory .
def copytree(src, dest):
...
We can call this function with positional arguments, which are assigned implicitly based on their position.
copytree(‘foo/’, ‘bar/’)
Or we can call it using keyword arguments, which are explicitly assigned based to their respective keywords. The order of keyword arguments does not matter and can be changed freely.
copytree(src=‘foo/’, dest=‘bar/)
Both of these function calls are equivalent and the caller can decide which version to use. Using positional arguments seems the most natural here, as keywords can feel superfluous when the meaning of the arguments can be derived from their position. When adding additional arguments however, the situation changes and using positional arguments can quickly become unclear.
def copytree(src, dest, dirs_exist_ok, symlinks):
...
copytree(‘foo/’, ‘bar/’, True, False)
Without looking at the function definition, it’s impossible to tell what the True and False arguments are doing here. A better way to call the function would be:
copytree(‘foo/’, ‘bar/’, dirs_exist_ok=True, symlinks=False)
Anyone who looks at this function call can immediately tell what it’s doing without having to look at the function definition.
Each caller is still free to call this function with positional arguments though, which can lead to unclarities and inconsistencies across the codebase. Keyword-only arguments can help enforce the proper usage of your functions.
Keyword-only arguments
To understand the syntax of keyword-only arguments, we’ll have to take a small detour explaining ‘varargs’ arguments first.
def f(x, y, *args):
...
In this function signature, *args is called a varargs argument. It swallows all positional arguments provided on top of x and y, allowing for a dynamic number of arguments. This is easily understood when looking at the following example call and the resulting values for each argument.
f(1, 2, 3, 4, 5)
x = 1, y = 2, args = [3, 4, 5]
Since Python 3, we can define additional arguments after the varargs argument. Since the varargs argument swallows all positional arguments, it’s impossible to fill these additional arguments with positional values.
def f(x, y, *args, flag):
...
In this case, the flag argument therefore needs to be provided by keyword. It is a keyword-only argument.
We might not want to use a varargs argument though. If our function only expects a limited amount of arguments, a varargs argument can swallow any mistakenly provided arguments, which might lead to unexpected function behavior. Luckily, we can keep the keyword-only behavior while dropping the varargs argument.
def f(x, y, *, flag):
...
This function will only take two positional arguments and requires the flag argument to be provided by keyword. Applying this to our previous example leads to the following function definition.
def copytree(src, dest, *, dirs_exist_ok, symlinks):
...
Which enforces callers to provide the dirs_exist_ok and symlinks arguments by keyword.
copytree(‘foo/’, ‘bar/’, dirs_exist_ok=True, symlinks=False)
Satisfying, isn’t it?
Bonus: Positional-only arguments
Using keyword-only arguments prevents us from calling functions with all positional arguments.
copytree(‘foo/’, ‘bar/’, True, False)
But it still allows calling the function with all keyword arguments.
copytree(src=‘foo/’, dest=‘bar/’, dirs_exist_ok=True,
symlinks=False)
While this is less of a problem, I would still argue that this is less clear than a combination of both positional and keyword arguments. Positional arguments implicitly provide additional clarity on how the function behaves. They rely on the natural logic of order, which can be understood at a glance. Just look at this example where we reorder the keyword arguments… and weep.
copytree(dest=‘bar/’, src=‘foo/’, dirs_exist_ok=True,
symlinks=False)
On top of the reordering problem, the src and dest argument names can now impossibly be changed without breaking the calling code, even though they were never intended to be called by name.
For example, let’s say we want to rename the dest argument to dst to be more consistent with our src argument name. Making this change will break any calling code that specifies dest with a keyword argument.
def copytree(src, dst, *, dirs_exist_ok, symlinks):
...
copytree(src=‘foo/’, dest=‘bar/’, dirs_exist_ok=True,
symlinks=False)
TypeError: copytree() got an unexpected keyword argument 'dest'
This can especially impose a problem for public interfaces, in libraries for instance. That’s why PEP 570 introduced positional-only arguments, whose behavior is equivalent but opposite to keyword-only arguments.
def copytree(src, dest, /, *, dirs_exist_ok, symlinks):
...
Here all arguments left of the / are positional-only arguments. This completes our search for the perfect function definition, which can now only be called using:
copytree(‘foo/’, ‘bar/’, dirs_exist_ok=True, symlinks=False)
Unfortunately, position-only arguments are only available starting from Python 3.8, so if you’re on a lower version, you’ll have to manage without them. But while they are a great additional tool to build cleaner interfaces, keyword-only arguments already go a long way by themselves!
Designing Python interfaces
With these tools in our toolbox, let’s have a look at how we can use them to design some beautiful interfaces. What follows is an opinionated set of guidelines. They are not all encompassing and you might want to deviate from them occasionally.
First of all, the meaning of positional arguments depends on a clear function name, so choose it wisely. Good function signatures consist of the same components as natural language sentences. The function name is the verb, describing the action of the function. The positional keywords are the object or subject of the action. And keyword arguments are the adjectives, which describe and modify the action of the function.
In general, required arguments make for good positional arguments, while keyword arguments should most often be optional. This fits well with our natural language comparison, where a sentence will still make sense when removing the adjectives, but not so much when removing the object or subject.
Note that this mostly works in one way though. While required arguments should almost always be positional, positional arguments don’t always need to be required. Especially when clear defaults can be provided, they can be made optional instead. Just look at some of the functions in the Python time module. When calling the gmtime function for instance, the secs argument is optional and will default to None if not explicitly provided, which results in the current time being returned.
time.gmtime([secs])
For functions that require a lot of arguments, it might be an option to turn some of these required arguments into keyword arguments if their meaning is not clearly implied by their position. This is also recognized by pylint, which will stop throwing too-many-arguments warnings in this case. However if you find yourself in this situation, it might be better to check if some logical defaults can be provided for some of the arguments, rendering them optional, or if the code can be refactored so less arguments are needed.
There’s probably an infinite amount of cases where you want to deviate from these guidelines, which is OK. As long as you know why it makes sense to deviate, they will still help you write clear function signatures.
Conclusion
I’ve been using and advocating for keyword-only arguments in the codebases I touch for quite some time now and I’m happy to see that a lot of my colleagues agree with their usefulness and their usage is spreading in ML6. For everyone else, I hope this blogpost could convince you as well that keyword-only arguments can help you design better interfaces.