Sunday, November 20, 2022

Slashes in Lua "require" function: Use periods/dots!

This is somewhat a kind of misunderstood but I hope this blog post will resolve this once for all. I often saw people using Lua require like this

local gamera = require("libs/gamera")
local nvec = require("libs/nvec")
-- Rest of the code

At first glance, there's nothing wrong right? No. Using slashes in require works because it's an accident, but no this is not a happy accident. What you should do is:

local gamera = require("libs.gamera")
local nvec = require("libs.nvec")
-- Rest of the code

So, why the dots there? Because require is not expecting path, it's expecting module name. To explain why, first let's take a tour to Programming in Lua Section 8.1, with important text marked as bold.

The path used by require is a little different from typical paths. Most programs use paths as a list of directories wherein to search for a given file. However, ANSI C (the abstract platform where Lua runs) does not have the concept of directories. Therefore, the path used by require is a list of patterns, each of them specifying an alternative way to transform a virtual file name (the argument to require) into a real file name.

So, it's clear that require does not expect a path to filename, but it expects module name (or virtual file name; we'll use module name from now on). To understand how Lua transform the module name to actual path that Lua will try to load, it's important to know about package.loaders.

package.loaders is an array of function which tries to load a module based on module name passed by require. The Lua 5.1 manual has more information about this, but I'll try to explain it as simple as possible. In Lua 5.1 (and LuaJIT), there are 4 loaders in this entry but I'll only explain the first 3, tried in this order:

  1. Checks for existing module loader in package.preload table with the module name (passed from require) as the table key, such that when called, it loads the module. If it's non-nil, then the value is returned.
  2. Replace all dots in module name to OS-specific directory separator (we'll call this file path). Then for each semicolon-separated path specified in package.path, substitute question mark with the file path then try to open that as Lua file. If it's loaded successfully then the function chunk is returned.
  3. For this, it needs 2 components: file path and entry point. Replace all dots in module name to OS-specific directory separator to get file path, and replace all dots in module name to underscore with luaopen_ prepended to get entry point. Then for each semicolon-separated path specified in package.cpath (note the "c" in cpath), it tries to load said file as shared library (or DLL in Windows), then returns Lua C function with specified entry point inside the shared library.
  4. It's all-in-one loader, doesn't matter in our case. 

If you're still confused, this pseudo-Python code will help you know how it works.

import os

package.preload = dict()
package.path = "?.lua;path/to/?.lua"
package.cpath = "?.dll;?.so;path/to/?.dll;path/to/?.so"

def loader_1(modname):
	module = pacakge.preload.get(modname)
	if module is not None:
		return module
	return f"no field package.preload['{module}']"

def loader_2(modname):
	file_path = modname.replace(".", os.sep)
	tested = []
	for path in package.path.split(";"):
		file_name = path.replace("?", file_path)
		chunk = load_lua_file(file_name)
		if chunk is not None:
			return chunk
		tested.append(f"no file '{file_name}'")
	return "\n".join(tested)

def loader_3(modname):
	file_path = modname.replace(".", os.sep)
	entry_point = "luaopen_" + modname.replace(".", "_")
	tested = []
	for path in package.cpath.split(";"):
		file_name = path.replace("?", file_path)
		module = open_shared_library(file_name) # dlopen or LoadLibraryA
		if module:
			symbol = get_symbol(module, entry_point) # dlsym or GetProcAddress
			if symbol is not None:
				return make_symbol_callable(symbol)
		close_shared_library(module) # dlclose or FreeLibrary
		tested.append(f"no file '{file_name}'")
	return "\n".join(tested)

If that's clear enough, then stop reading and start fixing your require by replacing slashes with dots!