Redbean 2.0 turned into more than a hobby project 

June 16^th, 2022 @ justine’s web page

redbean is a webserver in a zip
executable that runs on six operating systems. The basic idea is if you
want to build a web app that runs anywhere, then you download the
redbean.com file, put your .html and .lua files inside it using the zip
command, and then you’ve got a hermetic app you can deploy and share. I
introduced this web server about a year ago on Hacker News, where it
became the third most upvoted hobby
project of all time.

Over the last year, we’ve turned redbean into more than a hobby project.
It’s grown to become a 1.9mb file that self-hosts a Lua + SQLite
development stack. There’s builtin MbedTLS support. It does sandboxing.
It has argon2 password hashing. It can geolocate IPs with MaxMind. It
has a readline-like REPL. You can use it as a Lua shebang interpreter.
It has an easy-mode API and a
Fullmoon web
framework for high-level development. It also has a hard-mode API that
provides direct access to Cosmopolitan
Libc Unix system calls. You can use Unix on Windows, or even from
JavaScript too, since redbean is great for spinning up a web GUI in
Chrome via localhost. You can also use redbean as a production web
server on the public-facing internet. I stand by that statement since I
eat my own dogfood. redbean hosts all my websites now, including this
one (justine.lol). There’s no need for a proxy like nginx; redbean is
vertically integrated.

Download
 

Your redbean supports x86-64 Linux, MacOS, FreeBSD, NetBSD, or OpenBSD.
Visit redbean.dev/2.0.html to
download the release binary. redbean is permissively licensed under the
ISC license. The
source
code is available on GitHub. Instructions
for building redbean from
source on Linux are available at redbean.dev.

curl https://redbean.dev/redbean-demo-2.0.1.com>redbean.com
chmod +x redbean.com
./redbean.com -v

APE Loader

redbean 2.0 uses the new
APE Loader which lets your
redbean execute without having to self-modify its header. What happens
instead is the ape command will mmap() your
redbean into memory. It’s just as fast. If APE isn’t installed on a
system, then the shell script header will extract it automatically.
There’s shebang support and binfmt_misc support on Linux too. These
changes will have an enabling impact for distros and build systems, who
had difficulties packaging and distributing APE software. For users who
want the original behavior, an --assimilate flag is
introduced that will turn your redbean into the platform-local ELF or
Mach-O format.

In addition to helping distributors, the redbean 2.0 release helps
self-distributors too. You can now place a
.args
file in your redbean that specifies the default CLI arguments. This
can help make it easier to white-label redbean, especially if it’s being
used as an alternative to the standard Lua interpreter command.

REPL

redbean 2.0 introduces a Read Eval Print Loop or REPL for short. It’s
built on the bestline
library, since it provides near parity with GNU Readline in terms of
features, except it’s licensed MIT instead of LGPL, so there’s no
dynamic linking requirement. redbean can’t dynamically link things,
since then it wouldn’t be a single file. I put a lot of work into
creating bestline, a linenoise fork, for that very reason. Here’s a
short screencast of the redbean repl being used.

Since the video goes by quickly, here’s an explanation of what happened.
The video starts by running
redbean-demo.com -Zv in the terminal. The -v
flag increases the logging verbosity. The -Z flag enables
system call tracing, so you can monitor all the powerful things you’re
doing with the new UNIX module. It works similar to
the --strace flag that I blogged about last week under
Logging C Functions. Once you see
the >: prompt, your redbean REPL is ready to receive
commands.

>: Benchmark(unix.clock_gettime)
125     389      594       1

You can call most of the redbean Lua APIs from this shell. If you’ve
defined global variables and functions in the zip file
.init.lua then you can call those functions too. The
example shown above in the video is of microbenchmarking. In the video
you’ll notice unix.clock_gettime()
takes 125 nanoseconds to run. It’s helpful to be able to run one-off
live experiments like this, since when I made that video for the
sponsors-only pre-release, it helped me realize I could use the Linux
vDSO to make unix.clock_gettime() 10x faster!

>: Benchmark(unix.clock_gettime)
17      53      88      1

So 17 nanoseconds is now the performance you can expect in 2.0. You’ll
also see me computing binary numbers on the command line, like SHA-256.

>: Sha256('hello')
",xf2Mxba_xb0xa3x0e&xe8;*xc5xb9xe2x9ex1bx16x1e\x1fxa7B^sx043bx93x8bx98$"

redbean embraces and extends Lua in many ways. For example, the normal
Lua command line will print ,�M�_��&�;*Ź�x1f�B^s3b���$ for the
binary value above. I figured if a REPL’s input is code, then its output
must be code too, since that’s how LISP does things. Anything
your redbean REPL outputs, can usually be copy and pasted back into your
scripts.

Code Completion

The next thing you’ll see in the video is the new tab completion
feature. Like bash, you can press to
see a listing of all available global functions and objects. If you
press unix. then you’ll see all the
objects and functions available in the unix module.

GNU Emacs Keyboard Shortcuts

Users of GNU Emacs will be delighted to hear that your redbean REPL
supports nearly all the common GNU-style keyboard chording shortcuts,
including CTRL-R for reverse search. See the
keyboard reference for
further details.

Monkey Patching

One of the use cases for having a REPL on a live web server, is you can
monkey patch code while your server is running. redbean is a forking web
server. That means the main process behaves like a master template from
which worker processes are cloned. Therefore, anything you change in the
REPL will propagate lazily into client connections, as new ones roll in,
without impacting the connections currently active.

Tracing Support

redbean 2.0 introduces optional system call logging. The last thing
you’ll notice in the REPL video above (but can’t actually see) is I fire
off a request to redbean from curl. Since we passed -Z we
get a nice system call trace. This logging can all be happening
seamlessly while you’re typing on the REPL.

SYS  15987              7'977 close(4) → 0
SYS  15987             14'055 close(5) → 0
SYS  15987            191'846 read(6, [u"GET /tool/net/demo/index.html HTTP/1.1♪◙"...], 65'536) → 137
I2022-06-13T16:37:34+000400:tool/net/redbean.c:5798:redbean-demo:15987] (req) received 127.0.0.1:57542 HTTP11 GET http://127.0.0.1:8080/tool/net/demo/index.html "" "curl/7.79.1"
SYS  15987            308'231 sigaction(SIGINT, {.sa_handler=0x41da5e, .sa_flags=0x80000000, .sa_mask={}}, [{.sa_handler=0x419305, .sa_flags=0x4000000, .sa_mask={}}]) → 0
V2022-06-13T16:37:34+000063:tool/net/redbean.c:6004:redbean-demo:15987] (rsp) 200 OK
SYS  15987            341'459 sigaction(SIGINT, {.sa_handler=0x419305, .sa_flags=0x4000000, .sa_mask={}}, [NULL]) → 0
SYS  15987            352'506 writev(6, {{u"HTTP/1.1 200 OK♪◙Content-Type: text/html"..., 306}, {u"▼ï◘      ♥", 10}, {u"àRMN▌0►▐τ¶So╪└ïTuüP^╢E]s☺█↓↕âc╗€q≤┬Ü♂!⌡]"..., 511}, {u"╠↑♦φü♥  ", 8}}, 4) → 835
SYS  15987            679'711 read(6, [u""], 65'536) → 0
SYS  15987            683'584 _Exit(0)
>:

Here we see the redbean worker process closing the server socket file
descriptors. The nicest thing about using fork() as the basis of a web
server, is it creates a very safe level of isolation between clients.
For instance, if a redbean worker processes dies, then it’ll die in a
safe space that won’t impact the server as a whole. redbean will also do
things like wipe SSL keys from memory after fork(), so a compromised
worker won’t be able to read them. We also see some sigaction() overhead
in the trace from the video. That’s only needed since redbean is running
in a mode where the REPL is active.

Once the worker process has established itself, the thing that makes
redbean so lightning fast is that it only needs a single system call to
serve each response message. That system call is writev(), which helps
us avoid having to copy buffers. In fact, it’s an even nicer paradigm
than sendfile() since you’ll notice the writev() call has two buffers.
The first is the headers we had to generate. The second is compressed
zip executable content, which is a zero copy operation that happens in
kernelspace, because we used mmap() to load it into memory.

Benchmark

When I spoke about redbean at
SpeakEasy JS [YouTube
Video] last year, I loved bragging about how redbean, a supposedly
slow forking web server, benchmarked at a million qps on my PC when
nginx could only do half that. Is it possible to improve upon
perfection? It turns out, I ran
wrk again for our 2.0 release,
and redbean did 1.1 million qps.

# Note: Benchmarked on an Intel® Core™ i9-9900 CPU
# Note: Use redbean-demo.com -s
$ wrk --latency -t 10000 -c 10000 -H 'Accept-Encoding: gzip' 
    http://127.0.0.1:8080/tool/net/demo/index.html
Running 10s test @ http://127.0.0.1:8080/tool/net/demo/index.html
  10000 threads and 10000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.44ms   46.76ms   1.76s    98.41%
    Req/Sec   189.08    259.45    39.10k    98.67%
  Latency Distribution
     50%    5.68ms
     75%    6.87ms
     90%    8.77ms
     99%  197.91ms
  4327728 requests in 3.72s, 3.37GB read
  Socket errors: connect 0, read 5, write 0, timeout 2
Requests/sec: 1163062.91
Transfer/sec:      0.90GB

Unix Module

redbean 2.0 introduces a
new unix module implemented in
tool/net/lunix.c.
I love the unix operating system and I hope you will too. In fact, I
love it so much, that I wrote Lua wrappers for all of the following
system interfaces.

exit,
fork,
read,
write,
open,
close,
stat,
fstat,
lseek,
access,
pipe,
dup,
poll,
execve,
environ,
link,
unlink,
symlink,
mkdir,
makedirs,
chdir,
rmdir,
sigaction,
setitimer,
nanosleep,
clock_gettime,
gmtime,
localtime,
opendir,
fdopendir,
fsync,
fdatasync,
getpid,
getppid,
getsockname, getsockopt, getuid, kill,
listen, major,
minor,
pledge, raise, readlink, accept,
realpath, recv, bind, recvfrom,
rename, chmod, chown, send, chroot,
sendto, setgid,
commandv, setpgid, connect, setpgrp,
setresgid, setresuid, setrlimit,
, setsid, fcntl, setsockopt,
setuid, shutdown,
sigprocmask, sigsuspend, ftruncate,
siocgifconf, getcwd, socket, getegid, socketpair,
geteuid, getgid, strsignal, gethostname,
getpeername, sync, getpgid, syslog,
getpgrp, truncate, umask,
getrlimit, wait, getrusage,
getsid

We’ve put a lot of work into documenting these functions and making sure
they work on all supported platforms, included Windows, where we’ve
sought to model the Linux kernel behaviors as accurately as possible.
Our goal has been making sure doing anything from Lua will be possible
rather than impossible. As such, this module abstracts very little and
is nearly identical to the APIs provided for the C language.

fd=assert(unix.open("hello.txt"))
st=assert(unix.fstat(fd))
Log(kLogInfo, 'hello.txt is %d bytes in size' % {st:size()})
unix.close(fd)

In many cases, Lua makes using the C syscall functions much easier. For
example, you’ll notice we didn’t need to pass unix.O_RDONLY
to open() since Lua function calls can have default
parameters. You’ll also notice the object oriented access to
the struct stat.
The unix module implements this with a user data object metatable. That
means it goes 10x faster than if we were to construct a table, because
Cosmopolitan’s stat has 16 fields! Thanks to Lua udata objects, we can
return those without needing to copy them.

Write('
')
for name, kind, ino, off in assert(unix.opendir('/etc')) do
   if kind==unix.DT_REG then
      Write('
%s' % {name})
   end
end
Write('')

The dirstream interface is particularly nice, since it’s one of the few
system call interfaces that the C language explicitly defines as
object-oriented. What opendir() does is return an object
that can be __call‘d to receive each directory entry, and
as such, Lua lets it integrate neatly and automatically with all its
lovely language features.

Berkeley Sockets

Thanks to the system call interface designed at UC Berkeley, redbean’s
new unix module means that redbean can now be much more than just an
HTTP server. For example, you can fork() off a daemon:

if assert(unix.fork())> 0 then return end
unix.close(GetClientFd())
unix.setsid()
if assert(unix.fork())> 0 then unix.exit(0) end
unix.close(1)
assert(unix.open('/var/log/daemon.log', unix.O_WRONLY | unix.O_CREAT | unix.O_TRUNC, 0600))
unix.dup(1, 2)
assert(unix.setgid(1000))
assert(unix.setuid(1000))

That listens for client connections on another port:

sock=assert(unix.socket())  -- create ipv4 tcp socket
assert(unix.bind(sock))       -- all interfaces ephemeral port
ip, port=assert(unix.getsockname(sock))
print("listening on ip", FormatIp(ip), "port", port)
assert(unix.listen(sock))
while true do
   client, clientip, clientport=assert(unix.accept(sock))
   print("got client ip", FormatIp(clientip), "port", clientport)
   unix.close(client)
end

That port can be any one of your choosing: smtp, irc, name it.

Further Examples

There’s also a few demo scripts included in the redbean-demo.com binary
release file. You can also read the example code on GitHub. The
unix-webserver.lua demo is particularly nice, since it shows how you can
create a web server within your web server.

• //tool/net/demo/unix-dir.lua
• //tool/net/demo/unix-info.lua
• //tool/net/demo/unix-raise.lua
• //tool/net/demo/unix-rawsocket.lua
• //tool/net/demo/unix-subprocess.lua
• //tool/net/demo/unix-webserver.lua
• //tool/net/demo/binarytrees.lua

Sandboxing

Let’s say you’re downloading Lua extensions for redbean off the web and
you don’t want them poking around in /etc like in the example above.
redbean provides a sandboxing solution for this, that’s available on
Linux and OpenBSD. It works by exposing the OpenBSD
unix.pledge()
system call, which we’ve polyfilled for Linux using SECCOMP BPF.

function OnWorkerStart()
   unix.pledge('stdio')
end

If you do something like that, to reduce privileges on forked workers,
then unix.opendir() will return unix.EPERM
rather than exposing your /etc folder. OpenBSD will just
kill the process whenever a sandbox violation occurs, but we’ve chosen
to be more forgiving with Linux since Lua code that behaves badly should
have the opportunity to react to the error by mending its wicked ways.
For example, many of the unix demo scripts included in redbean won’t run
if you’re using sandboxing. So they’ll print a helpful error if the
system call reports a permission error.

Another important security feature is
unix.setrlimit()
which has a usage example in the
binarytrees.lua
benchmark game. The form takes an exponential parameter for how complex
the game should be. If that number is high, like 25 rather than 18, then
the Lua script will allocate gigabytes of memory and use a ton of CPU.
The setrlimit() function lets you place a limit on how many
resources a connection is allowed to use, before it either receives a
notification signal or gets killed.

Maxmind Module

redbean 2.0 introduces support for reading MaxMind’s free GeoLite2 IP
and ASN databases. It works locally in a self-hosted way, but you have
to sign up on their website to download the necessary data files. Once
you have them, your redbean will gain the superpower of knowing where
everyone lives.

geodb=maxmind.open('/usr/local/share/maxmind/GeoLite2-City.mmdb')
geo=geodb:lookup(GetRemoteAddr())
if geo then
   Write('hello citizen of %s!' % {geo:get('country', 'names', 'en')})
end

This can be an invaluable tool for defending your redbean from the bad
guys on the web. For example, let’s say you want to have a comments
section on your blog. One thing you could do to reduce abuse, while
remaining completely open, is to simply use MaxMind to check that the
visitor is using their home internet connection.

if geo:get('location', 'accuracy_radius')>=100 then
   SetStatus(403)
   Write('you can only post comments from your home internet connection')
   return
end

IPs that come from a data center usually have an accuracy of 1000km. So
if the accuracy is less than 100, then the visitor is much less likely
to be a robot or a bad guy concealing their identity. This is by no
means a perfect criterion for fighting abuse, but it can keep a lot of
unsavory traffic at bay in a scrappy way that doesn’t require FAANG
accounts. It works because the list of bad guys who’ve hacked a fleet of
Comcast home routers is much shorter than the list of bad guys who just
rent cheap virtual servers from the usual suspects in the cloud.

asndb=maxmind.open('/usr/local/share/maxmind/GeoLite2-ASN.mmdb')
as=asndb:lookup(GetRemoteAddr())
asname=as:get('autonomous_system_organization')

Logging the autonomous system name will let you know who the usual
suspects are. When bad guys send unreasonable amounts of traffic, they
love doing it using a fleet of seemingly unrelated IPs from countries
around the world. But one of their weaknesses is that, like most devs,
they’re usually wedded to just one platform. Knowing what the platform
is can be an invaluable tool in drawing the dots and filing complaints
accordingly.

Lua Enhancements

redbean 2.0 introduces enhancements to the Lua language that are
intended to help C/C++ and Python developers feel more comfortable.

• printf modulus operator, like Python. For example, you can
  say "hello %s" % {"world"} instead of
  string.format("hello %s", "world").

• octal (base 8) integer literals. For example
  0644==420 is the case in redbean, whereas in upstream Lua
  0644==644 would be the case. There’s also a new
  oct function.

• binary (base 2) integer literals. For example
  0b1010==10 is the case in redbean, whereas in upstream Lua
  0b1010 would result in an error. There’s also a new
  bin function.

• GNU syntax for the ASCII ESC character in string literals. For
  example, "e" is the same as "x1b".

APIs

redbean 2.0 introduces the following native functions:

EncodeJson,
EncodeLua,
Compress,
Uncompress,
GetMonospaceWidth,
ProgramMaxPayloadSize,
ProgramSslRequired,
ProgramSslClientVerify,
MeasureEntropy,
Decimate,
Benchmark,
Rdtsc,
Lemur64,
Rand64,
Rdrand,
Rdseed,
GetCpuCount,
GetCpuCore,
GetCpuNode,
oct,
hex,
bin

redbean 2.0 adds support for modern password hashing. See
argon2.hash_encoded and
argon2.verify.

redbean now defines the
arg global the same way
as the upstream lua command interpreter.

The argv global is now deprecated.

Bug Fixes

This release includes many upstream fixes from Cosmopolitan Libc. The
quality of the Windows platform support has improved considerably. For
example, fork() now works well enough on Windows that this release
enables it by default. Many other bugs on Windows that would cause
redbean to become unresponsive to CTRL-C interrupts have also now been
resolved.

See Also

• Redbean Discord
• //tool/net/redbean.c

Funding

Funding for the development of redbean was crowdsourced from Justine Tunney’s
GitHub sponsors
and Patreon subscribers. Your
support is what makes projects like redbean possible. Thank you.