Portable audio

One name, one return type

There are three audio roles, and each always hands back the same kind of object — on a board, on a desktop, everywhere:

Role

Returns

You have

You want

pcm_out(format=None, …)

PCMOutput

PCM bytes

to write() them

pcm_in(format=None, …)

PCMInput

nothing

to readinto()

audio_out(format=None, …)

AudioOut

a sample graph

to play() it

There is deliberately no audio_in. Playback has a player layer above raw PCM; capture has none, and audiodsp has no input-side node to give it one. A name implying otherwise would be the vagueness that caused the bug below.

This used to be inconsistent, and it cost real code. audiodev.auto.audio_out returned a raw PCMOutput while board_peripherals.audio_out returned an AudioOut wrapping one, so an app that could reach either factory could not know what it was holding and wrote:

pcm = audio_out(FMT, latency="low")
pcm = getattr(pcm, "transport", pcm)     # which one did I get?

It writes pcm_out(FMT, latency="low") instead and gets a PCMOutput. Note that pcm_out also needs no audiodsp in firmware — nothing pulls a sample graph — which is what lets a headless Connect speaker or a USB sound card run on a build with no DSP package in it.

Host or board is resolved by boarddev, not by audiodev (which must stay usable with no OmniPython board tree at all):

from boarddev import pcm_out
pcm = pcm_out(AudioFormat(44100, 2, 16), latency="low")

It resolves to board_peripherals, never board_config — importing that on the ESP32-P4 starts MIPI DSI and stalls the VM, and a headless audio app has no use for a display.

What a board will accept

A board publishes an AudioCapability as AUDIO_OUT / AUDIO_IN, and audiodev.negotiate() decides. Boards declare facts; they do not write validation code.

AUDIO_OUT = AudioCapability(
    AudioFormat(24000, 1, 16),   # default when format=None
    rates=None,                  # None = continuous (an I2S PLL); or a tuple
    channels=(1, 2),             # slot counts the WIRE will open
    native_channels=1,           # signals that reach a transducer
    bits=(16,),
    wire=I2SWire(0, sck=12, ws=10, sd=9, mck=13),
)

channels and native_channels are not the same thing, and conflating them is a bug. The ESP32-P4’s ES8311 clocks two slots into one speaker, so 44.1 kHz stereo opens I2S.STEREO and nothing is mixed down. A board that could not say this would have to either refuse stereo or remix it needlessly.

negotiate raises rather than substituting. A silently-different format sounds like working audio at the wrong pitch, which is the kind of bug nobody notices until it ships.

capability.wire is for the third kind of consumer: one that opens the peripheral itself and wants no Python device at all. audiodsp’s pump is the live example: pump.BusioDriver(capability.wire, ...) plays on the board’s I2S from C.

Channel conversion

When a board’s wire genuinely cannot take the requested channel count, negotiate says so and adapt_channels() bridges 1↔2. The conversion lives in audiodev — it is PCM byte work, the same kind _scale_pcm already does for software volume — but audiodev imports no audiodsp, so the default implementation is pure Python: correct everywhere, and slow.

A caller with audiodsp passes the C one in:

from audiodev.accel import best_remix
pcm = adapt_channels(pcm, fmt, remix=best_remix())

Capture works the same way in reverse. adapt_input_channels() reads the microphone at the wire’s channel count and hands you the count you asked for: a stereo wire averages to mono, and a mono wire copies to both channels. A board config’s pcm_in does this for you.

“Slow” is measured, not hand-waved. On a QT Py ESP32 Pico at 240 MHz, one 10 ms chunk of stereo→mono costs 0.63× realtime at 16 kHz, 0.94× at 24 kHz and 1.73× at 44.1 kHz. So pcm_out needing no audiodsp holds while the format matches the wire — the usual case, since a board whose wire takes two slots never remixes. A board that must genuinely mix down at a high rate needs audiodsp in firmware. That is a real constraint on board design, not a detail.

audiodev/__init__.py and every transport backend import no audiodsp at all; only sample_out.py and accel.py may. A test asserts this by walking the AST, because the rule was broken once without anyone noticing.

Directions

Playback: the audiosample contract

On MicroPython, play() only starts the sample: something has to keep servicing the output, or you hear the first 512 frames and then silence. Call out.attach(app) once after you create the output, or call out.service() from your own loop (see the audio pump for the details).

audio_out returns an audiodev.sample_out.AudioOut: play(sample, loop=False), stop(), pause(), resume(), playing. sample is anything satisfying CircuitPython’s audiosample pull protocol — a synthio.Synthesizer, an audiomixer.Mixer, an audiocore.RawSample / WaveFile, or any effect chained on top of one (audiofilters, audiodelays, audiofreeverb, audiospeed), all provided by the audiodsp usermod on MicroPython or the separately installed omnipython-audiodsp CPython distribution (import synthio, import audiomixer, …). CPython users install it from PyPI with:

python -m pip install omnipython-audiodsp

Neither omnipython distribution depends on it; importing audiodev and using raw PCM transports therefore remain available without synthesis support. This is the same object shape CircuitPython code already expects from audiobusio.I2SOut/audioio.AudioOut — a synth built for a CircuitPython tutorial runs unchanged against this port’s AudioOut.

On CircuitPython boards (board_configs/cp/) the contract is satisfied natively: there is no AudioOut wrapper. Where a CP board config has an audio_out role, it returns CircuitPython’s own audiobusio.I2SOut; without one, apps construct audiobusio.I2SOut(...)/audioio.AudioOut(...) directly (a CP board_peripherals.py is optional, see the board contract). Same protocol shape, zero adapter, because CircuitPython’s own output devices already speak it.

On MicroPython, AudioOut is a pure-Python pump wrapping a push-based transport (below): it pulls PCM from the sample via audiocore.get_buffer()/reset_buffer() on a lookahead schedule and pushes it into the transport’s write(). This is the only sound architecture here, not a stopgap: every real backend’s bottom layer is a push+queued transport (SDL_QueueAudio, WASAPI, the wasm bridge, machine.I2S.write), and an audio-thread callback into Python is unsafe even in C once it would need to pull a synth’s block graph, which allocates on the GC heap (documented in sdl2_audio.py). So playback flows only while something calls AudioOut.service() — the app tick, an asyncio pump, or attach(app) — same requirement every queued transport already has. No resampling: a sample’s own sample_rate is not renegotiated against the transport; a mismatch plays at the wrong pitch rather than raising, exactly like real DAC hardware and exactly like CircuitPython’s own AudioOut/I2SOut.

Raw PCM streaming is a role of its own, not an escape hatch through this one: call pcm_out() and get a PCMOutput — write, drain, awrite, adrain, volume/mute, codec. AudioOut.transport still exposes the same object, but reaching through it means constructing an AudioOut (and therefore requiring audiodsp) for a path that never plays a sample.

The audio pump

Nothing changes for you. audio_out(...) still hands back an AudioOut, play(sample) still plays it, pcm_out(...) still hands back a PCMOutput you write() to. The code you already have runs unchanged.

What changes is where the pulling happens. On a firmware built with the audiopump module, AudioOut hands your graph to a C thread the interpreter is not on — a pinned FreeRTOS task on esp32, a pthread on a desktop — which pulls every block and writes it straight into I2S. Nothing asks for it and no board config opts in; where the module is absent, every byte is the one the old path produced.

What it buys is that your app keeps running while the sound keeps its clock.

Three things to know before you rely on that on an ESP32:

  • Nothing writes to flash while the pump plays. A flash erase suspends the pump’s core until it finishes — no priority and no IRAM placement reaches it. Measured for a 512-byte write: 27–37 ms of stopped audio on the P4, 43–48 ms on the LilyGO T-Embed S3, against a 5.3 ms block. So no log line, no saved preset and no mip install while something is playing. A read costs nothing (5.9 ms worst on the S3, zero starved bytes), and a first import is a read.

  • A screen costs you ring depth, and the number is per board. On the P4, lighting a 720 × 720 panel is a standing PSRAM-bandwidth cost of about eleven points of a block before a finger touches it, and a GUI app wants 12 × 128 (32 ms). On the T-Embed’s SPI panel a lit screen costs nothing measurable and 4 × 128 (10.7 ms) is the knee — depth buys nothing after it. Drawing often is dearer than drawing big on both.

  • Two players share one I2S. The first client is pulled with nothing in the way; a second gets a root audiomixer.Mixer built for it, a voice each at level 1.0. The mixer sums, so two loud clients clip — the same arithmetic two CircuitPython AudioOuts into one DAC would do.

A WaveFile, an MP3Decoder or a VorbisDecoder cannot be pulled from that thread at all, because its get_buffer reads the filesystem; AudioOut.play() puts a prefetcher in front of it and the pump pulls a ring instead. And a fault is a sentence on stdout and playback on the old path, never a traceback from nowhere.

Asking how it is doing

audiodev.pump.health() returns one dict, or None on a firmware with no pump, so a status screen can ask unconditionally:

from audiodev import pump
print(pump.health())
# {'clients': 1, 'reentered': 0, 'dropped': 0, 'waited': 0, 'waited_us': 0,
#  'blocks': 1, 'bytes': 38400, 'running': True, 'ahead_ms': 400.0,
#  'fault': None, 'blocked': False}

dropped is the one that means audio was not heard. reentered is not: it counts service ticks that arrived while something was rearranging who is sounding, and a refused tick costs nothing — the buffer under it is hundreds of milliseconds deep and the next tick does what this one did not. A number that climbs there says your timer is faster than your own rearrangements. blocked True is the state where every play() goes to the interpreter thread, and fault says why. Each entry’s meaning is on Pump.health()’s docstring.

The mechanism, the driver per port, what a fault says and what is still unproven: lib/audiodev/README.md.

Stopping in the middle

Close a player mid-sample, or let a UI move on and stop draining, and what the transport was handed is the beginning of the render it would have got if nobody had stopped: the same bytes, in order, just fewer of them. Nothing is inserted, nothing skipped, and no zeroed block ever stands in for one that was not ready.

How much you get is not a promise and cannot be. The pump’s schedule is paced by the wall clock, so the stream ends on a block boundary somewhere near where you stopped, and two runs that stop after the same number of ticks can end at different bytes. Ask for the audio up to the moment you stopped; do not ask for a frame.

Measured by tests/pump_probes/stopped.py, which also says why a recording transport cannot ask this question.

A transport that stops consuming

AudioOut caps how far ahead it pulls, and that cap has to clear whatever the device needs queued before it starts playing — prebuffer_bytes on the device, which a wrapper forwards like queued_size. A wrapper that answers 0 there used to wedge the stream in silence.

It no longer can. The cap is only believed while the queue is moving: a queue that has not fallen for eight consecutive skips is fed anyway, until it does or until a second of audio is waiting, which is a dead sink rather than a priming one. AudioOut.stalls counts those episodes, and a number above 0 means the sink was not draining.

Capture and tones

pcm_in returns a PCMInput (readinto/areadinto); tone/buzzer roles return a ToneOutput (play(frequency, ...), frequency-based, not sample-based — unrelated to AudioOut.play(sample) above). Both report an AudioFormat plus a capabilities set; volume, gain, and mute use a normalized 0–100 scale and select hardware controls when a codec provides them, otherwise audiodev scales PCM in software.

Queued hosts also implement service(), queued_size(), is_active(), and clear() on the device (no-ops on MCU).

Codec-specific features remain reachable through device.codec. Shared half-duplex hardware uses AudioSession; opening the opposite direction while one direction owns the session raises OSError.

Recording while you play, on one I2S port

Some boards put the speaker codec and the microphone on one I2S port (the ESP32-P4 boards do). machine.I2S opens one direction per port, so there you can’t record and play at the same time through it. On firmware with the audio pump you can: audio_out opens the port as a TX/RX pair, and pcm_in returns an audiodev.pump.PumpPCMInput that reads the RX half of the same channel. Open them in either order; closing one leaves the other running.

out = board_peripherals.audio_out()
mic = board_peripherals.pcm_in()
out.play(sample, loop=True)
mic.open()
buf = bytearray(48000)          # one second at 24 kHz mono
mic.readinto(buf)               # the speaker keeps playing
print(mic.stats())              # {'captured': ..., 'dropped': 0, 'waiting': ...}

The two share one clock, so they share one rate. A recorder at another rate than the speaker is refused, and so is a sample at another rate while a recorder is open. pcm_out is machine.I2S, so it still can’t open alongside a recording. A board joins in by publishing sd_in=, the microphone’s pin, on its I2SWire.

Host backends (transports under AudioOut, or usable raw)

audiodev.sdl2_audio is the reference playback and real-microphone backend for MicroPython. It uses queued SDL audio and provides the same sync and async contract as hardware devices.

audiodev.win_audio is the Windows WASAPI backend (uwin32), selected ahead of sdl2_audio on Windows when available. On micropython.exe, a sample player on the default speaker plays through the audio pump instead, which writes to WASAPI from its own thread (how).

audiodev.alsa_audio plays and records through ALSA on Linux, from CPython and unix MicroPython, with no SDL2 or pygame. alsa_audio.devices() lists the sound cards; pass one as device="plughw:2,0". It is tried last, and only where there is a sound card.

audiodev.wasm_audio is the direct MicroPython WebAssembly backend. The compiled bridge owns Web Audio and permission state, while Python sees only neutral PCM buffers and queue operations. It supports mono/stereo 8-, 16-, and 32-bit signed or unsigned samples in either byte order at caller-selected rates. A host page must explicitly enable audio or microphone access first.

audiodev.worklet_audio is the worker half of an AudioWorklet output, for Python running in a Pyodide Web Worker. The page owns the AudioContext and the worklet and gives your worker a MessagePort; the worklet asks for blocks of 256 frames and this module answers, either from a render callback you set (WorkletOutput(port).source = render) or from the queue behind an ordinary pcm_out(format, port=port). It tells you the worklet’s frame offset, so you can place an event at an exact sample. Import it explicitly: automatic selection never picks it. The protocol is in lib/audiodev/README.md.

audiodev.i2s_audio adapts machine.I2S for MCU boards; boards construct it directly with board-specific pins/codec wiring, then wrap it in AudioOut.

Desktop board_peripherals may use audiodev.auto for host probing. Fixed host boards import a concrete backend. audiodev itself does not import displaydev.

CPython and Pyodide sample playback

audiodev.pygame_audio (CPython + pygame-ce, via SDL_QueueAudio on pygame’s bundled SDL) and audiodev.web_audio (PyScript/Pyodide, via AudioContext/getUserMedia) still exist and still work as raw PCMOutput/PCMInput transports and can back AudioOut when omnipython-audiodsp is installed. Automatic selection uses MicroPython wasm, Pyodide Web Audio, Windows uwin32, usdl2, then pygame-ce, in that order. Constructing AudioOut without audiocore fails immediately with the install command above; direct raw write()/readinto() remains independent.

Write your own transport

audiodev.auto looks its transport up in the same small registry as AutoDisplay (backends, explained in Displays). A transport is a module that provides pcm_out and pcm_in:

import backends
backends.register("audio", "mine", "my_audio", probe=lambda: my_card_present())

# board_config.py
backends.require("audio", "mine")

select_backend() tries the OMNIPYTHON_AUDIO environment variable (one name, nothing else), then what a config required, in order and skipping any whose probe says no, then the built-in transports in the order above. The built-in names are wasm_audio, web_audio, win_audio, sdl2_audio, pygame_audio and alsa_audio. A transport you registered is never picked unless it was required or you named it. On MicroPython, require is the only way a transport gets registered; nothing scans for installed ones.

Emulated devices (CI / no hardware)

audiodev.emulated_audio implements WAV files, a waveform generator, in-memory loopback, and a discard sink. It is never auto-selected:

from audiodev import AudioFormat
from audiodev.emulated_audio import pcm_in, pcm_out

fmt = AudioFormat(24000, 1, 16)
out = pcm_out(fmt, path="/sd/prompt.wav")
out.write(pcm_bytes)
out.close()

mic = pcm_in(path="/sd/prompt.wav")

WavPCMOutput doubles as the deterministic golden-file target for testing the whole AudioOut chain — synthio/Mixer rendered through AudioOut over emulated_audio.pcm_out(path=...), hash-compared, no audio hardware needed. See tests/test_audiodev.py.

ESP32-P4 status

The Waveshare ESP32-P4 configuration shares one I2S port between the ES8311 and the ES7210; with the audio pump, audio_out and pcm_in run at once (see “Recording while you play” above), and without it the two refuse each other. Playback exposes hardware DAC volume/mute and controls the speaker amplifier through AudioOut; capture exposes hardware ADC gain. Register, stream, session, and GPIO behavior is covered by host simulations.