OmniPython Website GitHub
OmniPython

Start here

  • OmniPython
  • New here?
  • Install

Libraries

  • omnipython
    • Get started
    • Install
    • Your first program
    • Questions
    • Guides
    • Bringing up a board
    • App and board config
    • Displays
    • appdev
    • multimer
    • Upgrade multimer from OmniPython 0.5
    • Portable audio
    • USB, both ways round
    • bledev: Bluetooth Low Energy
    • Bluetooth LE: REPL, files, pairing, MIDI and HID
    • The board’s name
    • wifi_manager
    • mdns
    • medialink
    • Using LVGL with OmniPython
    • vtterm: a terminal on a display
    • castdisplay: cast an app to a TV
    • omnipython-desktop
    • Direct MicroPython WebAssembly
    • PyScript and Pyodide
    • Android
    • Jupyter Notebook
    • Run the tests
    • Troubleshooting
    • Reference
    • The stack
    • API reference
    • USB support
    • CircuitPython’s APIs on MicroPython and CPython
    • Bluetooth LE support
    • vtterm escape codes
    • vtterm keys
    • castdisplay
    • Other modules
    • Board peripherals contract
    • Board configs
    • Tested boards
    • Arduino GIGA R1 WiFi with the Display Shield
    • Display drivers
    • Touch drivers
    • Wokwi hardware reference
    • How it works
    • How OmniPython fits together
    • Architecture
    • The PyDevices ecosystem
    • Display backend internals
    • multimer internals: the dispatcher and the wake sources
    • bledev internals: writing a backend
    • vtterm internals
    • How casting works
    • Changelog
    • Changelog
  • pygraphics
  • palettes
  • LVGL
  • pdwidgets
  • audiodsp
  • audioinstruments
  • audioeffects
  • Firmware and modules
OmniPython
  • OmniPython
  • medialink
  • Edit on GitHub

medialink

medialink sends live sound from one device to another over your local network: hold a button on one board, speak, let go, and the other board says it. A board and a computer are peers of the same kind, because it runs on MicroPython, CircuitPython and CPython alike.

import boarddev, medialink
from audiodev import AudioFormat

audio = AudioFormat(16000, 1, 16)
link = medialink.Link(medialink.UDPTransport(47800))
link.connect(("192.168.1.20", 47800))      # the other device
intercom = medialink.Intercom(link, boarddev.pcm_in(audio), boarddev.pcm_out(audio))

intercom.press()        # take the floor and talk
...                     # call intercom.service() from your loop
intercom.release()      # let go: the other side may talk now

medialink_walkie_talkie is the whole walkie-talkie, finding the other side by name. examples/medialink_simpletest.py sends one frame and shows a send to an absent peer failing. Three more put the same pieces to other uses: medialink_baby_monitor, medialink_doorbell and medialink_hands_free, described below.

Install

On a board, medialink is its own package in the OmniPython index, so a board that never streams doesn’t carry it:

import mip
mip.install("medialink", index="https://lib.omnipython.org")

On a computer it comes with omnipython. Finding a peer by name needs the mdns module (mdns): built into CircuitPython and into MicroPython firmware that carries it, and from omnipython-desktop with python-zeroconf on CPython.

Finding each other

Each side advertises itself by name, and looks the other up:

finder = medialink.MDNS()        # on a board: MDNS(network.WLAN(network.STA_IF))
finder.advertise("kitchen", 47800)
for name, address in finder.find("garage"):
    link.connect(address)

Either side may find the other first, and the side that is found learns the address of the side that called. So a computer whose firewall refuses incoming connections can still be a peer, as long as it is the one that calls. When you know the address, skip discovery and connect() to it.

Nearby, by Bluetooth LE

A board can also say where it is over Bluetooth LE, which finds a board in the same room without mDNS. Swap MDNS for BLE:

finder = medialink.BLE()
finder.advertise("kitchen", 47800)       # name, IP address and port
for name, address in finder.find("garage"):
    link.connect(address)

The sound still goes over the network; Bluetooth only carries the address. A computer can find a board this way but can’t advertise, since bledev’s computer backend only scans. Anyone nearby can read the advertisement, so it carries nothing secret: never a password.

Without the house network: a hotspot

Where the house Wi-Fi is weak or missing, two boards can make a network of their own. Swap MDNS for Hotspot, with a password both sides share:

finder = medialink.Hotspot("a long password")
finder.advertise("kitchen", 47800)
for name, address in finder.find("garage"):
    link.connect(address)

On the first find() the side whose name sorts first starts an access point named after itself; find() there returns nothing, because the other side calls it. The other side joins that access point and gets back the host’s address. Hotspot(..., host=True) picks the host yourself, and ssid= names the network.

Joining a hotspot takes a board off the network it was on. finder.deinit() takes the access point down, leaves the hotspot and rejoins the board’s own network, the one in secrets.py on MicroPython or settings.toml on CircuitPython. It works on boards with Wi-Fi, not on a computer.

Use it on a network you trust

medialink has no encryption and no passwords of its own. Anyone on the same network can read what you send. They can also connect to your device in place of its peer, take the floor, and send it sound or pictures.

So use it only on a network where you trust everyone, which matters most for a baby monitor, a doorbell or a camera. Keep it off public and guest Wi-Fi, and don’t forward its port to the internet. A hotspot’s password keeps out anyone who doesn’t have it.

When a peer isn’t there

connect(), request_floor() and Intercom.press() wait for the other side to answer. If it doesn’t, they raise medialink.PeerUnreachable. By default that is after one try, about a third of a second; Link(..., retries=N) tries N more times first.

While the link is open it keeps a board’s Wi-Fi awake. With power saving on, the radio holds incoming packets until it next wakes, which loses some and makes the rest late. UDPTransport(..., power_save=True) leaves the setting alone.

Each side pings the other every second. link.alive says whether the peer has been heard from in the last three seconds, and link.rtt_ms is the latest round trip. When the network drops and comes back, even at a new address, the pings find the peer again and the link carries on.

Taking turns

A walkie-talkie is half duplex: one side talks at a time. press() asks for the floor and returns False when the other side has it. When both press at once, one of them gets it and the other is refused, never both. A listener that hears nothing for a second and a half takes the floor back, so a talker that drops off the network doesn’t hold it.

On many boards the speaker and the microphone share one I2S port, so only one can be open. press() closes the speaker before it opens the microphone, and the speaker opens again when the other side next talks. If the microphone still won’t open, press() gives the floor back and raises.

Hands-free, one way, and a doorbell

Give an Intercom a medialink.VoiceActivity and nobody needs a button. It listens to the microphone while the floor is free, takes the floor when someone starts to speak, and gives it back when they have been quiet for hold_ms (600 ms by default). It sends from a little before the first word was heard (preroll_ms=120), so the word isn’t clipped. A button still works beside it: press() during a hands-free turn makes the turn the button’s.

voice = medialink.VoiceActivity(16000, hold_ms=800)
intercom = medialink.Intercom(link, mic, speaker, voice=voice)

VoiceActivity uses audiodsp’s speech detector where the firmware has audiofeatures, which ignores a fan or road noise that stays on. Elsewhere, CircuitPython among them, it uses a level gate: the room’s quiet sets a floor, and sound 10 dB over it counts. voice.detector says which you got. The gate can’t tell a voice from a door slamming. Both learn the room’s quiet from the first sound they hear, and pass over digital silence (frames in which every sample is the same), which a microphone can send while it starts.

When both sides speak at once, one gets the floor and the other’s words are lost until the turn ends, as with two buttons. While the peer talks, this side doesn’t listen, and it waits for the speaker to finish before it listens again, so it doesn’t send the peer’s own voice back. It also ignores the microphone’s first 60 ms each time it opens (settle_ms), since a microphone can click as it starts, and the click would otherwise take a turn of nothing back after every turn the peer took. medialink_hands_free is the whole thing.

To send only when someone says a wake word, put a medialink.WakeWord where the VoiceActivity was. It runs one of ESPHome’s microWakeWord models, such as “Hey Jarvis”, and opens the floor when the word is heard. The floor stays open for listen_ms (2 s) for speech to begin, then for as long as the voice activity hears it; the next turn needs the word again. Talk in the room without the word stays home.

voice = medialink.VoiceActivity(16000, hold_ms=800)
wake = medialink.WakeWord("/sdcard/models/hey_jarvis.json", 16000, voice=voice)
intercom = medialink.Intercom(link, mic, speaker, voice=wake)

The model’s .json manifest and its .tflite sit side by side, from ESPHome’s micro-wake-word-models. It needs firmware with tflite and audiodsp’s audiofeatures, which the ESP32-S3 and ESP32-P4 MicroPython builds can carry; WAKE_WORD in the hands-free example turns it on.

An Intercom with no speaker only sends, and one with no microphone only listens: Intercom(link, mic, None, voice=voice) is a baby monitor that sends while there is sound and nothing otherwise. medialink_baby_monitor shows “sound” and “quiet” on the listening side.

link.send_event("ring") sends a short message the peer acknowledges, sent again until it does, and link.events() on the other side returns what arrived, each once. intercom.play(pcm) plays a sound of this side’s own, such as a chime, without the microphone hearing it as speech. medialink_doorbell uses both: the door’s button rings a chime in the house, the visitor speaks hands-free, and the house holds a button to answer.

Sound quality and delay

Sound goes as 20 ms frames of 16-bit PCM. On a slow link, medialink.choose picks IMA ADPCM instead, a quarter the size: 16 kHz mono is 256 kbit/s as PCM and 64 kbit/s as ADPCM. You can pass either as Intercom(..., codec=medialink.ADPCM).

The listener waits for five frames (100 ms) before it starts to play, which absorbs the unevenness of Wi-Fi; Intercom(..., prebuffer=3) cuts the wait where the network is steadier. Between two boards on home Wi-Fi, sound took about a tenth of a second from one microphone to the other speaker.

A late burst (a slow moment on Wi-Fi, or a loop busy decoding a picture) leaves more than that waiting at the speaker, and because sound arrives as fast as it plays, the extra would stay for the rest of the turn. So when the speaker’s queue hasn’t come down to the prebuffer for two seconds, the listener skips a few frames to catch up. Between two ESP32-P4 boards, timed by ear (a burst sent, heard back from the far speaker), that brought a minute’s sound from half a second late down to under 0.2 s, and with the board also sending pictures, from 0.4 s to about 0.22 s. Each catch-up is a short skip you may hear. Intercom(..., target_ms=...) sets the queue it aims for, catch_up_ms= how long it waits first, and catch_up_ms=None keeps every frame.

On a CircuitPython board whose screen shows the console, every line you print scrolls and redraws that screen. On an 800x480 panel that stops your program for over 400 ms, and the frames that arrive meanwhile don’t fit the few the network stack keeps for you: a listener that printed “sound” at the start of each one-second turn heard 34 or 35 of its 50 frames. Frames a talker captured during the stall go out in a burst that the listener can’t keep either. With the console off the screen, every frame arrives. A OmniPython board config takes it off for you; a program that doesn’t import one does it itself, as Board configs shows.

intercom.stats says what came of it: latency_ms is from the far microphone to this speaker, underruns counts the times the speaker ran dry in the middle of a turn, and trimmed the frames skipped to catch up.

Pictures

medialink.Video sends JPEG pictures on the same link as the sound. Send one whenever the last has gone, call service() from your loop, and show what arrives with medialink.Screen:

video = medialink.Video(link)
screen = medialink.Screen(display_drv)   # from board_config

if video.ready:
    jpeg = camera.take()                 # a JPEG, or None
    if jpeg is not None:
        video.send(bytes(jpeg))
video.service()                          # sends the next few packets
frame = video.receive()                  # the newest picture, or None
if frame is not None:
    screen.show(frame.payload)

Three examples put it together. medialink_surveillance_camera is a camera you watch and listen to from another room. medialink_video_doorbell shows the door inside, with sound both ways. medialink_videophone sends pictures both ways. On a computer without a camera, set PICTURES to a folder of JPEG files and medialink.FileCamera sends them in turn.

A picture is cut into packets of the link’s size and put back together on the other side, byte for byte. If one packet is lost, that picture is dropped and the next one comes as usual. The receiver keeps only the newest pictures, so a slow screen shows the latest one rather than falling behind.

Video paces its packets to the link’s bandwidth, 2 Mbit/s for UDP unless you pass Video(link, bandwidth=...), and a few at a time, so sound goes out between a picture’s packets and a board’s network buffers don’t overflow. At that rate a 20 KB picture takes about 80 ms to send; a 300 KB photo takes over a second. Keep pictures small: a camera’s lower resolution and a JPEG quality around 60 do more than a faster link. On an ESP32-P4, camera.reconfigure(pixel_format=cameraif.PixelFormat.JPEG, frame_size=cameraif.FrameSize.VGA) has the chip scale the picture to 640x480 and encode it, about 7 KB, and camera.take() returns it. A 320x180 picture is 115 KB as raw pixels and about 3 KB as JPEG at quality 60, so JPEG is the codec, even where the encoding is done in software.

Screen halves a picture until it fits the display, centres it, and uses the hardware decoder where the firmware has one. The P4’s decoder works only at full size, so send pictures no bigger than the screen: a 1280x960 picture halved for an 800x480 screen took about 400 ms to decode and draw in software, and a 640x480 one 77 ms in hardware. A picture it can’t read is counted in screen.errors and skipped. The decoder in MicroPython firmware reads baseline JPEGs with 4:4:4, 4:2:2 or 4:2:0 colour, which is what cameras make; a progressive JPEG from a photo editor is skipped there, and shows on CPython, where Pillow decodes it.

Decoding a picture holds up the loop that also carries the sound. With pictures both ways on two ESP32-P4 boards, 320x240 pictures (27 ms each to decode and draw) left the sound clean, and 640x480 ones (83 ms) made it late and gappy, so a videophone wants SIZE = "QVGA". One-way pictures, as from a camera you watch, take 640x480 with clean sound.

The intercom takes only sound from the link and leaves pictures for the screen, and pictures alone don’t keep the floor: a talker whose camera is still sending but who has gone quiet loses it as usual. One camera sends to one watcher at a time, the last that said hello.

To watch a camera on a TV, run castdisplay’s castdisplay_medialink_tv.py as the watcher. It casts the pictures and sound to a Roku TV or to Windows Wireless Display over Miracast, so the board running it needs no screen of its own. On the camera side set RATE to 24000 and FPS to 5: an ESP32-P4 watcher keeps up with 5 pictures a second and leaves time for the sound, where at 10 a second it drops packets.

How it is built

Three layers, so a new kind of link is a short file:

  • Media: frames and their timestamps, the floor, the codec, pictures (Intercom, medialink.codec, Video).

  • Transport: open, send, receive, close, and the link’s packet size and rough bandwidth (medialink.Transport). UDP over IP is the one built.

  • Discovery: a name in, an address out (medialink.MDNS, medialink.BLE, medialink.Hotspot).

A Link joins a transport to one peer. Media frames go best effort; a frame bigger than one packet, a video frame say, is cut up and put back together. Hello, goodbye, the floor and events are acknowledged and retried.

Previous Next

© Copyright 2024-2026, Brad Barnett & PyDevices Contributors.