The Jaccl distributed backend requires MLX 0.30.1+, which includes the
RDMA over Thunderbolt support. The previous minimum version (0.29.3)
would fail at runtime with "The only valid values for backend are
'any', 'mpi' and 'ring' but 'jaccl' was provided."
Bump MLX dependency to >=0.30.1 and rename ibv_coordinators to
jaccl_coordinators to match MLX's naming conventions. This includes
the environment variable change from MLX_IBV_COORDINATOR to
MLX_JACCL_COORDINATOR.
Test plan:
Hardware setup: 3x Mac Studio M3 Ultra connected all-to-all with TB5
- Built a DMG [0]
- Installed on all Macs and started cluster.
- Requested a 2 node Tensor + MLX RDMA instance of Llama 3.3 70B (FP16).
- It started successfully.
- Queried the chat a few times. All was good. This didn't work
previously.
- Killed the instance and spawned Pipeline + MLX Ring Llama 3.3 70B (FP16).
Also started succesfully on two nodes and could be queried.
Still not working:
- Pipeline + MLX Ring on 3 nodes is failing. Haven't debugged that yet.
[0] https://github.com/exo-explore/exo/actions/runs/20467656904/job/58815275013
When Python's multiprocessing spawns child processes on macOS (using the
"spawn" method), it also spawns helper processes like the resource tracker
by executing:
./frozen_app -c "from multiprocessing.resource_tracker import main; main()"
A frozen PyInstaller app doesn't understand `-c` natively - it just runs
main(). This causes the resource tracker to fail silently.
This adds a minimal `-c` handler that intercepts the flag, extracts the
inline code, and exec()s it before main() runs. This is required for the
Process() spawn in runner_supervisor.py to work correctly in the DMG.
Note that the pyinstaller docs say `freeze_support` is supposed to make
this work, but it doesn't.
Test plan:
Hardware setup: 3x Mac Studio M3 Ultra connected all-to-all with TB5
- Built a DMG[0].
- Installed on the Macs.
- Started an instance. Got an error this time in ~/.exo/exo.log. The
last DMG from main doesn't show anything when an instance starts, this
now shows the errors.
[0] https://github.com/exo-explore/exo/actions/runs/20464409279/job/58804485197
Model loading fails silently when running from the DMG-packaged app,
despite working correctly with `uv run exo`. The bundled app spawns
child processes for model inference via multiprocessing, but these
processes fail to start in a frozen (PyInstaller) environment.
Add `freeze_support()` which is required for multiprocessing to work
in frozen applications.
Test plan:
Hardware setup: 3x Mac Studio M3 Ultra connected all-to-all with TB5
- Built a DMG using a modified .github/workflows/build-app.yml[0] to avoid
publishing it.
- Installed on all 3 Macs, replacing the existing Exo.
- Downloaded Llama 3.3 70B (FP16).
- Downloaded Qwen3 Coder 235B A22B (8-bit).
Things that work now but didn't on the previous app:
- Topology looks good, previously there was no discovery.
What didn't work:
- Started an instance with Pipeline + MLX Ring + 3 Nodes. Failed.
- Started an instance with Tensor + MLX RDMA + 2 Nodes. Failed.
Will continue debugging the instance starting issues separately.
[0] https://github.com/exo-explore/exo/actions/runs/20461320368
As I've been working on the .dmg, it's become clear we need a way to
test changes to the app. It's too hard to reproduce the full DMG locally
to be reasonable and much more convenient to test if it's signed.
Add a feature to the build-app workflow where if you push specifically
to the `test-app` branch it'll perform a build. The version is stubbed
to `0.0.0-alpha.0`, which is about as low as it gets in semver so you'll
always update away from it automatically with Sparkle. The resulting DMG
won't be pushed to S3 but will be uploaded as a GitHub Actions artifact.
I've been using similar commits to this for a while for testing. It's
worked well and not interfered with auto updating at all.
Test plan:
- Pushed this change to `test-app`.
- Generated action at
https://github.com/exo-explore/exo/actions/runs/20447213358/job/58752909332
- Installed the DMG on a Mac. It worked as intended.
## Motivation
README looks weird after last update.
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
I actually checked the file on GitHub this time.
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
Documentation will make contribution easier and communicate our
development philosophy and decision process. Closes#967
## Changes
Added `architecture.md` to docs/ and moved the images out of docs and
into their own docs/imgs/ folder
## Motivation
Addresses #974
```
INFO: pip is looking at multiple versions of exo to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement exo-pyo3-bindings (from exo) (from versions: none)
ERROR: No matching distribution found for exo-pyo3-bindings
```
## Changes
Describes Rust dependency for building from source
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
Tested locally and runs after this setup without exo-pyo3-bindings error
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
Fixes#960
The macOS app was incorrectly using port 8000 instead of the default
exo API port 52415. This caused confusion as the README correctly
documents port 52415 but the app was connecting to a different port.
## Motivation
Users need to know what **prerequisites** they need in order to run exo.
Simple addition to docs prevents future raised issues.
## Changes
Updated ``README.md``:
- to include installation instructions for
**[uv](https://github.com/astral-sh/uv)** and
**[macmon](https://github.com/vladkens/macmon)**.
Updated ``CONTRIBUTING.md``:
- to verify these prerequisites are met before starting development.
- Standardized on brew installation instructions for macOS users to keep
the guide simple.
## Why It Works
By listing these prerequisites upfront, users will set up their
environment correctly before attempting to run exo.
## Test Plan
### Manual Testing
MacBook Pro M4
- Verified that ``uv`` and ``macmon`` were missing initially, causing
failures
- after installing them via brew (as documented), uv run exo starts
successfully.
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
---------
Co-authored-by: Evan Quiney <evanev7@gmail.com>
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
Made a mistake on the merge of the last PR.
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
fix(downloads): use certifi for robust SSL certificate verification
## Description
This change updates the SSL context creation in \`download_utils.py\` to
explicitly use the \`certifi\` CA bundle. This ensures that the
application has access to a reliable, up-to-date set of root
certificates, which is critical for verifying SSL connections to
external services like Hugging Face.
## Problem
On macOS environments (and potentially others), Python's default SSL
context often fails to locate the system's root certificates. This leads
to \`aiohttp.client_exceptions.ClientConnectorCertificateError\` errors
when attempting to download models.
## Solution
By passing \`cafile=certifi.where()\` to
\`ssl.create_default_context()\`, we force the application to use the
trusted certificate store provided by the \`certifi\` package. This is a
standard best practice for cross-platform Python applications and
resolves the verification failure.
Evan
Sami Khan
Jake Hillion
rltakashige
majiayu000
Nightguarder
Heath Dutton🕴️
Alex Cheema
Olimbek Nizomov