Open
Conversation
Julia's stacked environments (LOAD_PATH / @v#.#) allow loading dev tools alongside project deps, but they use separate manifests. This can lead to incompatible versions being loaded, wasted precompilation, and [compat] violations. Workspaces solve this with a shared manifest, but require explicit [workspace] entries in every project. An overlay project bridges this gap: a globally-configured Project.toml whose deps get merged into every active environment as if it were a workspace member. Dev tools like Debugger, JET, and Cthulhu can be listed once in the overlay and resolved together with whatever project you're working on — sharing a single manifest and respecting [compat]. Configuration (checked in order, first wins): - Pkg.OVERLAY_PROJECT[] = "~/.julia/overlay" - ENV["JULIA_PKG_OVERLAY"] = "@devtools" Paths starting with @ are expanded via Base.load_path_expand, so "@devtools" resolves to ~/.julia/environments/devtools/Project.toml. The implementation injects the overlay into env.workspace in the EnvCache constructor (~5 lines), which means all downstream code — dep loading, compat intersection, resolve hashing, precompilation — works automatically since it already iterates over workspace members. When the overlay project itself is active (pkg> activate @devtools), only `add` and `rm` are supported — they write directly to the overlay's Project.toml without resolution or manifest updates. Other operations (develop, update, pin, free) error with a clear message. The REPL prompt shows [overlay] when the overlay project is active. Also fixes print_diff crash when both old and new PackageSpecs are not instantiated (no manifest entry), and uses absolute paths in workspace status output when the member is outside the project tree. Closes #4637 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Member
|
Neat! Would be good to add a test that preference merging works with an overlay project. |
Member
Author
Yes, I forgot to mention that. I am actually not sure it works with workspaces? |
Member
Author
|
Things that can be discussed:
|
Member
|
Very cool!
I'm not clear how it avoids this. Can you explain?
Feels like an extra layer of indirection. We could just pick |
Member
Author
It feels to me like you would want this path to be customizable to easily be able to swap between different overlay projects (or to disable it) etc. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototyped the overlay project feature I mentioned in #4637 (comment) with Claude.
This still needs code loading support to merge in the overlay project into the workspace.
The goal here is to avoid using the environment stack for dev tools (which has a seprate manifest) and instead always resolve the dev tools together with your active project.
This does not help "torn environments" if you load packages, do package operations and then load more packages.
Also, it is a bit ugly that if you share a manifest with someone they get all these devtool packages in there, maybe there should be two manifests, one that is everything reachable from the actual active project and one that is the "orphan" dpeendencies assuming the overlay environment would be removed...
TODO:
Usage:
🤖 description:
Julia's stacked environments (LOAD_PATH /
@v#.#) allow loading dev tools alongside project deps, but they use separate manifests. This can lead to incompatible versions being loaded, wasted precompilation, and [compat] violations. Workspaces solve this with a shared manifest, but require explicit [workspace] entries in every project.An overlay project bridges this gap: a globally-configured Project.toml whose deps get merged into every active environment as if it were a workspace member. Dev tools like Debugger, JET, and Cthulhu can be listed once in the overlay and resolved together with whatever project you're working on — sharing a single manifest and respecting [compat].
Configuration (checked in order, first wins):
Paths starting with @ are expanded via Base.load_path_expand, so "@devtools" resolves to ~/.julia/environments/devtools/Project.toml.
The implementation injects the overlay into env.workspace in the EnvCache constructor (~5 lines), which means all downstream code — dep loading, compat intersection, resolve hashing, precompilation — works automatically since it already iterates over workspace members.
When the overlay project itself is active (pkg> activate @devtools), only
addandrmare supported — they write directly to the overlay's Project.toml without resolution or manifest updates. Other operations (develop, update, pin, free) error with a clear message. The REPL prompt shows [overlay] when the overlay project is active.