Feel free to do your own analysis -- my informal experiments backs this up, though. I see worse results when I try to do anything in an unpopular language.
It makes sense, needing to train the model on things that aren't already in its weighs takes up valuable context. Until we have models that update their weights based on what they've seen in their recent sessions and learn like people, this will be a problem.
For now, though, between the results I'm seeing here, and the lack of need to look at code, I think this kills off any reason for me to use less popular languages.
There is a Common Lisp pro you are not seeing and that is that it has by far the best OOB debuggability/introspectability (especially when using SBCL) out of any practical language, while still having great performance
Your attempt was probably pretty, ahem, weak. How much support would you say you gave your goes, before you threw in the towel?
I don't think there's really ever a downside to leaning in and making use of a language or system that works for you. Trying to tell people they should just use the popular thing is, imo, bad advice to turn hackers and experimenters into boring people.
A day or so for each of the oddball languages; again, I'm still waiting for an argument on why there's any value here, since the entire point of an agentic system like this is that I don't have to read the code. Experiment with the AI, sure, but you've got a pretty high burden of proof to show that AI is going to pick it up without a high per-prompt token cost.
AI changes the constraints here for now, since it can't permanently learn things. I'm waiting until that changes, but right now it's better to use what it knows out of the box if you want good results.
A better language doesn't buy me anything other than performance; the reason to stick an AI in here is to remove interactions with the code. I don't care what the AI chooses to use, as long as it gets results.
is it that llms write "better" typescript than let's say elixir because it has seen more of it..? or is it that you're relying on something like effect-ts to keep llms from tripping over even small things?
coincidentally, "good code" in popular lang is rarely directly attributed to only that part; and it's also about the underlying principles it tries to follow in the code... another example; is it typescript that's good, or are "types" inherently making things/feedback loops easier to reason about in llms? (only using ts here for all example because it's probably one of the most "trained on" pl)
If it's samey regardless of language, isn't that in contradiction to your original theory? " .. the most important feature for agent performance is the popularity of the language .. "
No, not really. It's a similar flavor of output, but there's less iterations to get a correct result. The training is mostly about reducing error rates on generation.
I don't get how. I'm not interacting with the lisp, and the agents don't really get frustrated with slow compilation times or anything, and are perfectly adept at debugging.
The best that people have said about lisp is that evidence LLMs perform worse with it is weak.
I see you're interested in avoiding the need to read any code. It might surprise you to learn that autolith is very capable at reading and updating it's own code. The captured sessions at the linked page are three examples of this.
There are many other confounding factors here, the type of prompting, how familiar you are with the language idioms, the context you gave, random bad quality runs, etc.
Complement AI (https://complement.ai) | Member of Technical Staff | San Francisco or New York City | ONSITE | Full-time | $150K - $300K + Equity
Complement builds AI that autonomously operates factories. We connect the fragmented systems inside a factory (ERP, MES, PLCs, spreadsheets, etc) into a unified data layer, then deploy AI agents that optimize decisions across the whole operation for profit.
As a Member of Technical Staff, you'll work across the stack connecting factory data sources, building simulations to optimize scheduling, embedding with factory operators, and designing interfaces that surface recommendations. We care about what you've shipped under ambiguity more than which of those you specialize in.
Sensationalist title, the actual behavior described in the issue this links to is:
Rust language doesn't offer lookaround regex features
This is due to the standard library regex implementation not supporting lookaround, not the language itself. Other crates, such as fancy-regex [1] do support this via backtracking.
This might be of interest to others: Last night I stumbled across Hornbeam, a library in a similar vein from the author of Gunicorn that handles WSGI / ASGI apps as well as a specific wrapper for ML inference
Heh, that's my PR. Initially I thought it would be a trivial change, but then I realized I hadn't considered how it should interact with MDM / device posture functionality - these aren't features I'm personally using with the Android client, but are understandably important to enterprises.
I still hope to get back to that and try to get it to a state where it can be merged, but I need to figure out how to test the MDM parts of it properly, and ideally get a bit of guidance from the tailscale team on how it should work/is my implementation on the right track (think I had some open questions around the UI as well)
Let's stop moving the goalposts. Open source has a specific definition, and "they merge whatever code I want them to" isn't part of it. Just fork the client, compile it, and run it yourself.
I think you misunderstand how mesh VPNs work. Their primary purpose is as a control plane - introducing peers to each other so they can either communicate directly or via a relay (eg DERP) via per-node encryption. They should have no overhead compared to a single point to point encrypted tunnel like wireguard, because the “mesh” features are not in the data path.
The only real difference here is how the vpn product implements wireguard: userspace or kernel space, and how well tuned that implementation is. It might make sense to compare wireguard implementations, but (afaik) all are using one of several open source ones. Tailscale did some work to improve performance that they blogged about here https://tailscale.com/blog/more-throughput
Neither Nebula nor ZeroTier is based on Wireguard.
What they compare in the article are systems that provide some form of ACL, which is why bare Wireguard is not included. That means there are features in the data path that could have significant performance implications versus a simple tunnel. The impact of using ACL features isn't really a focus of the presented benchmarks, but they do mention a separate test of using iptables to bolt on access controls.
It looks like the host kernel is not in full control – there is a EL2-level hypervisor, pKVM [1] that is actually the highest-privilege domain. This is pretty similar to the Xen architecture [1] where the dom0 linux os in charge of managing the machine is running as a guest of the hypervisor.
No, KVM is also a type 1 hypervisor but it doesn't attempt (with the exception of pKVM and of hardware protection features like SEV, neither of which is routinely used by cloud workloads) to protect the guest from a malicious host.
KVM is a type 2 hypervisor as the "Dom 0" kernel has full HW access. Other guests are obviously isolated as configured and are like special processes to userspace.
It gets a bit blurry on AArch64 without and with VHE (Virtual Host Extensions) as without VHE (< ARMv8.1) the kernel runs in EL1 ("kernel mode") most of the time and escalates to EL2 ("hypervisor mode") only when needed, but with VHE it runs at EL2 all the time. (ref. https://lwn.net/Articles/650524/)
No, "type 2" is defined by Goldberg's thesis as "The VMM runs on an extended host [53,75], under the host operating system", where:
* VMM is treated as synonymous with hypervisor
* "Extended host" is defined as "A pseudo-machine [99], also called an extended machine [53] or a user machine [75], is a composite machine produced through a combination of hardware and software, in which the machine's apparent architecture has been changed slightly to make the machine more convenient to use. Typically these architectural changes have taken the form of removing I/O channels and devices, and adding system calls to perform I/O and and other operations"
In other words, type 1 ("bare machine hypervisor") runs in supervisor mode and type 2 runs in user mode. QEMU running in dynamic binary translation mode is a type 2 VMM.
KVM runs on a bare machine, but it
delegates some services to a less privileged component such as QEMU or crosvm or Firecracker. This is not a type 2 hypervisor, it is a type 1 hypervisor that follows security principles such as privilege separation.
KVM is pretty much the only hypervisor that cloud vendors use these days.
So it's true that "most cloud workloads run on type 1 hypervisors" (KVM is one) but not that most cloud vendors/workloads run on microkernel-like hypervisors, with the exception of Azure.
Then can you explain how cloud workloads is the revenge of the microkernel, since there is exactly 1 major cloud provider that uses a microkernel-like hypervisor?
By not running monolithic kernels on top of bare metal, rather virtualized, or even better with nested virtualization, thus throwing out the door all the supposedly performance advantages in the usual monolithic vs microkernel flamewar discussions, regarding context switching.
Additionally to make it to the next level, they run endless amount of container workloads.
This is explicitly called out as only weakly supported in that blog post: