17. pyxc: One-Step Executables

What I Am Building

Chapter 16 added --emit obj, --emit asm, and --emit llvm-ir. Producing a runnable binary from a pyxc program still needed an external tool:

pyxc --emit obj -o program.o program.pyxc
clang program.o runtime.c -o program   # ← still needed clang
./program

I want to remove that second step this chapter. After it:

pyxc --emit exe -o program program.pyxc
./program

No clang, no runtime.c, no separate link invocation. One command.

Source Code

git clone --depth 1 https://github.com/alankarmisra/pyxc-llvm-tutorial
cd pyxc-llvm-tutorial/code/chapter-17

Grammar

No grammar changes. The language is unchanged — this is purely a compiler-driver extension.

The Design

The key insight I'm leaning on: LLVM ships LLD — a full production linker — as a C++ library. Instead of shelling out to clang or ld, I can call lld::macho::link (or lld::elf::link on Linux) directly in-process. The pipeline becomes:

.pyxc → compile → temp .o
.o inputs → pass through
synthesize runtime .o (printd, putchard)
─────────────────────────────────────────
LLD links all .o files into executable

The runtime functions printd and putchard are generated as LLVM IR and emitted to a temporary .o — no runtime.c or external compiler needed.

What Changes

I'm adding six new pieces on top of chapter 16:

cl::list<string> InputFiles — the positional argument changes from a single string to a list, enabling multiple inputs.
EmitRuntimeObject — synthesizes printd and putchard as LLVM IR, emits them to a temporary .o.
CompileFileToObject — per-file compilation: open → lex → parse → codegen → .o.
PrepareFileModeModule — refactored out of EmitFileMode: the shared logic that builds __pyxc.global_init, registers it in llvm.global_ctors, and wraps main().
LinkExecutable + FindMacOSSDKRoot — LLD-as-library dispatch with platform-aware system library detection.
EmitExecutable — the new orchestrator that wires all of the above together.

One more change ties these together but isn't new behavior on its own: EmitModuleToFile (from chapter 16) now takes a Module*, an EmitKind, and an output path as parameters instead of reading them off file-scope globals. It has to — this chapter calls it three separate times against three different modules (the runtime object, each compiled .pyxc file, and the final wrapped file-mode module), and a single global-module version can't do that.

Accepting Multiple Input Files

Through chapter 16, the positional argument was a single optional cl::opt:

// Chapter 16
static cl::opt<std::string> InputFile(cl::Positional, ...);

I need it to be a list instead, so the driver can accept any number of .pyxc and .o files:

// Chapter 17
static cl::list<std::string>
    InputFiles(cl::Positional, cl::desc("[inputs]"), cl::ZeroOrMore,
               cl::cat(PyxcCategory));

I derive IsRepl from whether the list is empty now:

IsRepl = InputFiles.empty();

The --emit exe path also enforces the multi-input rule:

} else if (EmitKindOption == "exe") {
  EmitMode = EmitKind::Executable;
  if (OutputFile.empty() && InputFiles.size() > 1) {
    fprintf(stderr, "Error: multiple inputs require -o\n");
    return -1;
  }
  if (!OutputFile.empty())
    EmitOutputPath = OutputFile.getValue();
}

--emit llvm-ir, --emit asm, and --emit obj still require exactly one input and are unchanged.

Output Naming

When -o is omitted and there's exactly one input, I want the output to be the input with its extension stripped — and .exe appended on Windows:

static string DefaultExecutablePath(StringRef InputPath) {
  SmallString<256> OutputPath(InputPath);
  sys::path::replace_extension(OutputPath, "");
  string Result = OutputPath.str().str();
#ifdef _WIN32
  Result += ".exe";
#endif
  return Result;
}

sys::path::replace_extension handles both .pyxc and .o inputs uniformly: foo.pyxc → foo, mylib.o → mylib.

Synthesizing the Runtime

In --emit obj mode (chapter 16), I linked test binaries against runtime.c to get printd and putchard. For --emit exe mode, I want pyxc to synthesize those functions itself — no C file, no external compiler:

static bool EmitRuntimeObject(const string &ObjectPath) {
  LLVMContext Context;
  auto RuntimeModule = make_unique<Module>("pyxc.runtime", Context);
  auto *DoubleType = Type::getDoubleTy(Context);
  auto *Int32Type = Type::getInt32Ty(Context);
  auto *PointerType = llvm::PointerType::get(Context, 0);

  FunctionType *PrintfType =
      FunctionType::get(Int32Type, {PointerType}, true);
  Function *Printf = Function::Create(
      PrintfType, Function::ExternalLinkage, "printf", RuntimeModule.get());
  FunctionType *PutcharType =
      FunctionType::get(Int32Type, {Int32Type}, false);
  Function *Putchar = Function::Create(
      PutcharType, Function::ExternalLinkage, "putchar", RuntimeModule.get());

  FunctionType *PrintdType =
      FunctionType::get(DoubleType, {DoubleType}, false);
  Function *Printd = Function::Create(
      PrintdType, Function::ExternalLinkage, "printd", RuntimeModule.get());
  {
    BasicBlock *Entry = BasicBlock::Create(Context, "entry", Printd);
    IRBuilder<> RuntimeBuilder(Entry);
    auto *Format = RuntimeBuilder.CreateGlobalString("%f\n", "format");
    Value *Zero = ConstantInt::get(Int32Type, 0);
    Value *FormatPointer = RuntimeBuilder.CreateInBoundsGEP(
        Format->getValueType(), Format, {Zero, Zero}, "format.pointer");
    RuntimeBuilder.CreateCall(Printf, {FormatPointer, Printd->getArg(0)});
    RuntimeBuilder.CreateRet(ConstantFP::get(Context, APFloat(0.0)));
  }

  FunctionType *PutchardType =
      FunctionType::get(DoubleType, {DoubleType}, false);
  Function *Putchard = Function::Create(PutchardType,
      Function::ExternalLinkage, "putchard", RuntimeModule.get());
  {
    BasicBlock *Entry = BasicBlock::Create(Context, "entry", Putchard);
    IRBuilder<> RuntimeBuilder(Entry);
    Value *Character = RuntimeBuilder.CreateFPToUI(
        Putchard->getArg(0), Int32Type, "character");
    RuntimeBuilder.CreateCall(Putchar, {Character});
    RuntimeBuilder.CreateRet(ConstantFP::get(Context, APFloat(0.0)));
  }

  return EmitModuleToFile(RuntimeModule.get(), EmitKind::Object, ObjectPath);
}

The key points:

I use a fresh, independent LLVMContext and Module — separate from the user's program module. This isolates the runtime from user IR.
printf and putchar are declared as extern (they come from libc at link time).
printd and putchard are defined with ExternalLinkage so the linker can resolve the extern def printd(x) declarations in user code.
EmitRuntimeObject ends by calling EmitModuleToFile to write a real .o to a temp path. That .o gets added to the link list alongside user objects.

Per-File Compilation

I want each .pyxc input to go through its own full parse-codegen-emit cycle:

static bool CompileFileToObject(const string &Path, const string &ObjectPath,
                                bool *HasMain) {
  if (!OpenInputFile(Path))
    return false;

  ResetLexerState();
  ResetParserStateForFile();
  InitializeModuleAndManagers(false);
  IsRepl = false;
  getNextToken();
  FileModeLoop();
  CloseInputFile();

  if (HasMain)
    *HasMain = FunctionSignatures.find("main") != FunctionSignatures.end();
  if (!PrepareFileModeModule())
    return false;
  return EmitModuleToFile(TheModule.get(), EmitKind::Object, ObjectPath);
}

ResetLexerState and ResetParserStateForFile clear the persistent lexer and parser state between files, so each .pyxc compiles independently. This is what makes multi-file compilation safe — a global declared in a.pyxc doesn't silently bleed into b.pyxc.

Shared Codegen Finishing

In chapter 16, EmitFileMode contained all the logic for building __pyxc.global_init, validating main, and wrapping it. I want both the --emit obj path and the new per-file --emit exe path to share that, so I refactor it out into PrepareFileModeModule:

static bool PrepareFileModeModule() {
  if (!FileTopLevelStatements.empty()) {
    auto Block =
        make_unique<BlockStatementNode>(std::move(FileTopLevelStatements));
    auto Signature = make_unique<FunctionSignatureNode>(
        "__pyxc.global_init", vector<string>());
    auto FunctionDefinition = make_unique<FunctionDefinitionNode>(
        std::move(Signature), std::move(Block));

    bool SavedInGlobalInit = InGlobalInit;
    InGlobalInit = true;
    if (auto *FunctionIR = FunctionDefinition->codegen()) {
      InGlobalInit = SavedInGlobalInit;
      if (ShouldDumpIR())
        FunctionIR->print(errs());
      AddGlobalConstructor(FunctionIR);
    } else {
      InGlobalInit = SavedInGlobalInit;
      return false;
    }
  }

  auto Main = FunctionSignatures.find("main");
  if (Main != FunctionSignatures.end() &&
      Main->second->getNumParameters() != 0) {
    fprintf(stderr, "Error: main() must take no arguments\n");
    return false;
  }

  // A Pyxc main returns double. I preserve it under an internal name and
  // expose the conventional i32 main expected by native linkers.
  if (auto *UserMain = TheModule->getFunction("main")) {
    if (UserMain->getReturnType()->isDoubleTy()) {
      UserMain->setName("__pyxc.user_main");
      FunctionType *WrapperType =
          FunctionType::get(Type::getInt32Ty(*TheContext), false);
      Function *Wrapper = Function::Create(
          WrapperType, Function::ExternalLinkage, "main", TheModule.get());
      BasicBlock *Entry = BasicBlock::Create(*TheContext, "entry", Wrapper);
      IRBuilder<> WrapperBuilder(Entry);
      WrapperBuilder.CreateCall(UserMain);
      WrapperBuilder.CreateRet(
          ConstantInt::get(Type::getInt32Ty(*TheContext), 0));
    }
  }

  return true;
}

EmitFileMode now calls PrepareFileModeModule() followed by EmitModuleToFile. CompileFileToObject does the same. The logic lives in exactly one place. I save and restore InGlobalInit the same way I started doing back in chapter 15, rather than hard-resetting it to false — same reasoning as then: restoring whatever it actually was is a safer habit than assuming.

Linking with LLD

LinkExecutable dispatches to the right LLD driver based on the host triple:

static bool LinkExecutable(const vector<string> &Inputs,
                           const string &OutputPath) {
  Triple TargetTriple(sys::getDefaultTargetTriple());
  vector<string> ArgumentStorage;
  auto AddArgument = [&](const string &Argument) {
    ArgumentStorage.push_back(Argument);
  };

  if (TargetTriple.isOSDarwin()) {
    AddArgument("ld64.lld");
    AddArgument("-arch");
    AddArgument(TargetTriple.getArchName().str());
    AddArgument("-o");
    AddArgument(OutputPath);
    string SDKRoot = FindMacOSSDKRoot();
    if (!SDKRoot.empty()) {
      AddArgument("-syslibroot");
      AddArgument(SDKRoot);
      AddArgument("-L" + SDKRoot + "/usr/lib");
      AddArgument("-L" + SDKRoot + "/usr/lib/system");
      string SDKVersion = FindMacOSSDKVersion();
      AddArgument("-platform_version");
      AddArgument("macos");
      AddArgument(SDKVersion);
      AddArgument(SDKVersion);
    }
    for (const auto &InputPath : Inputs)
      AddArgument(InputPath);
    AddArgument("-lSystem");

    vector<const char *> Arguments;
    for (auto &Argument : ArgumentStorage)
      Arguments.push_back(Argument.c_str());
    return lld::macho::link(Arguments, outs(), errs(), false, false);
  }

  if (TargetTriple.isOSLinux()) {
    AddArgument("ld.lld");
    AddArgument("-o");
    AddArgument(OutputPath);
    for (const auto &InputPath : Inputs)
      AddArgument(InputPath);
    AddArgument("-lc");
    AddArgument("-lm");
    vector<const char *> Arguments;
    for (auto &Argument : ArgumentStorage)
      Arguments.push_back(Argument.c_str());
    return lld::elf::link(Arguments, outs(), errs(), false, false);
  }

  if (TargetTriple.isOSWindows()) {
    AddArgument("lld-link");
    AddArgument("/OUT:" + OutputPath);
    for (const auto &InputPath : Inputs)
      AddArgument(InputPath);
    vector<const char *> Arguments;
    for (auto &Argument : ArgumentStorage)
      Arguments.push_back(Argument.c_str());
    return lld::coff::link(Arguments, outs(), errs(), false, false);
  }

  fprintf(stderr, "Error: unsupported target for --emit exe\n");
  return false;
}

I don't pass crt1.o/crti.o/crtn.o the way a traditional Unix linker invocation would. Those are ELF startup objects; on macOS, dyld and libSystem handle process startup on their own, and a Mach-O link has no use for them.

The LLD API is the same on every platform: an array of const char* arguments (identical to what you'd pass on the command line), plus output/error streams and two flags — exitEarly (stop on first error) and disableOutput (dry-run). The return value is true on success.

This is a key architectural choice: LLD is called as a library, not as a subprocess. There's no fork/exec, no temporary shell script, no PATH lookup. If the library is linked into the pyxc binary, it's available.

SDK Detection

LLD's Mach-O linker needs a sysroot to find system headers and libSystem, plus a version string for the -platform_version flag. Both of these are things Xcode's own xcrun tool already knows how to answer correctly, so I lean on it rather than re-deriving the logic myself:

static string RunXcrun(const char *Arguments) {
  string Command = string("xcrun ") + Arguments + " 2>/dev/null";
  FILE *Pipe = popen(Command.c_str(), "r");
  if (!Pipe)
    return "";
  char Buffer[512];
  string Result;
  while (fgets(Buffer, sizeof(Buffer), Pipe))
    Result += Buffer;
  pclose(Pipe);
  while (!Result.empty() &&
         (Result.back() == '\n' || Result.back() == '\r' ||
          Result.back() == ' '))
    Result.pop_back();
  return Result;
}

FindMacOSSDKRoot tries an explicit override first, then asks xcrun:

static string FindMacOSSDKRoot() {
  if (const char *SDKRoot = getenv("SDKROOT"))
    return string(SDKRoot);
  string Result = RunXcrun("--sdk macosx --show-sdk-path");
  if (!Result.empty() && sys::fs::exists(Result))
    return Result;
  return "";
}

static string FindMacOSSDKVersion() {
  string Result = RunXcrun("--sdk macosx --show-sdk-version");
  if (!Result.empty())
    return Result;
  Triple TargetTriple(sys::getDefaultTargetTriple());
  VersionTuple Version = TargetTriple.getOSVersion();
  if (Version.getMajor()) {
    ostringstream Stream;
    Stream << Version.getMajor() << "." << Version.getMinor().value_or(0);
    return Stream.str();
  }
  return "11.0";
}

If xcrun doesn't resolve a version, I fall back to the OS version encoded in the host triple. Either way, the goal is the same: match whatever SDK version LLVM itself considers active, so ld64.lld's -platform_version check doesn't warn about a mismatch.

The -platform_version macos <min> <sdk> flag is required by the Mach-O linker to set the LC_BUILD_VERSION load command. Without it, the linker produces a warning or errors depending on the LLD version.

The Orchestrator

static bool EmitExecutable() {
  vector<string> ObjectFiles;
  vector<string> TemporaryFiles;
  bool SawMain = false;
  bool SawObjectInput = false;

  auto RemoveTemporaryFiles = [&]() {
    for (const auto &Path : TemporaryFiles)
      sys::fs::remove(Path);
  };

  for (const auto &InputPath : InputFiles) {
    if (IsPyxcInput(InputPath)) {
      int FileDescriptor = -1;
      SmallString<128> TemporaryPath;
      if (auto ErrorCode = sys::fs::createTemporaryFile(
              "pyxc", "o", FileDescriptor, TemporaryPath)) {
        fprintf(stderr, "Error: could not create temporary file: %s\n",
                ErrorCode.message().c_str());
        RemoveTemporaryFiles();
        return false;
      }
      if (FileDescriptor != -1)
        close(FileDescriptor);

      string ObjectPath = TemporaryPath.str().str();
      TemporaryFiles.push_back(ObjectPath);
      bool FileHasMain = false;
      if (!CompileFileToObject(InputPath, ObjectPath, &FileHasMain)) {
        RemoveTemporaryFiles();
        return false;
      }
      SawMain = SawMain || FileHasMain;
      ObjectFiles.push_back(ObjectPath);
      continue;
    }

    if (IsObjectInput(InputPath)) {
      ObjectFiles.push_back(InputPath);
      SawObjectInput = true;
      continue;
    }

    fprintf(stderr, "Error: unsupported input '%s'\n", InputPath.c_str());
    RemoveTemporaryFiles();
    return false;
  }

  if (!SawMain && !SawObjectInput) {
    fprintf(stderr, "Error: main() not found\n");
    RemoveTemporaryFiles();
    return false;
  }

  int RuntimeDescriptor = -1;
  SmallString<128> RuntimePath;
  if (auto ErrorCode = sys::fs::createTemporaryFile(
          "pyxc_runtime", "o", RuntimeDescriptor, RuntimePath)) {
    fprintf(stderr, "Error: could not create runtime object: %s\n",
            ErrorCode.message().c_str());
    RemoveTemporaryFiles();
    return false;
  }
  if (RuntimeDescriptor != -1)
    close(RuntimeDescriptor);

  string RuntimeObjectPath = RuntimePath.str().str();
  TemporaryFiles.push_back(RuntimeObjectPath);
  if (!EmitRuntimeObject(RuntimeObjectPath)) {
    RemoveTemporaryFiles();
    return false;
  }
  ObjectFiles.push_back(RuntimeObjectPath);

  if (EmitOutputPath.empty())
    EmitOutputPath = DefaultExecutablePath(InputFiles.front());
  bool Linked = LinkExecutable(ObjectFiles, EmitOutputPath);
  RemoveTemporaryFiles();
  return Linked;
}

A few things I want to call out here:

sys::fs::createTemporaryFile takes a file descriptor out-param in this overload. It actually creates and opens the file (so the name is guaranteed unique and reserved), and I have to close() that descriptor myself once I'm done needing it open — I only wanted the path, not a live handle.
RemoveTemporaryFiles is a local lambda, not a separate function. Every error path in this function needs to remove whatever temp files exist so far before returning false. It only ever gets used inside EmitExecutable, and the lambda already captures exactly the state it needs by reference.
The main() check. Nothing upstream of this function checks whether an entry point exists anywhere, so a missing main would otherwise silently make it all the way to the linker and fail there with a much less clear error. SawObjectInput exists so that "a .o might define main, I can't inspect it without disassembling it" doesn't turn into a false rejection — if any input is a pre-built object, I let the linker be the judge.
EmitOutputPath.empty() uses InputFiles.front() directly. By the time EmitExecutable runs, ProcessCommandLine has already guaranteed InputFiles is non-empty (it's how IsRepl gets set), so there's no need to re-check emptiness here.

New Headers and Build Changes

The new headers this chapter:

#include "lld/Common/Driver.h"       // lld::macho::link, lld::elf::link, lld::coff::link
#include "llvm/Support/VersionTuple.h"  // VersionTuple for OS version extraction

The three LLD driver macros need to appear at file scope to register the drivers:

LLD_HAS_DRIVER(elf)
LLD_HAS_DRIVER(coff)
LLD_HAS_DRIVER(macho)

CMakeLists.txt finds LLD as its own CMake package alongside LLVM, and links its libraries explicitly — llvm_map_components_to_libnames only resolves LLVM's own components, not LLD's:

find_package(LLD REQUIRED CONFIG HINTS "${LLVM_DIR}/../lld" NO_DEFAULT_PATH)
include_directories(SYSTEM ${LLD_INCLUDE_DIRS})
...
target_link_libraries(pyxc PRIVATE ${LLVM_LIBS} lldCommon lldELF lldMachO lldCOFF)

Known Limitations

Target is always the host. The SDK is detected for the machine running pyxc. Cross-compilation is not supported.

main() always exits 0. The int main() wrapper returns 0 unconditionally. There is no way to set a non-zero exit code from a pyxc program yet.

Runtime is always linked in. Even if a program never calls printd or putchard, the runtime object is included in every --emit exe link. A future chapter could strip unreferenced symbols with LTO.

SDK detection depends on xcrun being on PATH. FindMacOSSDKRoot shells out to xcrun and returns an empty string if that fails, in which case LinkExecutable skips the -syslibroot and -platform_version arguments entirely. If you have a non-standard Xcode installation and xcrun isn't resolving correctly, set the SDKROOT environment variable to the SDK path before running pyxc — that override is checked before xcrun is ever invoked.

No debug information. The emitted executables contain no DWARF. Debuggers cannot map instructions back to pyxc source lines.

Multi-file globals are independent. Each .pyxc file gets its own __pyxc.global_init. If two files declare globals with the same name, the linker reports a duplicate-symbol error. There's no cross-file global sharing.

Try It

The minimal case

cat hello.pyxc

extern def printd(x)
def main():
    printd(42)

pyxc --emit exe -o hello hello.pyxc
./hello

42.000000

Default output name

pyxc --emit exe hello.pyxc   # produces ./hello
./hello

Global init runs before main

extern def printd(x)
var total = 0
for var i = 1, i < 6, 1:
    total = total + i
def main():
    printd(total)   # 15.000000

Linking two files

# lib.pyxc
def add(a, b): return a + b

# main.pyxc
extern def printd(x)
extern def add(a, b)
def main():
    printd(add(3, 4))

pyxc --emit exe -o prog main.pyxc lib.pyxc
./prog

7.000000

Linking a pre-built object

pyxc --emit obj -o lib.o lib.pyxc
pyxc --emit exe -o prog main.pyxc lib.o
./prog

Inspect the IR before linking

pyxc --dump-ir --emit exe -o prog main.pyxc

Build and Run

cd code/chapter-17
cmake -S . -B build && cmake --build build
./build/pyxc --emit exe -o hello hello.pyxc
./hello

llvm-lit -v test/

What's Next

Chapter 18 adds a static type system.

Need Help?

Build issues? Questions?

GitHub Issues: Report problems
Discussions: Ask questions

Include:

Your OS and version
Full error message
Output of cmake --version, ninja --version, and llvm-config --version

I'll help you figure it out.

< 16. pyxc: Emitting Native Code

18. pyxc: A Static Type System >