SciLuigi adds a simple level of audit logging to Luigi [1] (See the AuditTrailHelpers mixin in https://github.com/pharmbio/sciluigi/blob/master/sciluigi/au... for details), together with an API that makes it much easier to work do changes to the workflow dependency graph, when you're in development/exploratory phase.
Personally moved to improve on the ideas in SciLuigi, but replacing Luigi with Go's concurrency primitives, in SciPipe [2].
In SciPipe, an accompanying ".audit.json" file, containing the full history of commands and parameters used to create every single output file of the workflow (even intermediate ones), to help keep track of how each file was produced.
Among lightweight solutions, the popular ones these days seem to be NextFlow [3], Snakemake [4], BPipe [5] and others. You'd really have to check out the "awesome pipelines" [6] list, to get any kind of overview.
When more infrastructure setup is feasible, I think PachyDerm [7] (mentioned elsewhere in the thread) looks really powerful with its "Git for data" approach. Something I'd wish to use as an overarching solution within which to run my SciPipe workflows.
