I don't have knowledge of the "Common Workflow Language" but I am not sure Go's concurrency is really that much of a selling point for composition. Particularly for composition of evaluation since I would imagine how Go does concurrency would bleed through into your implementation code (probably not what you want.. or maybe it is... or maybe it doesn't for you library?). That is I'm not sure Go is really good at composition compared to functional programming languages.
For example one could use a monadic style data structure such .NET's reactive Observable (which has an analog in many different languages) and allow you to compose a stream like definition independent of how it runs. You can then feed this definition (Observable) to an evaluator which could run it on Go using channels or it could run it on Go using a cluster of machines with ZeroMQ.
I think the selling point to your library is that it compiles to native code but my question is with out changing code can I switch to running it on a cluster instead of local (something I could do with a language with an abstract form of concurrency using monads or Observables or even just using the Actor model)? It looks like I can right?
The idea for cluster support has so far been to implement connectors for resource managers like SLURM, and basically keeping scipipe as an orchestrator.
That is, for multi-node jobs. As long as you can stay within the 16-32 or so cores on a typical HPC node (in our cluster at least), scipipe should be great for that. I think some means of simple resource management (to not overstress these 16-32 cores) is needed, but that can be done in a simple way by e.g. using a central go-routine that lends out "the right to use a cpu core" on demand.
Thanks for interesting feedback. I will think about this!
For example one could use a monadic style data structure such .NET's reactive Observable (which has an analog in many different languages) and allow you to compose a stream like definition independent of how it runs. You can then feed this definition (Observable) to an evaluator which could run it on Go using channels or it could run it on Go using a cluster of machines with ZeroMQ.
I think the selling point to your library is that it compiles to native code but my question is with out changing code can I switch to running it on a cluster instead of local (something I could do with a language with an abstract form of concurrency using monads or Observables or even just using the Actor model)? It looks like I can right?