Documentation

Fractalide documentation

Introduction

This manual tells you how to write subnets, components and contracts for the Fractalide component / contract collection.

What is this?

Fractalide is a free and open source service programming platform using dataflow graphs. Graph nodes represent computations, while graph edges represent typed data (may also describe tensors) communicated between them. This flexible architecture can be applied to many different computation problems, initially the focus will be Microservices to be expanded out into the Internet of Things.

Fractalide is in the same vein as the NSA’s Niagrafiles (now known as Apache-NiFi) or Google’s TensorFlow but stripped of all Java, Python and GUI bloat. Fractalide faces big corporate players like Ab Initio, a company that charges a lot of money for dataflow solutions.

Truly reusable and reproducible efficient nodes is what differentiates Fractalide from the others. It’s this feature that allows open communities to mix and match nodes quickly and easily.

Features

Fractalide stands on the shoulders of giants by combining the strengths of each language into one programming model.

Op Technology Safe Zero-cost Abstractions Reuse Reproducible Distributed Type System Concurrent Service Config Man.

NixOS

+

Nix Expr

+

Rust

+

Flow-based Programming

+

Cap’n Proto

=

Fractalide Model

What’s new from different perspectives

Nix Programmers

Fractalide brings safe, fast, reusable, black-box dataflow functions and a means to compose them.

Tagline: "Nixpkgs is not enough! Here have 'Nixfuncs' too."

Rust Programmers

Fractalide brings reproducible, reusable, black-box dataflow functions, a means to compose them and a congruent model of configuration management.

Tagline: Safety extended beyond the application boundary into infrastructure.

Flow-based Programmers

Fractalide brings safe fast reproducible classical Flow-based programming components, and a congruent model of configuration management.

Tagline: Reproducible components!

Programmers

Fractalide brings safe, fast, reusable, reproducible, black-box dataflow functions, a means to compose them and a congruent model of configuration management.

Tagline: Here, have a beer!

Solved problems

Modules-code coupling

Language level modules become tightly coupled with the rest of the code, moving around these modules also poses a problem.

Solution

An unanticipated outcome occurred when combining FBP and Nix. It’s become our peanut butter and jam combination, so to say, but requires a bit of explaining, so hang tight.

Reproducibility

Nix is a content addressable store, so is git, so is docker, except that docker’s SHA resolution is at container level and git’s SHA resolution is at changeset level. Nix on the other hand has a SHA resolution at package level, and it’s known as a derivation. If you’re trying to create reproducible systems this is the correct resolution. Too big and you’re copying around large container sized images with multiple versions occupying gigabytes of space, too small and you run into problems of git not being able to scale to support thousands of binaries that build an operating system. Therefore Nix subsumes Docker.

Indeed it’s these simple derivations that allow python 2.7 and 3.0 to exist side-by-side without conflicts. It’s what allows the Nix community to compose an entire operating system, NixOS. These derivations are what makes NixOS a congruent configuration management system, and congruent systems are reproducible systems. They have to be.

Reusability

Flow-based programming in our books has delivered on its promise. In our system FBP components are known as a nodes and they are reusable, clean and composable. It’s a very nice way to program computers. Though, we’ve found, the larger the network of nodes, the more overhead required to build, manage, version, package, connect, test and distribute all these moving pieces. This really doesn’t weigh well against FBP’s advantages. Still, there is this beautiful reusable side that is highly advantageous! If only we could take the good parts?

Reproducibility + Reusability

When nix is assigned the responsibility of declaratively building fbp nodes, a magic thing happens. All that manual overhead of having to build, manage and package etc gets done once and only once by the node author, and completely disappears for everyone thereafter. We’re left with the reusable good parts that FBP has to offer. Indeed the greatest overhead a node user has, is typing the node's name. We’ve gone further and distilled the overhead to a few lines, no more intimidating than a typical config file such as Cargo.toml:

{ agent, edges, mods, pkgs }:

agent {
  src = ./.;
  edges = with edges; [ PrimText FsPath ];
  mods = with mods.rs; [ rustfbp rusqlite ];
  osdeps = with pkgs; [ sqlite pkgconfig ];
}

Now just to be absolutely clear of the implications; it’s possible to call an extremely complex community developed hierarchy of potentially 1000+ nodes, where each node might have different https://crates.io dependencies, they might have OS level dependencies such as openssl etc and nix will ensure the entire hierarchy is correctly built and made available. All this is done by just typing the node name and issuing a build command.

It’s this feature that sets us apart from Google TensorFlow and Apache-NiFi. It contains the DNA to build a massive sprawling community of open source programmers, this and the C4, that is. It’s our hope anyway!

Complex configuration management model

The vast majority of system configuration management solutions use either the divergent or convergent model.

We’re going to quote Steve Traugott’s excellent work vebatim.

Divergent
divergent

"One quick way to tell if a shop is divergent is to ask how changes are made on production hosts, how those same changes are incorporated into the baseline build for new or replacement hosts, and how they are made on hosts that were down at the time the change was first deployed. If you get different answers, then the shop is likely divergent.

The symptoms of divergence include unpredictable host behavior, unscheduled downtime, unexpected package and patch installation failure, unclosed security vulnerabilities, significant time spent "firefighting", and high troubleshooting and maintenance costs."

— Steve Traugott
Convergent
convergent

"The baseline description in a converging infrastructure is characteristically an incomplete description of machine state. You can quickly detect convergence in a shop by asking how many files are currently under management control. If an approximate answer is readily available and is on the order of a few hundred files or less, then the shop is likely converging legacy machines on a file-by-file basis.

A convergence tool is an excellent means of bringing some semblance of order to a chaotic infrastructure. Convergent tools typically work by sampling a small subset of the disk - via a checksum of one or more files, for example - and taking some action in response to what they find. The samples and actions are often defined in a declarative or descriptive language that is optimized for this use. This emulates and preempts the firefighting behavior of a reactive human systems administrator - "see a problem, fix it." Automating this process provides great economies of scale and speed over doing the same thing manually.

Because convergence typically includes an intentional process of managing a specific subset of files, there will always be unmanaged files on each host. Whether current differences between unmanaged files will have an impact on future changes is undecidable, because at any point in time we do not know the entire set of future changes, or what files they will depend on.

It appears that a central problem with convergent administration of an initially divergent infrastructure is that there is no documentation or knowledge as to when convergence is complete. One must treat the whole infrastructure as if the convergence is incomplete, whether it is or not. So without more information, an attempt to converge formerly divergent hosts to an ideal configuration is a never-ending process. By contrast, an infrastructure based upon first loading a known baseline configuration on all hosts, and limited to purely orthogonal and non-interacting sets of changes, implements congruence. Unfortunately, this is not the way most shops use convergent tools…​"

— Steve Traugott
Solution
Congruent
congruent

"By definition, divergence from baseline disk state in a congruent environment is symptomatic of a failure of code, administrative procedures, or security. In any of these three cases, we may not be able to assume that we know exactly which disk content was damaged. It is usually safe to handle all three cases as a security breach: correct the root cause, then rebuild.

You can detect congruence in a shop by asking how the oldest, most complex machine in the infrastructure would be rebuilt if destroyed. If years of sysadmin work can be replayed in an hour, unattended, without resorting to backups, and only user data need be restored from tape, then host management is likely congruent.

Rebuilds in a congruent infrastructure are completely unattended and generally faster than in any other; anywhere from ten minutes for a simple workstation to two hours for a node in a complex high-availability server cluster (most of that two hours is spent in blocking sleeps while meeting barrier conditions with other nodes).

Symptoms of a congruent infrastructure include rapid, predictable, "fire-and-forget" deployments and changes. Disaster recovery and production sites can be easily maintained or rebuilt on demand in a bit-for-bit identical state. Changes are not tested for the first time in production, and there are no unforeseen differences between hosts. Unscheduled production downtime is reduced to that caused by hardware and application problems; firefighting activities drop considerably. Old and new hosts are equally predictable and maintainable, and there are fewer host classes to maintain. There are no ad-hoc or manual changes. We have found that congruence makes cost of ownership much lower, and reliability much higher, than any other method."

— Steve Traugott

Fractalide does not violate the congruent model of Nix, and it’s why NixOS is a dependency. Appreciation for safety has extended beyond the application boundary into infrastructure as a whole.

Language choice

A language needed to be chosen to implement Fractalide. Now as Fractalide is primarily a Flow-based programming environment, it would be beneficial to choose a language that at least gets concurrency right.

Solution

Rust was a perfect fit. The concept of ownership is critical in Flow-based Programming. The Flow-based scheduler is typically responsible for tracking every Information Packet (IP) as it flows through the system. Fortunately Rust excels at getting the concept of ownership right. To the point of leveraging this concept that a garbage collector is not needed. Indeed, different forms of concurrency can be layered on Rust’s ownership concept. One very neat advantage Rust gives us is that we can very elegantly implement Flow-based Programming’s idea of concurrency. This makes our scheduler extremely lightweight as it doesn’t need to track IPs at all. Once an IP isn’t owned by any component, Rust makes it wink out of existance, no harm to anyone.

API contracts

It’s easy to disrespect API contracts in a distributed services setup.

Solution

We wanted to ensure there was no ambiguity about the shape of the data a node receives. Also if the shape of data changes, the error must be caught at compile time. Cap’n Proto schema fits these requirements, and fits them perfectly when nix builds the nodes calling the Cap’n Proto schema. Because, if a schema changes, nix will register the change and will rebuild everything (nodes and subgraphs) that depends on that schema, thus catching the error. We’ve also made it such, during graph load time agents cannot connect their ports unless they use the same Cap’n Proto schema. This is a very nice safety property.

The mandatory Hello-like World example.

From a fresh install of NixOS (using the nixos-unstable channel) we’ll build the fractalide virtual machine (fvm) and execute the humble NAND logic gate on it.

$ git clone https://github.com/fractalide/fractalide.git
$ cd fractalide
$ nix-build --argstr node test_nand
...
$ ./result
boolean : false

1. Quick Start to Building an NOT logic gate

The objective of this Quick Start is to demonstrate how to create a Capnproto Schema called an edge, Rust agent and a subgraph hierarchy in Fractalide. Each Fractalide feature can be easily demonstrated by building a NAND logic gate then composing that into a contrived NOT logic gate. We shan’t go into details about each edge agent and subgraph instead the emphasis is how these pieces interact with each other. Please reference the specialized documentation in each of the relevant directories for more information.

Note the NOT and NAND logic gates are purely for example purposes and would never be used in a production system.

  • git clone the Fractalide source code:

$ git clone git://github.com/fractalide/fractalide.git
$ cd fractalide
  • Find a good place to in the Fractalide edges directory to add your capnproto schema. For instance a simple boolean schema for a Nand logic gate might go into edges/prim/bool. Where prim is short for primative.

The directory will have one file:

edges/prim/bool/default.nix
{ edge, edges }:

edge {
  src = ./.;
  edges =  with edges; [];
  schema = with edges; ''
    struct PrimBool {
            bool @0 :Bool;
    }
  '';
}
  • Now we need to make your new edge seen by the system. Insert your newly created edge into edges/default.nix.

edges/default.nix
  { pkgs, support, ... }:
let
callPackage = pkgs.lib.callPackageWith (pkgs // support);
in
# insert in alphabetical order to reduce conflicts
rec {
  # raw
  ...
  PrimText = callPackage ./generic/text {};
  PrimBool = callPackage ./edges/prim/bool {};
  ...
}
  • Do a test compilation of your edge with this command:

$ nix-build -A edges.PrimBool

If you see something like the below, then it successfully compiled the edge and it’s ready to be used by agents.

/nix/store/jy9yjnnmlpc7bzaq5ihjqwiywrx59fw4-PrimBool

The edges/default.nix file contains all the edges which abstract out capnproto schema for this Fractal: edges/default.nix

  • Ensure your soon to be created NAND agent will have the right crate dependencies by navigating to the modules/rs/crates/Cargo.toml file and adding the relevant crates as needed:

modules/rs/crates/Cargo.toml
[lib]

[package]
name = "all_crates"
version = "0.0.0"

[dependencies]
rustfbp = { path = "../rustfbp" }
capnp = "*"
capnpc = "*"
nom = "*"
...

You will only need rustfbp and capnp for this NAND example. Those dependencies are already in the file, but we’ll pretend they aren’t. The [lib] and all_crates in the [package] section are just placeholders and is only there to appease the cargo generate-lockfile command. The all_crates [package] should never be used.

Next you run ./update.sh. You should see similar output as the below:

[stewart@rivergod:~/dev/fractalide/fractalide/modules/rs/crates]$ ./update.sh
Compiling cargo2nix
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
/home/stewart/dev/fractalide/fractalide/modules/rs/crates
Generating lockfile
    Updating registry `https://github.com/rust-lang/crates.io-index`
Running Cargo2nix
Prefetching byteorder-1.1.0
Prefetching capnp-0.8.11
Prefetching capnpc-0.8.7
Prefetching kernel32-sys-0.2.2
Prefetching lazy_static-0.2.8
Prefetching libc-0.2.30
Prefetching libloading-0.4.1
Prefetching memchr-1.0.1
Prefetching nom-3.2.0
Prefetching num_cpus-1.6.2
Prefetching threadpool-1.7.0
Prefetching winapi-0.2.8
Prefetching winapi-build-0.1.1
Done
There's a bug in cargo2nix please manually check that all build_dependencies don't resolve to an undefined nix closure.
For example if you search for winapi_build_0_0_0, this should be changed to winapi_build_0_1_1_
Please make a pull request to resolve this issue in cargo2nix.

As noted in the output there is a minor bug with cargo2nix. Please manually ensure build_dependencies don’t resolve to incorrect nix expressions in the generated modules/rs/cates/default.nix file. The typical case is winapi_0_0_0 should be winapi_0_1_1_ (or whatever the latest winapi version is). A safe way is to search for 0_0_0 and correct these instances as needed.

  • The next step is to build our Rust NAND gate agent.

Find a good place to create our NAND gate is nodes/rs/maths/boolean/nand/lib.rs:

$ mkdir -p nodes/rs/maths/boolean/nand
$ touch nodes/rs/maths/boolean/nand/lib.rs

The contents of the lib.rs should be this:

nodes/rs/maths/boolean/nand/lib.rs
#[macro_use]
extern crate rustfbp;


agent! {
  input(a: prim_bool, b: prim_bool),
  output(output: prim_bool),
  fn run(&mut self) -> Result<Signal> {
    let a = {
        let mut msg_a = try!(self.input.a.recv());
        let boolean: prim_bool::Reader = msg_a.read_schema()?;
        boolean.get_bool()
    };
    let b = {
        let mut msg_b = try!(self.input.b.recv());
        let boolean: prim_bool::Reader = msg_b.read_schema()?;
        boolean.get_bool()
    };

    let mut out_msg = Msg::new();
    {
      let mut boolean = out_msg.build_schema::<prim_bool::Builder>();
      boolean.set_bool(if a == true && b == true {false} else {true});
    }
    try!(self.output.output.send(out_msg));
    Ok(End)
  }
}

Notice the prim_bool code, these are referencing the prim/bool/default.nix edge we created earlier.

Notice the lines below:

extern crate rustfbp;

This code includes our rustfbp and capnp crates into the Rust agent code.

We’ve still not tied the edges nor crates dependencies into the NAND implemenation yet. This is done next.

  • You will need to add a default.nix to your new NAND component.

$ touch nodes/rs/maths/boolean/nand/default.nix

Then insert the below into the default.nix

nodes/rs/maths/boolean/nand/default.nix
{ agent, edges, mods, pkgs }:

agent {
  src = ./.;
  edges = with edges; [ PrimBool ];
  mods = with mods.rs; [ rustfbp capnp ];
  osdeps = with pkgs; [];
}

Notice edges = with edges; [ PrimBool ]; is where we will compile the Capnproto schema which gets copied it into the /tmp/nix-build-prim_bool-*-drv/ directory at build time (all automated by nix, don’t worry about it). This is how your Rust compilation will see the compiled capnproto schema.

Also mods = with mods.rs; [ rustfbp capnp ]; is where we included our crate dependencies as specified in the modules/rs/crates/Cargo.toml file.

  • We need to make our NAND seen by the system by adding it to nodes/rs/default.nix

nodes/rs/default.nix
{ pkgs, support, ... }:
let
  callPackage = pkgs.lib.callPackageWith (pkgs // support // self);
  # insert in alphabetical order to reduce conflicts
  self = rec {
    ...
    maths_boolean_nand = callPackage ./maths/boolean/nand {};
    ...
    };
in
  self
  • Now that the NAND logic gate is tied into Fractalide we can compile it:

$ cd path/to/fractalide
$ nix-build -A components.rs.maths_boolean_nand

Congratulations, you’ve created and compiled your first edge and Rust agent. Now we will move on to creating a subgraph and our final step, the NOT gate.

  • Create the NOT subgraph:

mkdir -p nodes/rs/maths/boolean/not
touch nodes/rs/maths/boolean/not/default.nix

Then insert the below into default.nix:

nodes/rs/maths/boolean/not/default.nix
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    input => input clone(${msg_clone})
    clone() clone[1] -> a nand(${maths_boolean_nand}) output => output
    clone() clone[2] -> b nand()
  '';
}

Notice the ${maths_boolean_nand} and ${msg_clone}. Nix will replace these with fully qualified paths to the compiled agents at compile time. msg_clone is a different agent, you may reference the source code at nodes/rs/msg/clone.

  • Add your new NOT subgraph to the nodes/rs/default.nix

nodes/rs/default.nix
{ pkgs, support, ... }:
let
  callPackage = pkgs.lib.callPackageWith (pkgs // support // self);
  # insert in alphabetical order to reduce conflicts
  self = rec {
  ...
    maths_boolean_nand = callPackage ./maths/boolean/nand {};
    maths_boolean_not = callPackage ./maths/boolean/not {};
  ...
  };
in
  self
  • Let’s compile our newly created NOT subgraph:

$ nix-build -A nodes.rs.maths_boolean_not
/nix/store/xdp2l67gdmxi7fagxnbanavcxd93mlr0-maths_boolean_not

The subgraph will compile to :

/nix/store/xdp2l67gdmxi7fagxnbanavcxd93mlr0-maths_boolean_not/lib/lib.subgraph
input => input clone(/nix/store/wb6fgpz9hk7fg1f6p9if81s1xhflhy2x-msg_clone)
clone() clone[1] -> a nand(/nix/store/bi0jacqxz1az1bbrc8470jbl7z3cmwdn-maths_boolean_nand) output => output
clone() clone[2] -> b nand()

Notice the ${maths_boolean_nand} and ${msg_clone} were replaced with fully qualified paths. This output is meant for the fvm (Fractalide Virtual Machine) to parse and isn’t meant to be edited by humans.

  • Let us prepare to run our new NOT component. This is where we create imsgs which contain the actual values to be passed into the NOT gate.

First, edit nodes/rs/test/not/default.nix so that it looks like this:

nodes/rs/test/not/default.nix
{ subgraph, imsg, nodes, edges }:

let
  PrimBool = imsg {
    class = edges.PrimBool;
    text = "(bool=true)";
    option = "create";
  };
in
subgraph {
 src = ./.;
 flowscript = with nodes.rs; ''
  '${PrimBool}' -> input not(${maths_boolean_not}) output -> input io_print(${maths_boolean_print})
 '';
}

Notice the section of code:

PrimBool = imsg {
  class = edges.PrimBool;
  text = "(bool=true)";
  option = "create";
};

This declares an imsg, it defines the values to initialize your edges/prim/bool edge.

  • Next, you’ll need to compile test_not:

$ nix-build --argstr node test_not
...
/nix/store/a4lb3b9jjylvrl77kv0wb8m5v137f6j1-test_not
  • Then run it:

$ ./result
boolean : false
  • Conclusion

This concludes the Quick Start, demonstrating the building of a Capnproto schema which composes into an edge, a Rust agent and a Flowscript subgraph. It also demonstrates how to add crates.io crate dependencies and how to run the top level not subgraph.

2. Nodes collection

The Nodes collection consists of Subgraphs and Agents. A Subgraph or an Agent may be referred to as a Node.

2.1. Subgraphs

2.1.1. What?

A Subgraph consists of an implementation and an interface. The interface is implemented using a simple interface description language called Flowscript which describes how data flows through Agents and other Subgraphs. The result is an interface that consists of a minimal set of well named ports, thus hiding complexity.

A simple analogy would be this gentleman’s pocket watch.

watchcalls 35

2.1.2. Why?

Composition is an important part of programming, allowing one to hide implementation detail.

2.1.3. Who?

People who want to focus on the Science tend to work at these higher abstractions, they’d prefer not getting caught up in the details of programming low level nodes and hand specifications to programmers who’ll make efficient, reusable and safe Agents. Though programmers will find Subgraphs indispensable as they allow for powerful abstractions.

2.1.4. Where?

The Nodes directory is where all Agents and Subgraphs go. Typically one might structure a hierarchy like such:

── wrangle
   ├── default.nix <------
   ├── aggregate
   ├── anonymize
   ├── print
   ├── processchunk
   │   ├── default.nix <------
   │   ├── agg_chunk_triples
   │   ├── convert_json_vector
   │   ├── extract_keyvalue
   │   ├── file_open
   │   └── iterate_paths
   └── stats

See the above default.nix files? Those are Subgraphs and they hide the entire directory level they reside in from higher levels in the hierarchy. Thus processchunk (a Subgraph) looks like yet another Node to wrangle (another Subgraph). Indeed wrangle is completely unable to distinguish between an Agent and a Subgraph.

2.1.5. How?

The Subgraph default.nix requires you make decisions about two types of dependencies.

  • What Nodes are needed?

  • What Edges are needed?

default.nix
{ subgraph, imsg, nodes, edges }:
let
  imsgTrue = imsg {
    class = edges.PrimBool;
    text = ''(boolean=true)'';
  };
in
subgraph {
  src = ./.;
  flowscript = with nodes.rs;''
    nand(${maths_boolean_nand})
    '${imsgTrue}' -> a nand()
    '${imsgTrue}' -> b nand()
    nand() output -> input io_print(${maths_boolean_print})
  '';
}
subnet ex10
  • The { subgraph, nodes, edges }: lambda passes in three arguments, the subgraph builder, edges which consists of every Edge or Edge Namespace, and the nodes argument which consists of every Node and Node Namespace in the system.

  • The subgraph building function accepts these arguments:

    • The src attribute is used to derive a Subgraph name based on location in the directory hierarchy.

    • The flowscript attribute defines the business logic. Here data flowing through a system becomes a first class citizen that can be manipulated. Nodes and Edges are brought into scope between the opening '' and closing '' double single quotes by using the with nodes; with edges; syntax.

  • Nix assists us greatly, in that each node name (the stuff between the curly quotes ${…​}) undergoes a compilation step resolving every name into an absolute /path/to/compiled/lib.subgraph text file and /path/to/compiled/libagent.so shared object.

  • This compilation is lazy and only referenced names will be compiled. In other words Subgraph could be a top level Subgraph of a many layer deep hierarchy and only referenced Nodes will be compiled in a lazy fashion, not the entire fractalide/nodes folder.

This is the output of the above Subgraph's compilation:

$ cat /nix/store/1syrjhi6jvbvs5rvzcjn4z3qkabwss7m-test_sjm/lib/lib.subgraph
nand(/nix/store/7yzx8fp81fl6ncawk2ag2nvfc5l950xb-maths_boolean_nand)
'/nix/store/fx46blm272yca7n3gdynwxgyqgw90pr5-prim_bool:(boolean=true)' -> a nand()
'/nix/store/fx46blm272yca7n3gdynwxgyqgw90pr5-prim_bool:(boolean=true)' -> b nand()
nand() output -> input io_print(/nix/store/k67wiy6z4f1vnv35vdyzcqpwvp51j922-maths_boolean_print)

Mother of the Flying Spaghetti Monster, what is that? One really doesn’t need to be concerned about this target, as it’s meant to be processed by the Fractalide Virtual Machine. It’s worth noting that those hashes hint at something powerful. Projects like docker and git implement this type of content addressable store. Except docker's granularity is at container level, and git's granularity is at revision level. Our granularity is at package or library level. It allows for reproducible, deterministic systems, instead of copying around "zipped" archives, that quickly max out your hard drive.

2.1.6. Flowscript syntax is easy

Everything between the opening '' and closing '' is flowscript, i.e:

{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
                       <---- here
  '';
}
subnet ex0
Agent initialization:
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    agent_name(${name_of_agent})
  '';
}
subnet ex1
Referencing a previously initialized agent (with a comment):
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    agent_name(${name_of_agent}) // <──┐
    agent_name()                 // <──┴─ same instance
  '';
}
subnet ex1
Connecting and initializing two agents:
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    agent1(${name_of_agent1}) output_port -> input_port agent2(${name_of_agent2})
  '';
}
subnet ex3

If the connection between output_port and input_port have the same schema, then the connection is upgraded to a new name called an edge.

Creating an imsg or an exposed edge
{ subgraph, imsg, nodes, edges }:
let
  imsgTrue = imsg {
    class = edges.PrimBool;
    text = ''(boolean=true)'';
  };
in
subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    '${imsgTrue}' -> a agent(${name_of_agent})
  '';
}
subnet ex4
More complex iMsg or Exposed Edge
{ subgraph, imsg, edges, nodes }:
let
  UiJsCreate = imsg {
    class = edges.UiJsCreate;
    text = ''(type="div", style=(list=[(key=(text="display"), val=(text="flex")), (key=(text="flex-direction"), val=(text="column"))]))'';
    option = "create";
  };
in
subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    td(${ui_js_nodes.flex})
    '${UiJsCreate}' -> input td()
  '';
}
subnet ex5

Learn more about Edges.

Creating an subgraph input port
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    subgraph_input => input agent(${name_of_agent})
  '';
}
subnet ex6
Creating an subgraph output port
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    agent(${name_of_agent}) output => subgraph_output
  '';
}
subnet ex7
Subgraph initialization:
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    subgraph(${name_of_subgraph})
  '';
}
subnet ex8
Initializing a subgraph and agent then connecting them:
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    subgraph(${name_of_subgraph})
    agent(${name_of_agent})
    subgraph() output -> input agent()
  '';
}
subnet ex9
Output array port:
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    db_path => input clone(${msg_clone})
    clone() clone[0] => db_path0
    clone() clone[1] => db_path1
    clone() clone[2] => db_path2
    clone() clone[3] => db_path3
  '';
}
subnet ex11
clone[1] is an array output port and in this particular Subgraph Messages are being replicated, a copy for each port element. The content between the [ and ] is a string, so don’t be misled by the integers. There are two types of node ports, a simple port (which doesn’t have array elements) and an array port (with array elements).
Input array port:
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    add0 => add[0] adder(${path_to_adder})
    add1 => add[1] adder()
    add2 => add[2] adder()
    add3 => add[3] adder() output -> output
  '';
}
subnet ex15

Array ports are used when the number of ports are unknown at Agent development time, but known when the implemented Agent is used in a Subgraph. The adder Agent demonstrates this well, it has an array input port which allows Subgraph developers to choose how many integers they want to add together. It really doesn’t make sense to implement an adder with two fixed simple input ports then be constrained when you need to add a third number.

Hierarchical naming:
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    input => input clone(${msg_clone})
    clone() clone[0] -> a nand(${maths_boolean_nand})
    clone() clone[1] -> b nand() output => output
  '';
}
subnet ex13

The Node and Edge names, i.e.: ${maths_boolean_nand} seem quite long. Fractalide uses a hierarchical naming scheme. So you can find the maths_boolean_not node by opening to the nodes/rs/maths/boolean/not/default.nix file. The whole goal of this is to avoid name shadowing among potentially hundreds to thousands of nodes.

Explanation of the Subgraph:

This Subgraph takes an input of a Hidden Edge type prim_bool over the input port. A Msg is cloned by the clone node and the result is pushed out on the array output port clone using elements [0] and [1]. The nand() node then performs a NAND boolean logic operation and outputs a prim_bool data type, which is then sent over the Subgraph output port output.

The above implements the not boolean logic operation.

Abstraction powers:
{ subgraph, nodes, edges }:
let
  imsgTrue = imsg {
    class = edges.PrimBool;
    text = ''(boolean=true)'';
  };
in
subgraph {
  src = ./.;
  flowscript = with nodes.rs; ''
    '${imsgTrue}' -> a nand(${maths_boolean_nand})
    '${imsgTrue}' -> b nand()
    nand() output -> input not(${maths_boolean_not})
    not() output -> input print(${maths_boolean_print})
  '';
}
subnet ex14

Notice we’re using the not node implemented earlier. One can build hierarchies many layers deep without suffering a run-time performance penalty. Once the graph is loaded into memory, all Subgraphs fall away, like water, after an artificial gravity generator engages, leaving only Agents connected to Agents.

Namespaces
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes; with edges; ''
    listen => listen http(${net_http_nodes.http})
    db_path => input clone(${msg_clone})
    clone() clone[1] -> db_path get(${app_todo_nodes.todo_get})
    clone() clone[2] -> db_path post(${app_todo_nodes.todo_post})
    clone() clone[3] -> db_path del(${app_todo_nodes.todo_delete})
    clone() clone[4] -> db_path patch(${app_todo_nodes.todo_patch})

    http() GET[/todos/.+] -> input get() response -> response http()
    http() POST[/todos/?] -> input post() response -> response http()
    http() DELETE[/todos/.+] -> input del() response -> response http()
    http() PATCH[/todos/.+] -> input patch()
    http() PUT[/todos/.+] -> input patch() response -> response http()
  '';
}
subnet ex12

Notice the net_http_nodes and app_todo_nodes namespaces. Some fractals deliberately export a collection of Nodes. As is the case with the net_http_nodes.http node. When you see a fullstop ., i.e. xxx_nodes.yyy you immediately know this is a namespace. It’s also a programming convention to use the _nodes suffix to indicate a namespace. Lastly, notice the advanced usage of array ports with this example: GET[/todos/.+], the element label is actually a regular expression and the implementation of that node is slightly more advanced You can read more about this in the HOWTO.

2.2. Agents

2.2.1. Rust

What?

Executable Subgraphs are defined as a network of Agents, which exchange typed data across predefined connections by message passing, where the connections are specified externally to the processes. These Agents can be reconnected endlessly to form different executable Subgraphs without having to be changed internally.

Why?

Functions in a programming language should be placed in a content addressable store, this is the horizontal plane. The vertical plane should be constructed using unique addresses into this content addressable store, critically each address should solve a single problem, and may do so by referencing multiple other unique addresses in the content addressable store. Users must not have knowledge of these unique addresses but a translation process should occur from a human readable name to a universally unique address. Read more about the problem.

Nix gives us the content addressable store which allows for reproducibility, and these agents give us reusablility. The combination is particularly potent form of programming.

Once you have the above, you have truly reusable and reproducible functions. Fractalide nodes are just this, and it makes the below so much easier to achieve:

* Open source collaboration
* Open peer review of nodes
* Nice clean reusable nodes
* Reproducible applications
Who?

Typically programmers will develop Agents. They specialize in making Agents as efficient and reusable as possible, while people who focus on the Science give the requirements and use the Subgraphs. Just as a hammer is designed to be reused, so Subgraphs and Agents should be designed for reuse.

Where?

The Agents are found in this nodes directory, or the nodes directory of a fractal.

processchunk
├── default.nix
├── agg_chunk_triples
│   ├── default.nix <---
│   └── lib.rs
├── convert_json_vector
│   ├── default.nix <---
│   └── lib.rs
├── extract_keyvalue
│   ├── default.nix <---
│   └── lib.rs
├── file_open
│   ├── default.nix <---
│   └── lib.rs
└── iterate_paths
    ├── default.nix <---
    └── lib.rs

Typically when you see a lib.rs in the same directory as a default.nix you know it’s an Agent.

How?

An Agent consists of two parts:

  • a nix default.nix file that sets up an environment to satisfy rustc.

  • a rust lib.rs file implements your agent.

The agent Nix function.

The agent function in the default.nix requires you make decisions about three types of dependencies.

  • What edges are needed?

  • What mods from crates.io are needed?

  • What osdeps or operating system level dependencies are needed?

{ agent, edges, mods, pkgs }:

agent {
  src = ./.;
  edges = with edges.rs; [ ];
  mods = with mods.rs; [ rustfbp ];
  osdeps = with pkgs; [];
}
  • The { agent, edges, mods, pkgs }: lambda imports: The edges attribute which consists of every edge available on the system. The mods attribute set consists of every crate on https://crates.io. Lastly the pkgs pulls in every third party package available on NixOS, here’s the whole list.

  • The agent function builds the rust lib.rs source code, and accepts these arguments:

    • The src attribute is used to derive an Agent name based on location in the directory hierarchy.

    • The edges lazily compiles schema and composite schema ensuring their availability. Sometimes when the types are rust primatives there doesn’t need to be a type in the square brackets. Otherwise the type is derived from the edges directory tree.

    • The mods specifies exactly which mods are needed in scope.

    • The osdeps specifies exactly which pkgs, or third party operating system level libraries such as openssl needed in scope.

Only specified dependencies and their transitive dependencies will be pulled into scope once the agent compilation starts.

This is the output of the above agent's compilation:

/nix/store/dp8s7d3p80q18a3pf2b4dk0bi4f856f8-maths_boolean_nand
└── lib
    └── libagent.so
The agent! Rust macro

This is the heart of Fractalide. Everything revolves around this API. The below is an implementation of the ${maths_boolean_nand} agent seen earlier.

#[macro_use]
extern crate rustfbp;

agent! {
  input(a: bool, b: bool),
  output(output: bool),
  fn run(&mut self) -> Result<Signal> {
    let a = self.input.a.recv()?;
    let b = self.input.b.recv()?;
    let res = ! (a && b);
    self.output.output.send(res)?;
    Ok(End)
  }
}

An explanation of each of the items should be given. All expresions are optional except for the run function.

input:
#[macro_use]
extern crate rustfbp;

agent! {
  input(input_name: bool),
  fn run(&mut self) -> Result<Signal> {
    let msg = self.input.input_name.recv()?;
    Ok(End)
  }
}

The input port, is a bounded buffer simple input channel rust typed data as messages.

inarr:
#[macro_use]
extern crate rustfbp;

agent! {
  inarr(input_array_name: i32),
  fn run(&mut self) -> Result<Signal> {
    let mut sum = 0;
    for (_id, elem) in &self.inarr.input_array_name {
      sum += elem.recv()?;
    }
    Ok(End)
  }
}

The inarr is an input array port, which consists of multiple elements of a port. They are used when the Subgraph developer needs multiple elements of a port, for example an adder has multiple input elements. This adder agent may be used in many scenarios where the amount of inputs are unknown at agent development time.

output:
#[macro_use]
extern crate rustfbp;

agent! {
  output(output_name: prim_bool),
  fn run(&mut self) -> Result<Signal> {
    self.output.output_name.send(true)?;
    Ok(End)
  }
}

The humble simple output port. It doesn’t have elements and is fixed at subgraph development time.

outarr:
#[macro_use]
extern crate rustfbp;

agent! {
  outarr(out_array_name: bool),
  fn run(&mut self) -> Result<Signal> {
    for p in self.outarr.out_array_name.elements()? {
      self.outarr.out_array_name.send(true)?;
    }
    Ok(End)
  }
}

The outarr port is an output array port. It contains elements which may be expanded at subgraph development time.

option:
agent! {
  option(bool),
  fn run(&mut self) -> Result<Signal> {
    let mut opt = self.option.recv();
    // use opt to configure something
    Ok(End)
  }
}

The option port gives the subgraph developer a way to send in parameters such as a connection string and the message will not be consumed and thrown away, that message may be read on every function run. Whereas other ports will consume and throw away the message.

accumulator:
agent! {
  accumulator(prim_bool),
  fn run(&mut self) -> Result<Signal> {
    let acc = self.ports.accumulator.recv()?;
    // use the accumulator to start accumulating something.
    Ok(End)
  }
}

The accumulator gives the subgraph developer a way to start counting at a certain number. It’s a way of passing in initial state.

run:

This function does the actual processing and is the only mandatory expression of this macro. You’ve seen many examples already.

2.2.2. Idris

What?

Executable Subgraphs are defined as a network of Agents, which exchange typed data across predefined connections by message passing, where the connections are specified externally to the processes. These Agents can be reconnected endlessly to form different executable Subgraphs without having to be changed internally.

Why?

Functions in a programming language should be placed in a content addressable store, this is the horizontal plane. The vertical plane should be constructed using unique addresses into this content addressable store, critically each address should solve a single problem, and may do so by referencing multiple other unique addresses in the content addressable store. Users must not have knowledge of these unique addresses but a translation process should occur from a human readable name to a universally unique address. Read more about the problem.

Nix gives us the content addressable store which allows for reproducibility, and these agents give us reusablility. The combination is particularly potent form of programming.

Once you have the above, you have truly reusable and reproducible functions. Fractalide nodes are just this, and it makes the below so much easier to achieve:

* Open source collaboration
* Open peer review of nodes
* Nice clean reusable nodes
* Reproducible applications
Who?

Typically programmers will develop Agents. They specialize in making Agents as efficient and reusable as possible, while people who focus on the Science give the requirements and use the Subgraphs. Just as a hammer is designed to be reused, so Subgraphs and Agents should be designed for reuse.

Where?

The Agents are found in this nodes directory, or the nodes directory of a fractal.

nodes
└── idr
    └── file
        └── open
            ├── File <--- contains the `file_open` idris agent code
            │   └── *.idr
            ├── Tests <--- contains the `file_open` idris agent tests
            │   └── *.idr
            ├── default.nix <--- configures the environment to compile the agent
            └── agent.ipkg <--- `idris` uses the deps made available by `default.nix`

When you see an agent.ipkg in the same directory as a default.nix you know it’s an Agent.

How?

An Agent consists of three parts:

  • a nix default.nix file that sets up an environment to satisfy the idris repl/compiler.

  • an idris agent.ipkg tells idris how to compile the source code in the directories.

  • the idris *.idr source code within capitalized directories.

The agent Nix function.

The agent function in the default.nix requires you make decisions about three types of dependencies.

  • What edges inter-agent idris types are to be pulled in?

  • What mods which are idris library dependencies available online.

  • What osdeps or operating system level dependencies are needed?

{ agent, edges, mods, pkgs }:

agent {
  src = ./.;
  edges = with edges.idr; [ TestVect ];
  mods = with mods.idr; [ contrib ];
  osdeps = with pkgs; [ openssl ];
}
  • The { agent, edges, mods, pkgs }: lambda imports: The edges attribute which consists of every edge available on the system. The mods attribute set consists of every mod in the modules/idr directory. Lastly the pkgs pulls in every third party package available on NixOS, here’s the whole list.

  • The agent function builds the idris agent.ipkg and associated source code, and accepts these arguments:

    • The src attribute is used to derive the source code and Agent name based on location in the nodes directory hierarchy.

    • The edges makes inter-agent idris types available just before build time.

    • The mods specifies exactly which modules are needed for the agent to build.

    • The osdeps specifies exactly which pkgs, or third party operating system level libraries such as openssl needed in scope.

Only specified dependencies and their transitive dependencies will be pulled into scope once the agent compilation starts, or when you run a development shell.

This is the output of the above agent's compilation:

/nix/store/dp8s7d3p80q18a3pf2b4dk0bi4f856f8-file_open
└── lib
    └── libagent.js
nix-shell

It’s convenient to use a REPL, idris-mode with your editor while developing an agent.

Here are the steps to setup a development environment.

In the root directory of fractalide, issue these commands:

  • $ nix-shell -A mods.idr.idrisfbp ← or whatever the attribute path of the idris agent you want

  • $ cd modules/idr/idrisfbpcd to your chosen agent in the nodes hierachy

  • $ source $setup

  • $ build

  • $ run emacs .

For the run command to work you need to have installed your editor via your system level configuration.nix file, as it references the /run/current-system/sw/bin/* path.

You may also start a REPL and use your integrated idris-modes in your editor. Note all the dependencies have been alias’ed to `idris.

$ type idris
idris is aliased to `idris -i /nix/store/0ijgdwdb9bfwwkgcxac25p2mxl161ljb-base-1.1.0/lib/1.1.0-git:PRE/base -i /nix/store/12cs9pgsdq4rhnzxjdk5hqv5rc9v60pb-prelude-1.1.0/lib/1.1.0-git:PRE/prelude -i /nix/store/506r1qqayw6j2nb4dsfvb716n9x8ndmj-contrib-1.1.0/lib/1.1.0-git:PRE/contrib -i /home/stewart/dev/fractalide/fractalide/modules/idr/idrisfbp/idris_libs'

Each idris command in each agent development environment will contain a different set of associated alias’ed paths. It might be a good idea to create a throw away agent that contains all the useful dependencies, create a development shell using that agent then navigate around the nodes hierarchy without closing your editer each time.

3. Edge Collection

3.1. What?

An Edge is a bounded buffer message passing channel between two ports belonging to their respective agents.

Terms:

  • A contract is defined as: The Cap’n Proto schema on the output port of the upstream agent MUST be the same as the Cap’n Proto schema on the input port of the downstream agent. If the two are the same, the contract is said to be satisfied, otherwise it is unsatisfied.

There are three phases building up to a successful Edge formation:

  • During agent development time an agent's port is assigned a Cap’n Proto schema.

  • During subgraph development time the syntax -> or => is used to instruct an upstream agent's port to connect to a downstream agent's port later at run-time. It represents a connection between two nodes. Though it is not yet an edge because the contract might not be satisfied.

  • Lastly, the graph is successfully loaded into the virtual machine without errors. This act means all agent contracts were satisfied, and all subgraph connections are now classified as edges.

Once an edge is formed, it becomes a bounded buffer message passing channel, which can only contain messages with data in the shape of whatever the Cap’n Proto schema is.

So despite you seeing only Cap’n Proto schema in this directory, the concept of an Edge is more profound. Hence we would prefer naming this concept after it’s grandest manifestation, and in the process, the name encapsulates all of the above information. The name should also tie in with the concept of a node in graph theory, as such we use nodes and edges to construct subgraphs.

3.1.1. Exposed Edge or iMsg

When developing a subgraph there comes a time when the developer wants to inject data into an agent or another subgraph. One needs to use an exposed edge or an imsg (initial message) which has this syntax:

{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes; with edges; ''
    '${prim_bool}:(boolean=true)' -> INPUT_PORT NAME()
  '';
}

Exposed edges or imsgs kick start agents into action, otherwise they won’t start. Due to the dataflow nature of agents they will politely wait for data before they start doing anything. This is the equivalent of passing the path of a data source to some executable on the command line. Without the argument the program would just sit or fail. Another way of looking at it might be a pipeline that has come to the surface to accept some form of input.

We use the name exposed edge to differentiate between a hidden edge, but by far the most common usage is imsg and edge.

3.1.2. Hidden Edge or Edge

Hidden edges are represented with this syntax -> and =>, and are used to control the direction flowing data. Hence the process of programming a subgraph is essentially digging ditches and laying pipelines between buildings.

Examples of hidden edges

  • From one agent to another agent:

{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes; with edges; ''
    agent1() output -> input agent2()
  '';
}
  • Into a subgraph:

{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes; with edges; ''
    output => input subgraph()
  '';
}
  • Out of a subgraph:

{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes; with edges; ''
    agent() output => output
  '';
}

3.2. Why?

  • Contracts between components are critical for creating living systems.

  • Schema ensure we do not need to parse strangely formatted stdin data.

  • A node does one and only one thing, and the schema represents a language the node speaks. If you want a node to do something, you have to speak it’s language.

3.3. Who?

Typically subgraph developers will be interested in hidden and exposed edges.

3.4. Where?

The edges directory is where all the schema go:

├── key
│   └── value
│       └── default.nix
├── list
│   ├── command
│   │   └── default.nix
│   ├── text
│   │   └── default.nix
│   ├── triple
│   │   └── default.nix
│   └── tuple
│       └── default.nix
├── maths
│   ├── boolean
│   │   └── default.nix
│   └── number
│       └── default.nix

3.5. How?

Edges may depend on other edges.

The { edge, edges }: lambda passes in two arguments, the edge builder and edges which consists of every Edge or Edge Namespace in the system. The edge building function accepts these arguments:

  • The src attribute is used to derive an Edge name based on location in the directory hierarchy.

  • The edges attribute resolve transitive dependencies and ensures your agent has all the needed files to type check.

  • a Cap 'n Proto schema. This is the heart of the contract, this is where you may create potentially complex deep hierarchies of structured data. Please read more about the schema language.

{ edge, edges }:

edge {
  src = ./.;
  edges = with edges; [ command ];
  schema = with edges; ''
    @0xf61e7fcd2b18d862;
    using Command = import "${command}/src/edge.capnp";
    struct ListCommand {
        commands @0 :List(Command.Command);
    }
  '';
}

3.5.1. Naming of Cap’n Proto structs and enums

Please use CamelCase for struct and enum names. The naming should reflect this manner:

  • if the schema is in folder /edges/maths/boolean then the struct should have name MathsBoolean.

  • if the schema is in fractal /edges/net/http/response then the struct should have name NetHttpResponse.

The same naming applies for Cap’n Proto enums and interfaces. It’s crucial this naming is adopted.

3.5.2. One struct per Fractalide schema

We prefer composition of schema, and the schema must have fully qualified struct names. Hence, this is example shouldn’t be used:

{ edge, edges }:

edge {
  src = ./.;
  edges = with edges; [ command ];
  schema = with edges; ''
    @0xf61e7fcd2b18d862;
    struct Person {
      name @0 :Text;
      birthdate @3 :Date;

      email @1 :Text;
      phones @2 :List(PhoneNumber);

      struct PhoneNumber {
        number @0 :Text;
        type @1 :Type;

        enum Type {
          mobile @0;
          home @1;
          work @2;
        }
      }
    }

    struct Date {
      year @0 :Int16;
      month @1 :UInt8;
      day @2 :UInt8;
    }
  '';
}

The Date name can collide!

Schema are pulled into agent's scope just before compile time, now we are unable to predict what combinations will happen. So if we have two schema that have struct Date …​ then a name collision will take place. Therefore to avoid this scenario please put struct Date …​ into it’s own schema and import it via this mechanism.

3.5.3. Cap’n Proto import

Fractalide resolves transitive dependencies for you but you have to use this method:

{ edge, edges }:

edge {
  src = ./.;
  edges = with edges; [ command ];
  schema = with edges; ''
    @0xf61e7fcd2b18d862;
    using CommandInstanceName = import "${command}/src/edge.capnp";
    struct ListCommand {
        commands @0 :List(CommandInstanceName.Command);
    }
  '';
}

You must pull explicitly mention the edge you want to import via the ` edges = with edges; [ command ];` Then you must import it via this mechanism: using CommandInstanceName = import "${command}/src/edge.capnp"; and lastly use it …​ commands @0 :List(CommandInstanceName.Command); …​

Out of curiosity what does the output of the above list_command contract function look like?

$ cat /nix/store/3s25icpbf1chayvrxwbyxr9qckn7x669-list_command/src/edge.capnp
@0xf61e7fcd2b18d862;
using CommandInstanceName = import "/nix/store/bgh37035cbr49r7mracmdwwjx9sbf4nr-command/src/edge.capnp";

struct ListCommand {
    commands @0 :List(CommandInstanceName.Command);
}

The generated Rust code consists of the list_command, command and tuple contract concatenated together.

4. Services

4.1. What?

Services are reusable and consist of agents, subgraphs and edges. They may be connected to other services on the same system or on remote machines in a high performance cluster quite easily.

4.2. Why?

Services allow programmers to collaborate and distill common configurable patterns that solve general problems. Essentially a service in this context is no different from a top level subgraph, albeit the interface has been nix’ified. The `nix interface is required because this is where nixos takes over and declaratively handles conguent configuration management. It does things like sets up dependencies such as database and other services that might need to be run for the subgraph to operate properly. This layer of software has been well tested by the nix community and ties in with every other service created by the nixos community.

This abstraction allows you to pull in legacy services, plug a Cap’n Proto functionality and start talking to Fractalide services.

4.3. Who?

Anyone who has completed a high quality, documented hierarchy of components, subnets and contracts, and where the hierarchy makes sense to expose as a service. We use the [C4](../CONTRIBUTING.md) so your services will be merged in quickly.

4.4. Where?

Service implementations by convention exist in a fractal’s root folder. It doesn’t make sense to expose every fractal as a service. Though if the fractal is dependent on other services such as databases, then it might make sense. Nevertheless, the fractals which have a service.nix may have their components and contracts imported into higher level subnets via the Fractals importing mechanism. This higher level subnet may then be exposed as a service.

4.5. How?

  • First create a fractal

  • Update the service.nix file to reflect what options you want exposed on the service interface.

  • Ensure the subnet you want is correctly configured.

  • Add a line the fractalide/services/default.nix where you make your service visible to the rest of the fractalide community.

4.5.1. Two ways to run a service

From your Fractal

This approach is when you have a service you don’t consider generic and is not worth sharing with the community.

in your configuration.nix put these lines:

configuration.nix
{ config, pkgs, ... }:
let
  fractalide = import fetchFromGitHub {
    owner = "fractalide";
    repo = "fractalide";
    rev = "b9590b6{ git revision you want }f130ed79432c722";
    sha256 = "0jz58mmax5{ correct sha256 generated by nix }yycmzr26xwfqa";
  } {};
in
{
  require = fractalide.services ++ [ "/path/to/private/dev/fractals/fractal_your_fractal/service.nix" ];
  services.workbench = {
    enable = true;
    bindAddress = "192.168.0.14";
    port = 8000;
  };
  services.your_service = {
    enable = true;
    bindAddress = "192.168.0.15";
    port = 8001;
  };
  # ... other options you want
}
From Fractalide

in your configuration.nix put these lines:

configuration.nix
let
  fractalide = import fetchFromGitHub {
    owner = "fractalide";
    repo = "fractalide";
    rev = "b9590b6{ git revision you want }f130ed79432c722";
    sha256 = "0jz58mmax5{ correct sha256 generated by nix }yycmzr26xwfqa";
  } {};
in
{
require = fractalide.services;
services.workbench = {
        enable = true;
        bindAddress = "127.0.0.1";
        port = 8000;
};
# ... other options you want
  • At this point, you may either choose to build nixops infrastructure scripts and deploy or simply run $ nixos-rebuild {test, boot , switch} to set up the services on your own system.

  • Do help out building generic services for programmers to speed up their work!

5. Fractals

Fractals are third party Fractalide libraries.

5.1. What?

Fractals are importable 3rd party sets of nodes, subgraphs and edges, this folder hierarchy is the single source of truth regarding where to find each community fractal.

5.2. Why?

Fractals allow the community to develop their own projects in their own git repository, and once ready, may plug into dev/fractalide/fractals folder hierarchy which represents the spine of Fractalide. Once inserted your fractal is available for use by everyone.

5.3. Who?

Anyone making high quality, documented nodes, subgraphs and edges may plug into this folder hierarchy. We use the C4 so your fractals will be merged in quickly.

5.4. Where?

Each fractal needs to have it’s own hierarchical folder. For example, HTTP would be placed in the fractalide/fractals/net/http folder.

5.5. How?

Say you wish to create a http project, these are the exact steps involved:

  • Ensure you use this directory structure convention:

dev
├── fractalide
│   └── fractals
│       └── net
│           └── http
│               └── default.nix
└── fractals
    ├── fractal_net_http
    └── ... more community fractals cloned
$ cd <your/development/directory>
$ git clone \https://gitlab.com/fractalide/fractalide.git
$ NIX_PATH="nixpkgs=https://github.com/NixOS/nixpkgs/archive/125ffff089b6bd360c82cf986d8cc9b17fc2e8ac.tar.gz:fractalide=/path/to/dev/fractalide" && export NIX_PATH` (1)
$ mkdir dev/fractals && cd dev/fractals
$ git clone git://github.com/fractalide/fractal_workbench.git fractal_net_http (2)
$ git remote set-url origin git://new.url.you.control.here
$ mkdir -p dev/fractalide/fractals/net/http/ (3)
1 Take note when setting the NIX_PATH environment variable, it must include the path to your newly cloned fractalide repo i.e.: NIX_PATH=…​:fractalide=/path/to/dev/fractalide.
Should you start a new shell, type <ctrl>-r then type 125ff this will search your command history for the above command, or just persist the command in your ~/.bashrc file.
2 The fractal_workbench repo provides a minimum correct structure for your fractal. Keep the repo naming convention fractal_* for your repo as it’ll be easy for the community to see this is a fractalide related project.
3 Create your needed directory hierarchy dev/fractalide/fractals/net/http/default.nix.

Insert the below code into a file called default.nix which sits in the above folder.

dev/fractalide/fractals/net/http/default.nix
{ pkgs
, support
, edges
, nodes
, fetchFromGitHub
, ...}:
let
  fractal = fetchFromGitHub {
    owner = "fractalide";
    repo = "fractal_net_http";
    rev = "66ad3bf74b04627edc71227b3e5b944561854367";
    sha256 = "1vs1d3d9lbxnyilx8g45pb01z5cl2z3gy4035h24p28p9v94jx1b";
  };
  /*fractal = ../../../../fractals/fractal_net_http;*/
in
  import fractal {inherit pkgs support edges nodes; fractalide = null;}
  • The pkgs, support, nodes, edges and fetchFromGitHub are arguments passed into this closure.

  • The let expression contains a fetchFromGitHub expression describing the location of the fractal.

    • owner of the git repository i.e.: github.com/fractalide

    • repo is the git repository in question i.e.: github.com/fractalide/fractal_net_http

    • rev indicates the git revision you want to import i.e.: https://github.com/fractalide/fractal_net_http/commit/66ad3bf74b04627edc71227b3e5b944561854367

    • sha256 is a neat nix mechanism to assist in deterministic builds. To obtain the correct sha256 try build your project with an incorrect sha256 (change the first alpha-numeric character in 1vs1d3d9lbxnyilx8g45pb01z5cl2z3gy4035h24p28p9v94jx1b to a 2). Nix will download the repository and check that the actual sha256 matches against what you incorrectly inserted. Nix will tell you what the correct sha256 is. Copy it and insert the correct sha256, replacing 2vs1d3d9lbxnyilx8g45pb01z5cl2z3gy4035h24p28p9v94jx1b.

  • Please notice the line: /*fractal = ../../../../fractals/fractal_net_http;*/ This line allows you to tell nix not to refer to the remote repo but your local clone of fractal_net_http. Comment out the fetchFromGitHub expression and uncomment the above local repo clone path. When you publish your fractal upstream, ensure this line is commented out! Please use a relative path compatible with the above directory structure convention as it will work for everyone and we don’t have to hunt for the correct folder. Just un/comment and go!

  • The line: import fractal {inherit pkgs support edges nodes; fractalide = null;} is where we import your closures into `fractalide’s set of closures. Thus making your nodes available to everyone.

The last step is to expose the exact nodes / edges to dev/fractalide/nodes/default.nix and dev/fractalide/edges/default.nix.
This is done in this manner:

  • Open dev/fractalide/nodes/default.nix and seek out the net_http_nodes attribute. This is what it looks like: net_http_nodes = fractals.net_http.nodes; In this case there is no specific top level node you’d wish to expose so the convention _nodes is used to indicate this. Whereas if you have a specific node you’d wish to expose then you’d name it as such: net_http = fractals.net_http.nodes.http. Why the .http? well that’s what the node is named in the namespace here. Please notice the lack of the *_nodes when exporting a single node.

  • Regarding importing edges, typically you don’t need to import edges, but there are times when you need a special edge which must operate on the public side of the fractal and thus usable across a number of fractals, say the net_http_edges.request You’d use a similar mechanism as above when exposing your edges.

5.6. Incremental Builds

Incremental Builds speed up the development process, so that one doesn’t have to compile the entire crate from scratch each time you make a change to the source code.

Fractalide expands the nix-build system for incremental builds. The Incremental Builds only work when debug is enabled. They also need the path to a cache folder. The cache folder can be created from an old result by the buildCache.sh script. Per default the cache folder is saved in the /tmp folder of your system. Incremental Builds permit you to compile a crate without having to recompiled the crate dependency tree.

Here is an example how you can build with the Incremental Build System:

$ cd dev/fractalide
$ nix-build --argstr debug true --argstr cache $(./support/buildCache.sh) --argstr subgraph workbench

If you’re using NixOS, please ensure you have not set nix.useSandbox = true;, otherwise Incremental Compilation will fail.

5.7. There is a service.nix file! What is it?

5.8. Two ways to execute your fractal

5.8.1. Executing from within Fractalide

$ cd dev/fractalide
$ nix-build --argstr rs workbench
$ ./result
  • advantages

    • Incremental recompilation needed for development

  • disadvantages

    • wetware needed to plug into dev/fractalide/fractals to get incremental recompilation

    • long build command

5.8.2. Executing from with the Fractal

$ cd /dev/fractals/fractal_workbench
$ nix-build
$ ./result
  • advantages

    • faster to test by just issuing the nix-build command

    • convenient for CI & CD of you specific subgraph

    • don’t have to plug it into dev/fractalide/fractals

  • disadvantages

    • no incremental recompilation

6. HOWTO

6.1. Steps

6.1.1. Fractalide installation

Virtualbox guest installation
Building the Fractalide Virtual Machine (FVM)

Once logged into your virtualbox guest issue these commands:

$ git clone https://github.com/fractalide/fractalide.git
$ cd fractalide
$ nix-build

Let us inspect the content of the newly created symlink called result.

$ readlink result
/nix/store/ymfqavzrgmj3q3aljgwvh769fq9dszp2-fvm
$ tree result
result
└── bin
    └── fvm

$ file result/bin/fvm
result/bin/fvm: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/8lbpq1vmajrbnc96xhv84r87fa4wvfds-glibc-2.24/lib/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped
Peek under the hood

You shouldn’t need to care too much about this during your everyday programming, but it’s pleasant deviation from most normal workflows and thus should be explained.

Let’s build a subgraph that runs a contrived maths_boolean_nand agent.

$ nix-build --argstr node test_nand

This replaces the result symlink with a new symlink pointing to a generated file.

$ readlink result
/nix/store/zld4d7zc80wh38qhn00jqgc6lybd2cdi-test_nand

Let’s investigate the contents of this executable file.

$ cat result
/nix/store/ymfqavzrgmj3q3aljgwvh769fq9dszp2-fvm/bin/fvm /nix/store/jk5ibldrvi6cai5aj1j00p8rgi3zw4l7-test_nand

Notice that we’re passing the path of the actual test_nand subgraph into the fvm.

What does the contents of the actual /nix/store/jk5ibldrvi6cai5aj1j00p8rgi3zw4l7-test_nand file look like (the argument to fvm)?

$ cat /nix/store/jk5ibldrvi6cai5aj1j00p8rgi3zw4l7-test_nand/lib/lib.subgraph
'/nix/store/ynm9ipggdvxhzi5l2kkz9cgiqgvq2g87-prim_bool:(bool=true)' -> a nand(/nix/store/y919fp98qw33w0cs2wn5wzwgwpwgbchs-maths_boolean_nand) output -> input io_print(/nix/store/4fnk9dmky6jni4f4sbrzl1xsj50m3mb0-maths_boolean_print)
'/nix/store/ynm9ipggdvxhzi5l2kkz9cgiqgvq2g87-prim_bool:(bool=true)' -> b nand()

$ file /nix/store/jk5ibldrvi6cai5aj1j00p8rgi3zw4l7-test_nand/lib/lib.subgraph
/nix/store/jk5ibldrvi6cai5aj1j00p8rgi3zw4l7-test_nand/lib/lib.subgraph: ASCII text

The --argstr node xxx are arguments passed into the nix-build executable. Specifically

$ man nix-build
...
       --argstr name value
           This option is like --arg, only the value is not a Nix expression but a string. So instead
           of --arg system \"i686-linux\" (the outer quotes are to keep the shell happy) you can say
           --argstr system i686-linux.
...

The name node refers to the top level graph to be executed by the fvm. nix compiles each of the agents and inserts their paths into subgraphs. The fvm knows how how to recursively load the entire hierarchy of subgraphs which contain fully qualified paths to their composed agents.

Quick feel of the system
A = (Graph setup + tear down):
$ nix-build --argstr node bench_load
/nix/store/ij8jri0z1k5n447f9s0x5yfx5p9iqnnf-bench_load

$ sudo nice -n -20 perf stat -r 10 -d ./result
...
       3.684139058 seconds time elapsed                                          ( +-  0.56% )
B = (Graph setup + tear down + message pass + increment):
$ nix-build --argstr node bench
/nix/store/mfl206ccv86wvyi2ra5296l8n1bks24x-bench

$ sudo nice -n -20 perf stat -r 10 -d ./result

 Performance counter stats for './result' (10 runs):

       6638.755996      task-clock (msec)         #    1.443 CPUs utilized            ( +-  0.57% )
           268,864      context-switches          #    0.040 M/sec                    ( +-  0.47% )
             3,047      cpu-migrations            #    0.459 K/sec                    ( +- 10.08% )
            82,417      page-faults               #    0.012 M/sec                    ( +-  0.03% )
    18,012,749,608      cycles                    #    2.713 GHz                      ( +-  0.66% )  (50.10%)
   <not supported>      stalled-cycles-frontend
   <not supported>      stalled-cycles-backend
    18,396,303,772      instructions              #    1.02  insns per cycle          ( +-  0.10% )  (62.48%)
     3,008,536,908      branches                  #  453.178 M/sec                    ( +-  0.06% )  (73.97%)
        13,396,472      branch-misses             #    0.45% of all branches          ( +-  1.01% )  (74.08%)
     6,955,828,023      L1-dcache-loads           # 1047.761 M/sec                    ( +-  0.50% )  (63.04%)
       184,998,022      L1-dcache-load-misses     #    2.66% of all L1-dcache hits    ( +-  0.81% )  (29.73%)
        49,018,759      LLC-loads                 #    7.384 M/sec                    ( +-  0.99% )  (26.13%)
         3,032,354      LLC-load-misses           #    6.19% of all LL-cache hits     ( +-  1.56% )  (37.74%)

       4.601455409 seconds time elapsed                                          ( +-  0.66% )
(Message Passing + Increment) = B - A:
>>> 4.601455409 - 3.684139058
0.9173163509999998

This just gives you a feel for the system:

  • 3.7 secs to setup 10,000 [rust agents](./nodes/bench/inc/lib.rs) + teardown 10,000 agents.

  • 4.6 sces to setup 10,000 agents + message pass 10,000 times + increment 10,000 times + teardown 10,000 agents.

  • 0.9 sec to message pass 10,000 times + increment 10,000 times.

A Todo backend

We will design an http server backend that’ll host a set of todos. It will provide the following HTTP features : GET, POST, PATCH/PUT, DELETE. The actual todos will be saved in a sqlite database. The client will use json to communicate with the server.

A todo had the following fields :

  • id : a unique integer id, that is used to retrieve, delete and patch the todos.

  • title : a string, that represents the goal of the todo and will be displayed.

  • completed : a boolean, to remember if the todo has been completed or not.

  • order : a positive integer, used to display the todos in a certain order.

The http server responds to these requests:

  • GET
    The request looks like GET http://localhost:8000/todos/1. The server, after it receives a "GET" request along with a numeric id, will respond with the corresponding todo in the database, otherwise it will return a 404.

  • POST
    The request looks like POST http://localhost:8000/todos. The content of the request must be json that correspond to a todo. The id field is ignored. e.g. : { "title": "Create a todo http server", "order": 1 }

  • PATCH or PUT
    The request looks like PUT http://localhost:8000/todos/1. The content of the request is the fields to update. ex : { "completed": true }

  • Delete
    The request looks like DELETE http://localhost:8000/todos/1. This will delete the todo with the id 1.

The Big Picture
global http

The centre of gravity revolves around the http agent. It receives requests from users and dispatches them to four other subgraphs, one subgraph for each HTTP feature. Each subgraph processes the request and provide a response. Before we approach the HTTP feature subgraphs let’s take a look at the http agent.

The HTTP Agent

The implementation code can be found here.

The http agent

The http agent has one array output port for each HTTP method, and the elements of each array output ports is actually a fast rust regex.

For example, http() GET[^/news/?$] will match the request with method GET and url http://../news or http://../news/.

A Msg is sent on the output port of http with the schema net_http_request. We will just use the fields id, url, content. The id is the unique id for the request. It must be provided in the response corresponding to this request. The url is the url given by the user. The content is the content of the request, or the data given by the user.

The http agent expects a Msg with the schema net_http_response. A response has an id, which corresponds to the request id. It also has a status_code, which is the response code of the request. By default, it’s 200 (OK). The content is the data that is sent back to the user.

The http agent must be started with an iMsg of type net_http_address. It specifies the address and port on which the server listens:

http listen
The GET Subgraph
get
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes; with edges; ''
    db_path => db_path get_sql()
    input => input id(${todo_get_id}) id -> get get_sql(${sqlite_local_get})
    get_sql() id -> id todo_build_json(${todo_build_json})
    get_sql() response -> todo todo_build_json()
    id() req_id -> id todo_add_req_id(${todo_add_req_id})
    todo_build_json() json -> playload build_resp(${todo_build_response})
    get_sql() error -> error build_resp()
    build_resp() response -> response todo_add_req_id() response => response
   '';
}

A request follows this path:

  • Enters the subgraph via the virtual port request

  • Then enters the agent get_id. This agent has two output ports : req_id and id. The req_id is the id of the http request, given by the http agent. The id is todo id retrieved from the url (ie: given the url http://../todos/2, the number 2 will be sent over the id port).

  • The url id enters the sql_get agent, that retrieve a Msg from a database corresponding to the id.

  • If the id exists, a Msg is send to build_json that contains the json of the todo.

  • If the id doesn’t exist in the database, a Msg is send on the error port.

  • The build_request will receive Msg on one of its two input ports (error or playload). If there is an error, it will send a 404 response, or otherwise, it will send a 200 repsonse with the json as data.

  • This new response now goes into the add_req_id agent, which retrieves the req_id from the request, and sets it in the new response.

  • The response now leaves the subgraph.

Now we can connect the http agent to the get subgraph, to retrieve all the GET http request.

http_get
http() GET[^/todos/.+$] -> request get()
get() response -> response http()

Please understand how the code maps to the above diagram, as these particular diagrams shall not be repeated.

The POST Subgraph
post
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes; with edges; ''
    db_path => db_path insert_todo()
    input => input todo_get_todo(${todo_get_todo}) todo -> input cl_todo(${msg_clone})
    cl_todo() clone[0] -> insert insert_todo(${sqlite_local_insert})
    cl_todo() clone[1] -> todo todo_build_json(${todo_build_json})
    insert_todo() response -> id todo_build_json()
    todo_get_todo() req_id -> id todo_add_req_id(${todo_add_req_id})
    todo_build_json() json -> playload todo_build_response(${todo_build_response})
    todo_build_response() response -> response todo_add_req_id() response => response
   '';
}

A request will follow this path :

  • Enters the subgraph by the virtual port request

  • Enters the agent``get_todo. get_todo sends req_id and the content, which is converted from json into a new schema app_todo.

  • The todo schema is then cloned and sent to two agents.

  • One clone goes to sql_insert, which sends out the url id of the todo found in the database. This id is send in build_json.

  • The build_json receives the database id and the todo, and merges them together in json format.

  • This approach allows the building of a response with json as the content.

  • add_req_id then add the req_id in the reponse

  • The response is sent out

The post subgraph is then connected to the http output port :

http() POST[/todos/?$] -> request post()
post() response -> response http()
The DELETE Subgraph
delete
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes; with edges; ''
    input => input id(${todo_get_id})
    db_path => db_path delete_sql()
    id() id -> delete delete_sql(${sqlite_local_delete})
    delete_sql() response -> playload build_resp(${todo_build_response})
    id() req_id -> id todo_add_req_id(${todo_add_req_id})
    build_resp() response -> response todo_add_req_id() response => response
   '';
}

This subgraph is easier than the two before, hence nearly self-explainatory.

  • The req_id and the id are obtained in get_id.

  • The id is send to sql_delete, which returns the id to build_response.

  • build_response simply fill the http response with the id

  • add_req_id add the http id

The delete subgraph is connect to the http output port :

http() DELETE[/todos/.+] -> request delete()
delete() response -> response http()
The PATCH Subgraph
patch

The patch subgraph is a little more complicated, because of the synch agent. Let first see what happend without it :

patch_without_sync

The idea of the stream is this:

  • Get the new "todos" values in the request

  • In parallel, retrieve the old value of the todo from the database.

  • Then, send the old and the new values to a merge agent, which builds the resulting todo

Now this graph has a problem; if there the todo is new then an old todo cannot be found in the database. In this case, the new edge between get_todo and merge and the error edge between sql_get and build_respone are completely concurrent, thus an issue will arise if a Msg is sent over the error edge when sql_get cannot find a todo in the database. At the same time get_todo will have recognized that it’s a new todo and will have sent a Msg over the new edge. This will insert 2 Msgs into the old input port, where the first Msg is incorrect. A solution is to add a synch agent which has outgoing edges old/new and error. If an error is received, it’s immediately communicated to build_respone and discards the old/new Msg. If it receives a new Msg, it forwards the new and old Msgs to merge. This ensures all Msgs are well taken care of.

To simplify the graph a little, we’ve not mentioned the edge from synch to patch_sql. A Msg is send from the former with the todo id, whichs need to be updated. But all the logic, with synch, is exactly the same. The complete figure is:

patch_final
{ subgraph, nodes, edges }:

subgraph {
  src = ./.;
  flowscript = with nodes; with edges; ''
    input => input todo_get_todo(${todo_get_todo})
    db_path => db_path patch_sql()
    todo_get_todo() id -> get get_sql(${sqlite_local_get})
    synch(${todo_patch_synch})
    get_sql() response -> todo synch() todo -> old merge(${todo_patch_json})
    todo_get_todo() raw_todo -> raw_todo synch() raw_todo -> new merge()
    get_sql() id -> id synch() id -> id patch_sql(${sqlite_local_patch})
    merge() todo -> msg patch_sql()
    patch_sql() response -> playload build_resp(${todo_build_response})
    get_sql() error -> error synch() error -> error build_resp()
    todo_get_todo() req_id -> id todo_add_req_id(${todo_add_req_id})
    build_resp() response -> response todo_add_req_id() response => response
   '';
}
Executing the graph
$ nix-build --argstr node workbench_test
$ ./result

Now’s the time to test the graph. Please follow these steps:

  • Open firefox:

  • Install and open the resteasy firefox plugin

  • Post : http://localhost:8000/todos/

  • Open "data"

  • Select "custom"

  • Keep Mime type empty

  • Put { "title": "A new title" } in the textbox.

  • Click send

  • Notice the 200 response.

You can also fiddle with

Install into your environement via Configuration.nix

Insert this into your Configuration.nix

{ config, pkgs, ... }:

let
  fractalide = import /path/to/your/cloned/fractalide {};
in
{
  require = fractalide.services;
  services.workbench = {
    enable = true;
    bindAddress = "127.0.0.1";
    port = 8003;
  };
...
}
$ sudo nixos-rebuild switch -I fractalide=/path/to/your/cloned/fractalide

6.2. Tokio-*

We’re waiting patiently for the much anticipated https://github.com/tokio-rs/ code to land. That’s when we’ll get services talking to other services and http clients via tokio.