Skip to main content
net8.0

Pipes is a clean and powerful stream processing library that lets you build and connect reusable streaming components

Published
Dec 19, 2024
1y ago
Total Downloads
10.9K
10,905

Install

dotnet add package LanguageExt.Pipes --version 5.0.0-beta-42
ProductVersions
.NET
net8.0

README

LanguageExt.Pipes

This feature of language-ext is based on the wonderful work of Gabriella Gonzalez on the Haskell Pipes library. I have had to make some significant changes to make it work in C#, but the essence is the same, and the core types and composition of the components is exactly the same.

  • If you find this feature confusing at first, and it wouldn't be surprising as it's quite a complex idea, there are some examples in the EffectsExample sample in the repo

Conventional stream programming forces you to choose only two of the following three features:

  1. Effects
  2. Streaming
  3. Composability

If you sacrifice Effects you get IEnumerable, which you can transform using composable functions in constant space, but without interleaving effects (other than of the imperative kind).

If you sacrifice Streaming you get 'Traverse' and 'Sequence', which are composable and effectful, but do not return a single result until the whole list has first been processed and loaded into memory.

If you sacrifice Composability you write a tightly coupled for loops, and fire off imperative side-effects like they're going out of style. Which is streaming and effectful, but is not modular or separable.

Pipes gives you all three features: effectful, streaming, and composable programming. Pipes also provides a wide variety of stream programming abstractions which are all subsets of a single unified machinery:

On top of that, Pipes has more advanced features, including bi-directional streaming. This comes into play when fusing clients and servers:

All of these are connectable and you can combine them together in clever and unexpected ways because they all share the same underlying type.

The pipes ecosystem decouples stream processing stages from each other so that you can mix and match diverse stages to produce useful streaming programs. If you are a library writer, pipes lets you package up streaming components into a reusable interface. If you are an application writer, pipes lets you connect pre-made streaming components with minimal effort to produce a highly-efficient program that streams data in constant memory.

To enforce loose coupling, components can only communicate using two commands:

Pipes has four types of components built around these two commands:

Pipes uses parametric polymorphism (i.e. generics) to overload all operations.

You've probably noticed this overloading already:

This overloading is great when it works, but when connections fail they produce type errors that appear intimidating at first. This section explains the underlying types so that you can work through type errors intelligently.

Producer, Consumer, Pipe, and Effect are all special cases of a single underlying type: Proxy. This overarching type permits fully bidirectional communication on both an upstream and downstream interface.

You can think of it as having the following shape:

Proxy<RT, UOut, UIn, DIn, DOut, A>

      Upstream | Downstream
          +---------+
          |         |
    UOut â—„--       â—„-- DIn   -- Information flowing upstream
          |         |
    UIn --â–º        --â–º DOut  -- Information flowing downstream
          |    |    |
          +----|----+
               |
               A

The four core types do not use the upstream flow of information. This means that the UOut and DIn in the above diagram go unused unless you use the more advanced features.

Pipes uses type synonyms to hide unused inputs or outputs and clean up type signatures. These type synonyms come in two flavors:

  • Concrete type synonyms that explicitly close unused inputs and outputs of the Proxy type

  • Polymorphic type synonyms that don't explicitly close unused inputs or outputs

The concrete type synonyms use Unit to close unused inputs and Void (the uninhabited type) to close unused outputs:

  • Effect: explicitly closes both ends, forbidding awaiting and yield

     Effect<RT, A> = Proxy<RT, Void, Unit, Unit, Void, A>
      
              Upstream | Downstream
                  +---------+
                  |         |
            Void â—„--       â—„-- Unit
                  |         |
            Unit --â–º       --â–º Void
                  |    |    |
                  +----|----+
                       |
                       A
    
  • Producer: explicitly closes the upstream end, forbidding awaiting

     Producer<RT, OUT, A> = Proxy<RT, Void, Unit, Unit, OUT, A>
      
              Upstream | Downstream
                  +---------+
                  |         |
            Void â—„--       â—„-- Unit
                  |         |
            Unit --â–º       --â–º OUT
                  |    |    |
                  +----|----+
                       |
                       A
    
  • Consumer: explicitly closes the downstream end, forbidding yield

      Consumer<RT, IN, A> = Proxy<RT, Unit, IN, Unit, Void, A>
     
              Upstream | Downstream
                  +---------+
                  |         |
            Unit â—„--       â—„-- Unit
                  |         |
              IN --â–º       --â–º Void
                  |    |    |
                  +----|----+
                       |
                       A
    
  • Pipe: marks both ends open, allowing both awaiting and yield

      Pipe<RT, IN, OUT, A> = Proxy<RT, Unit, IN, Unit, OUT, A>
       
              Upstream | Downstream
                  +---------+
                  |         |
            Unit â—„--       â—„-- Unit
                  |         |
              IN --â–º       --â–º OUT
                  |    |    |
                  +----|----+
                       |
                       A
    

When you compose Proxy using | all you are doing is placing them side by side and fusing them laterally. For example, when you compose a Producer, Pipe, and a Consumer, you can think of information flowing like this:

            Producer                Pipe                 Consumer
         +------------+          +------------+          +-------------+
         |            |          |            |          |             |
   Void â—„--          â—„--  Unit   â—„--         â—„--  Unit  â—„--           â—„-- Unit
         |  readLine  |          |  parseInt  |          |  writeLine  |
   Unit --â–º         --â–º string  --â–º          --â–º string --â–º           --â–º Void
         |     |      |          |    |       |          |      |      |
         +-----|------+          +----|-------+          +------|------+
               v                     v                       v
               ()                    ()                      ()

Composition fuses away the intermediate interfaces, leaving behind an Effect:

                       Effect
        +-----------------------------------+
        |                                   |
  Void â—„--                                 â—„-- Unit
        |  readLine | parseInt | writeLine  |
  Unit --â–º                                 --â–º Void
        |                                   |
        +----------------|------------------+
                        Unit

Changes from Previous Version

No structural changes between v5.0.0-beta-41 and v5.0.0-beta-42.

Adoption Assessment

Assess this package against its supported frameworks, license, maintenance signals and dependency graph before making it a project standard.

This version is marked prerelease. Validate its API and upgrade path before production use.Compatibility metadata lists 1 target framework; confirm the exact target used by your application.

Related Guides & Workflows

Learn how to evaluate, integrate, or audit this package in your .NET project.