Skip to main content
net8.0

Pipes is a clean and powerful stream processing library that lets you build and connect reusable streaming components

Published
Apr 7, 2025
1y ago
Total Downloads
10.9K
10,905

Deprecated Package Version

Reason: Legacy

LanguageExt.Pipes was home to the Pipes compositional streaming system. That is now in LanguageExt.Streaming along with lots more streaming functionality!

Suggested alternative:LanguageExt.Streaming*

Install

dotnet add package LanguageExt.Pipes --version 5.0.0-beta-50
ProductVersions
.NET
net8.0

README

LanguageExt.Pipes

If you find this feature confusing at first, and it wouldn't be surprising as it's quite a complex idea, there are some examples in the EffectsExample sample in the repo

Conventional stream programming forces you to choose only two of the following three features:

  1. Effects
  2. Streaming
  3. Composability

If you sacrifice Effects you get IEnumerable, which you can transform using composable functions in constant space, but without interleaving effects (other than of the imperative kind).

If you sacrifice Streaming you get Traverse and Sequence, which are composable and effectful, but do not return a single result until the whole list has first been processed and loaded into memory.

If you sacrifice Composability you write a tightly coupled for loops, and fire off imperative side-effects like they're going out of style. Which is streaming and effectful, but is not modular or separable.

Pipes gives you all three features: effectful, streaming, and composable programming. Pipes also provides a wide variety of stream programming abstractions which are all subsets of a single unified machinery:

The T suffix types (ProducerT, ConsumerT, PipeT, and EffectT) are the more generalist monad-transformers. They can lift any monad M you like into them, supplementing the behaviour of Pipes with the behaviour of M. The non-T suffix types (Producer, Consumer, Pipe, and Effect) only support the lifting of the Eff<RT, A> type. They're slightly easier to use, just less flexible.

All of these are connectable and you can combine them together in clever and unexpected ways because they all share the same underlying type: PipeT.

The pipes ecosystem decouples stream processing stages from each other so that you can mix and match diverse stages to produce useful streaming programs. If you are a library writer, pipes lets you package up streaming components into a reusable interface. If you are an application writer, pipes lets you connect pre-made streaming components with minimal effort to produce a highly-efficient program that streams data in constant memory.

To enforce loose coupling, components can only communicate using two commands:

  • yield: Send output data
  • awaiting: Receive input data

Pipes has four types of components built around these two commands:

  • Producer and ProducerT yield values downstream and can only do so using: Producer.yield and ProducerT.yield.
  • Consumer and ConsumerT await values from upstream and can only do so using: Consumer.awaiting and ConsumerT.awaiting.
  • Pipe and PipeT can both await and yield, using: Pipe.awaiting, PipeT.awaiting, Pipe.yield, and PipeT.yield.
  • Effect and EffectT can neither yield nor await and they model non-streaming components.

Pipes uses parametric polymorphism (i.e. generics) to overload all operations. The operator | connects Producer, Consumer, and Pipe by 'fusing' them together. Eventually they will 'fuse' together into an Effect or EffectT. This final state can be .Run(), you must fuse to an Effect or EffectT to be able to invoke any of the pipelines.

Producer, ProducerT, Consumer, ConsumerT, Pipe, Effect, and EffectT are all special cases of a single underlying type: PipeT.

You can think of it as having the following shape:

PipeT<IN, OUT, M, A>

      Upstream | Downstream
          +---------+
          |         |
     IN --â–º         --â–º OUT  -- Information flowing downstream
          |    |    |
          +----|----+
               |
               A

Pipes uses type synonyms to hide unused inputs or outputs and clean up type signatures. These type synonyms come in two flavors:

  • Concrete type synonyms that explicitly close unused inputs and outputs of the Proxy type.

  • Polymorphic type synonyms that don't explicitly close unused inputs or outputs.

The concrete type synonyms use Unit to close unused inputs and Void (the uninhabited type) to close unused outputs:

  • EffectT: explicitly closes both ends, forbidding awaiting and yield:

     EffectT<M, A> = PipeT<Unit, Void, M, A>
      
              Upstream | Downstream
    
                  +---------+
                  |         |
           Unit --â–º         --â–º Void
                  |    |    |
                  +----|----+
                       |
                       A
    
  • ProducerT: explicitly closes the upstream end, forbidding awaiting:

     ProducerT<OUT, M, A> = PipeT<Unit, OUT, M, A>
      
              Upstream | Downstream
    
                  +---------+
                  |         |
           Unit --â–º         --â–º OUT
                  |    |    |
                  +----|----+
                       |
                       A
    
  • ConsumerT: explicitly closes the downstream end, forbidding yield:

      ConsumerT<IN, M, A> = PipeT<IN, Void, M, A>
     
              Upstream | Downstream
    
                  +---------+
                  |         |
             IN --â–º         --â–º Void
                  |    |    |
                  +----|----+
                       |
                       A
    

When you compose PipeT using | all you are doing is placing them side by side and fusing them laterally. For example, when you compose a ProducerT, PipeT, and a ConsumerT, you can think of information flowing like this:

            ProducerT                     PipeT                  ConsumerT
         +-------------+             +------------+           +-------------+
         |             |             |            |           |             |
  Unit --â–º  readLine   --â–º  string --â–º  parseInt  --â–º  int  --â–º  writeLine  --â–º Void
         |      |      |             |      |     |           |      |      |
         +------|------+             +------|-----+           +------|------+
                |                           |                        |  
                A                           A                        A

Composition fuses away the intermediate interfaces, leaving behind an EffectT:

                        EffectT
        +-------------------------------------+
        |                                     |
 Unit --â–º   readLine | parseInt | writeLine   --â–º Void
        |                                     |
        +------------------|------------------+
                           |
                           A

This EffectT can be Run() which will return the composed underlying M type. Or, if it's an Effect will return the composed underlying Eff<RT, A>.

Changes from Previous Version

v5.0.0-beta-49v5.0.0-beta-50
Deprecated
This version was marked as deprecated.

Adoption Assessment

Assess this package against its supported frameworks, license, maintenance signals and dependency graph before making it a project standard.

This version is marked prerelease. Validate its API and upgrade path before production use.NuGet marks this package as deprecated and points to LanguageExt.Streaming as an alternative.Compatibility metadata lists 1 target framework; confirm the exact target used by your application.

Related Guides & Workflows

Learn how to evaluate, integrate, or audit this package in your .NET project.