Quickstart

This guide will walk you through the process of setting up Fluvio and then will introduce you to the basic concepts of Fluvio.

Install Fluvio

Fluvio is installed via the Fluvio Version Manager, also known as fvm.

To install fvm, run the following command:

curl -fsS https://hub.infinyon.cloud/install/install.sh | bash

As part of the initial setup, fvm will also install the Fluvio CLI available in the stable channel as of the moment of installation.

Fluvio is stored in $HOME/.fluvio, with the executable binaries stored in $HOME/.fluvio/bin.

Example:

$ curl -fsS https://hub.infinyon.cloud/install/install.sh | bash

☁️  Downloading fluvio version manager, fvm
   target arch aarch64-apple-darwin
⬇️ Installing fvm
done: FVM installed successfully at /Users/telant/.fvm
help: Add FVM to PATH using source $HOME/.fvm/env
fluvio install dir /Users/$USER/.fluvio
If version of fluvio is already installed, you can run 'fvm install' or 'fvm switch' to change versions
☁️ Installing fluvio
info: Downloading (1/5): fluvio@0.11.0
info: Downloading (2/5): fluvio-cloud@0.2.15
info: Downloading (3/5): fluvio-run@0.11.0
info: Downloading (4/5): cdk@0.11.0
info: Downloading (5/5): smdk@0.11.0
done: Installed fluvio version 0.11.0
done: Now using fluvio version 0.11.0
🎉 Install complete!
fluvio: 💡 You'll need to add '~/.fvm/bin' and ~/.fluvio/bin/' to your PATH variable
fluvio:     You can run the following to set your PATH on shell startup:
fluvio:       echo 'export PATH="${HOME}/.fvm/bin:${HOME}/.fluvio/bin:${PATH}"' >> ~/.zshrc

Create your first Fluvio cluster

Create a free InfinyOn Cloud account to get access to InfinyOn Hub. The Hub is a data streaming store for pre-built connectors, smartmodules and other workflow components.

Your InfinyOn Cloud account has free credits, which you may use to run Fluvio clusters in the cloud.

Head over to the InfinyOn Cloud sign-up page and create an account. Depending on which method you choose in account creation, you can log in with OAuth2 or username/password.

Login to InfinyOn Cloud using OAuth

$ fluvio cloud login --use-oauth2
A web browser has been opened at https://infinyon-cloud.us.auth0.com/activate?user_code=GLMC-QDDJ.
Please proceed with authentication.

Login with your Username/Password

$ fluvio cloud login
InfinyOn Cloud email: john@example.com
Password:

Start a Cluster

Start cluster on InfinyOn Cloud:

$ fluvio cloud cluster create

Create your first topic

Topics are used to store data and send data streams.

You can create a topic with the following command:

$ fluvio topic create quickstart-topic

Where quickstart-topic is the name of your topic

Produce Records

You can send data (aka produce) to your topic.

Let’s try to produce text to your topic interactively:

$ fluvio produce quickstart-topic
> hello world!
Ok!

Typing anything and then pressing Enter will send a record to your topic.

Press Ctrl+C to exit the interactive producer prompt.

Consume Records

You can read data (aka consume) from your topic.

This command will create a consumer that listens to your topic for new records and then prints it to the screen:

$ fluvio consume quickstart-topic
Consuming records from the end of topic 'quickstart-topic'. This will wait for new records

To see this in action, open another terminal and produce new data.

To see previously sent data, you can add an option to your consume command to request a starting offset with the -B <offset> flag.

$ fluvio consume quickstart-topic -B -d
hello world!

Flags:

• -B flag is used to specify the starting offset. Defaults to 0
• -d closes the consumer connection after all data has been sent.

Integrate with Connectors

InfinyOn offers a growing number of connectors to communicate with external services. In this example, we will be covering the HTTP Source connector. The connector polls data from an HTTP endpoint that returns a random quote every 3 seconds to a topic called quotes.

Save the following configuration file on your machine:

apiVersion: 0.1.0
meta:
  version: 0.2.5
  name: http-quotes
  type: http-source
  topic: quotes
http:
  endpoint: https://demo-data.infinyon.com/api/quote
  interval: 3s

Running the HTTP Connector

To start a connector in InfinyOn Cloud, use the following command:

$ fluvio cloud connector create -c quotes-source-connector.yml

Use the following command to see the connector status.

$ fluvio cloud connector list
 NAME                 TYPE               VERSION  CDK  STATUS   LOG-LEVEL
 http-quotes          http-source        0.2.6    V3   Running  info

We can monitor new data in the connector’s topic with fluvio consume quotes

$ fluvio consume quotes
Consuming records from 'quotes'
{"quote":"We cannot solve our problems with the same thinking we used when we created them.","by":"Albert Einstein"}
{"quote":"Whatever you are, be a good one.","by":"Abraham Lincoln"}
{"quote":"You can't build a reputation on what you're going to do.","by":"Henry Ford"}
{"quote":"Success is not final, failure is not fatal: It is the courage to continue that counts.","by":"Winston Churchill"}

You may delete your cloud connector with the following command:

fluvio cloud connector delete http-quotes

Transform Data with SmartModules

SmartModules are user-defined functions compiled into WebAssembly and applied to data streaming for inline data manipulation. You can use SmartModules in the producers, consumers, as well as Connectors. InfinyOn has several pre-compiled SmartModules that you can use out of the box. Alternatively, you use SmartModule Developer Kit (smdk) to build your own.

Download a SmartModule from the Hub

InfinyOn Hub has a growing library of SmartModules available for download:

$ fluvio hub smartmodule list
  SMARTMODULE                              Visibility
  infinyon-labs/array-map-json@0.1.0       public
  infinyon-labs/dedup-filter@0.0.2         public
  infinyon-labs/json-formatter@0.1.0       public
  infinyon-labs/key-gen-json@0.1.0         public
  infinyon-labs/regex-map-json@0.1.1       public
  infinyon-labs/regex-map@0.1.0            public
  infinyon-labs/rss-json@0.1.0             public
  infinyon-labs/stars-forks-changes@0.1.2  public
  infinyon/jolt@0.3.0                      public
  infinyon/json-sql@0.2.1                  public
  infinyon/regex-filter@0.1.0              public

Using the jolt SmartModule to turn JSON records into sentences

Let’s download the Smartmodule to our cluster:

$ fluvio hub smartmodule download infinyon/jolt@0.3.0
... cluster smartmodule install complete

Check the cluster to ensure it has been successfully downloaded:

$ fluvio smartmodule list
  SMARTMODULE          SIZE
  infinyon/jolt@0.3.0  611.5 KB

Next, we’ll create a transform file and test the output.

Create a SmartModule transformation file

SmartModules can be chained together and often require additional parameters. Fluvio uses a YAML file is used to define the transformations.

transforms:
  - uses: infinyon/jolt@0.3.0
    with:
      spec:
        - operation: shift
          spec:
            quote: ""

Jolt is a complex Smartmodule that allows you to perform multiple types of JSON transformations. For additional information, check out the SmartModule Jolt docs.

Test the SmartModule

$  fluvio consume quotes --transforms-file tranforms.yaml -T=2
Consuming records from 'quotes' starting 2 from the end of log
"The greatest glory in living lies not in never falling, but in rising every time we fall."
"Simplicity is the ultimate sophistication."

We are consuming the last two quotes topic records and transforming the json into a string.

Creating a data pipeline applying the SmartModule to the Connector

Let’s say we don’t use the authors in the quotes; instead, only the quote represented strings. We can accomplish this result by simply applying the transformation to the connector.

Let’s create a new http-source connector and add the transformation:

apiVersion: 0.1.0
meta:
  version: 0.2.5
  name: string-quotes
  type: http-source
  topic: string-quotes
http:
  endpoint: https://demo-data.infinyon.com/api/quote
  interval: 3s
transforms:
  - uses: infinyon/jolt@0.3.0
    with:
      spec:
        - operation: shift
          spec:
            quote: ""

Ready to run the connector:

$ fluvio cloud connector create -c string-quotes-source-connector.yml

Use the following command to see the connector status.

$ fluvio cloud connector list
 NAME                 TYPE               VERSION  CDK  STATUS   LOG-LEVEL
 string-quotes        http-source        0.2.5    V3   Running  info
 http-quotes          http-source        0.2.5    V3   Running  info

Let’s take a look at string-quotes

$ fluvio consume string-quotes
Consuming records from 'string-quotes'
"It's not whether you get knocked down, it's whether you get up."
"Honesty is the first chapter in the book of wisdom."

We now have two topics running in parallel and producing different results with a simple SmartModule transformation. When you apply inline transformations, the number of possibilities is virtually endless.

Clean-up Resources

During this tutorial, we’ve created connectors that continue generating traffic to our cloud cluster. Run the following commands to clean up:

fluvio cloud connector delete http-quotes
fluvio cloud connector delete string-quotes
fluvio topic delete quotes
fluvio topic delete string-quotes

Build Stateful Services (Preview)

Stateful Services is currently in preview. With stateful services, you can chain services, accumulate state, and perform window-based aggregates.

If you are interested to participate in our early access release, sign up here.