OpenFn/lightning
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Repository files navigation
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="ExDoc v0.38.4">
<meta name="project" content="Lightning v2.15.1">
<title>Lightning — Lightning v2.15.1</title>
<link rel="stylesheet" href="dist/html-elixir-RLZO5U2C.css" />
<script defer src="dist/sidebar_items-D81971C1.js"></script>
<script defer src="docs_config.js"></script>
<script defer src="dist/html-Y223O6DN.js"></script>
</head>
<body>
<script>(()=>{var t="ex_doc:settings",e="dark";var o="dark",s="light";var E="sidebar_state",n="closed";var r="sidebar_width";var a="sidebar-open";var i=new URLSearchParams(window.location.search),S=i.get("theme")||JSON.parse(localStorage.getItem(t)||"{}").theme;(S===o||S!==s&&window.matchMedia("(prefers-color-scheme: dark)").matches)&&document.body.classList.add(e);var d=sessionStorage.getItem(E),A=d!==n&&!window.matchMedia(`screen and (max-width: ${768}px)`).matches;document.body.classList.toggle(a,A);var c=sessionStorage.getItem(r);c&&document.body.style.setProperty("--sidebarWidth",`${c}px`);var p=/(Macintosh|iPhone|iPad|iPod)/.test(window.navigator.userAgent);document.documentElement.classList.toggle("apple-os",p);})();
</script>
<div class="body-wrapper">
<button id="sidebar-menu" class="sidebar-button sidebar-toggle" aria-label="toggle sidebar" aria-controls="sidebar">
<i class="ri-menu-line ri-lg" title="Collapse/expand sidebar"></i>
</button>
<nav id="sidebar" class="sidebar">
<div class="sidebar-header">
<div class="sidebar-projectInfo">
<a href="https://openfn.github.io/lightning" class="sidebar-projectImage">
<img src="assets/logo.png" alt="Lightning" />
</a>
<div>
<a href="https://openfn.github.io/lightning" class="sidebar-projectName" translate="no">
Lightning
</a>
<div class="sidebar-projectVersion" translate="no">
v2.15.1
</div>
</div>
</div>
<ul id="sidebar-list-nav" class="sidebar-list-nav" role="tablist" data-extras=""></ul>
</div>
</nav>
<output role="status" id="toast"></output>
<main class="content page-extra" id="main" data-type="extras">
<div id="content" class="content-inner">
<div class="top-search">
<div class="search-settings">
<form class="search-bar" action="search.html">
<label class="search-label">
<span class="sr-only">Search documentation of Lightning</span>
<input name="q" type="text" class="search-input" placeholder="Press / to search" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" />
</label>
<button type="submit" class="search-button" aria-label="Submit Search" tabindex="-1">
<i class="ri-search-2-line ri-lg" aria-hidden="true"></i>
</button>
<button type="button" tabindex="-1" class="search-close-button" aria-hidden="true">
<i class="ri-close-line ri-lg" title="Cancel search"></i>
</button>
</form>
<div class="autocomplete">
</div>
<button class="icon-settings display-settings">
<i class="ri-settings-3-line"></i>
<span class="sr-only">Settings</span>
</button>
</div>
</div>
<div id="top-content">
<div class="heading-with-actions top-heading">
<h1>OpenFn/Lightning <a href="https://dl.circleci.com/status-badge/redirect/gh/OpenFn/lightning/tree/main"><img src="https://dl.circleci.com/status-badge/img/gh/OpenFn/lightning/tree/main.svg?style=svg" alt="CircleCI"/></a> <a href="https://codecov.io/gh/OpenFn/lightning"><img src="https://codecov.io/gh/OpenFn/lightning/branch/main/graph/badge.svg?token=FfDMxdGL3a" alt="codecov"/></a> <img src="https://img.shields.io/docker/pulls/openfn/lightning" alt="Docker Pulls"/> <a href="https://digitalpublicgoods.net/r/openfn"><img src="https://img.shields.io/badge/Verified-DPG-3333AB?logo=data:image/svg%2bxml;base64,PHN2ZyB3aWR0aD0iMzEiIGhlaWdodD0iMzMiIHZpZXdCb3g9IjAgMCAzMSAzMyIgZmlsbD0ibm9uZSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPHBhdGggZD0iTTE0LjIwMDggMjEuMzY3OEwxMC4xNzM2IDE4LjAxMjRMMTEuNTIxOSAxNi40MDAzTDEzLjk5MjggMTguNDU5TDE5LjYyNjkgMTIuMjExMUwyMS4xOTA5IDEzLjYxNkwxNC4yMDA4IDIxLjM2NzhaTTI0LjYyNDEgOS4zNTEyN0wyNC44MDcxIDMuMDcyOTdMMTguODgxIDUuMTg2NjJMMTUuMzMxNCAtMi4zMzA4MmUtMDVMMTEuNzgyMSA1LjE4NjYyTDUuODU2MDEgMy4wNzI5N0w2LjAzOTA2IDkuMzUxMjdMMCAxMS4xMTc3TDMuODQ1MjEgMTYuMDg5NUwwIDIxLjA2MTJMNi4wMzkwNiAyMi44Mjc3TDUuODU2MDEgMjkuMTA2TDExLjc4MjEgMjYuOTkyM0wxNS4zMzE0IDMyLjE3OUwxOC44ODEgMjYuOTkyM0wyNC44MDcxIDI5LjEwNkwyNC42MjQxIDIyLjgyNzdMMzAuNjYzMSAyMS4wNjEyTDI2LjgxNzYgMTYuMDg5NUwzMC42NjMxIDExLjExNzdMMjQuNjI0MSA5LjM1MTI3WiIgZmlsbD0id2hpdGUiLz4KPC9zdmc+Cg==" alt="DPG Badge"/></a></h1>
<a href="https://github.com/OpenFn/lightning/blob/main/README.md#L1" title="View Source" class="icon-action" rel="help">
<i class="ri-code-s-slash-line" aria-hidden="true"></i>
<span class="sr-only">View Source</span>
</a>
</div>
<p>Lightning ⚡ (aka "OpenFn v2") is a workflow automation platform that's used to
automate critical business processes and integrate information systems. From
last-mile services to national-level reporting, it boosts efficiency &
effectiveness while enabling secure, stable, scalable interoperability and data
integration at all levels.</p><p><strong>Use it online at <a href="https://app.openfn.org">app.openfn.org</a></strong></p><p><strong>Explore in a sandbox on <a href="#demo-sandbox">demo.openfn.org</a></strong></p><p><strong>Or learn more at
<a href="https://docs.openfn.org/documentation/about-lightning">docs.openfn.org</a></strong></p><blockquote><p>OpenFn <strong>Lightning</strong> is:</p><ul><li>the <strong>latest version</strong> of OpenFn: first launched in 2014, it's been tried
and tested by NGOs and governments in 40+ countries</li><li>fully <strong>open source</strong>: there's no "community edition" and "premium edition",
you get the same product whether you are self-hosting or using the
OpenFn.org software-as-a-service</li><li>the leading <a href="https://digitalpublicgoods.net/">DPGA</a> certified <strong>Digital
Public Good</strong> for workflow automation</li><li>a <a href="https://digitalsquare.org/digital-health-global-goods">Digital Square</a>
certified <strong>Global Good for Health</strong></li></ul></blockquote><p align="center"> <img width="2349" alt="image" src="https://github.com/OpenFn/lightning/assets/8732845/61b1025c-b65e-4a76-a280-1023fcd51592"></p><h2 id="contents" class="section-heading"><a href="#contents" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Contents</span></h2><ul><li><a href="#demo">Demo</a></li><li><a href="#features">Features</a><ul><li><a href="#build">Build</a></li><li><a href="#monitor">Monitor</a></li><li><a href="#manage">Manage</a></li><li><a href="#roles-and-permissions">Roles and permissions</a></li><li><a href="#roadmap">Roadmap</a></li></ul></li><li><a href="#getting-started">Getting Started</a><ul><li><a href="#run-via-docker"><strong>Run</strong> via Docker</a></li><li><a href="#deploy-on-external-infrastructure"><strong>Deploy</strong> on external infrastructure</a></li><li><a href="#dev-on-lightning-locally"><strong>Dev</strong> on Lightning locally</a></li></ul></li><li><a href="#security-and-standards">Security and Standards</a></li><li><a href="#contribute-to-this-project"><strong>Contribute</strong> to this project</a><ul><li><a href="#pick-up-an-issue">Pick up an issue</a></li><li><a href="#open-a-pull-request">Open a pull request</a></li><li><a href="#generate-the-docs-pages">Generate the docs pages</a></li></ul></li><li><a href="#server-specs-for-self-hosting">Server Specs for Self-Hosting</a></li><li><a href="#benchmarking">Benchmarking</a></li><li><a href="#troubleshooting">Troubleshooting</a><ul><li><a href="#problems-with-environment-variables">Problems with environment variables</a></li><li><a href="#problems-with-postgres">Problems with Postgres</a></li><li><a href="#problems-with-debian">Problems with Debian</a></li><li><a href="#problems-with-docker">Problems with Docker</a></li><li><a href="#problems-with-rambo">Problems with Rambo</a></li></ul></li><li><a href="#support">Support</a></li></ul><h2 id="demo" class="section-heading"><a href="#demo" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Demo</span></h2><p>Watch a short <a href="https://www.youtube.com/watch?v=BNaxlHAWb5I">demo video</a> or
explore a <strong><a href="https://demo.openfn.org/">public sandbox</a></strong> with the login details
below, but please note that this deployment is reset every night at 12:00:00 UTC
and is 100% publicly accessible. <strong><em>Don't build anything you want to keep, or
keep private!</em></strong></p><pre><code class="makeup elixir" translate="no"><span class="ss">username</span><span class="p">:</span><span class="w"> </span><span class="n">demo</span><span class="na">@openfn</span><span class="o">.</span><span class="n">org</span><span class="w">
</span><span class="ss">password</span><span class="p">:</span><span class="w"> </span><span class="n">welcome12345</span></code></pre><h2 id="features" class="section-heading"><a href="#features" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Features</span></h2><h3 id="build" class="section-heading"><a href="#build" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Build</span></h3><p>Plan and build workflows using Lightning's visual interface to quickly define
when, where and what you want your automation to do.</p><p align="center"> <img width="1679" alt="image" src="https://github.com/OpenFn/Lightning/assets/8732845/15afafe7-561b-4d79-9cd8-5d31506a9031"></p><p>Use our
<a href="https://github.com/OpenFn/kit/blob/main/packages/cli/README.md#openfncli">CLI</a>
to quickly build, edit and deploy projects from the comfort of your own code
editor.</p><h3 id="monitor" class="section-heading"><a href="#monitor" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Monitor</span></h3><p>Monitor all workflow activity in one place.</p><p align="center"> <img width="1680" alt="image" src="https://github.com/OpenFn/Lightning/assets/8732845/909dba85-e0d1-4bce-8a17-5949386c6375"></p><ul><li>Filter and search runs to identify issues that need addressing and follow how
a specific request has been processed</li><li>Configure alerts to be notified on run failures</li><li>Receive a project digest for a daily/weekly/monthly summary of your project
activity</li></ul><h3 id="manage" class="section-heading"><a href="#manage" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Manage</span></h3><p>Manage users and access by project.</p><p align="center"> <img width="2560" alt="image" src="https://github.com/OpenFn/lightning/assets/8732845/5411e6a6-14f2-4ff1-a37f-d4156cf40e97"></p><h3 id="roles-and-permissions" class="section-heading"><a href="#roles-and-permissions" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Roles and permissions</span></h3><p>Authorization is a central part of Lightning. As such, users are given different
roles which determine what level of access they have for resources in the
application. For more details about roles and permissions in Lightning, please
refer to our
<a href="https://docs.openfn.org/documentation/about-lightning#roles-and-permissions">documentation</a>.</p><h3 id="roadmap" class="section-heading"><a href="#roadmap" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Roadmap</span></h3><p>View our
<a href="https://github.com/orgs/OpenFn/projects/3/views/1">public GitHub project</a> to
see what we're working on now and what's coming next.</p><h2 id="getting-started" class="section-heading"><a href="#getting-started" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Getting Started</span></h2><ul><li>If you only want to <a href="#run-via-docker"><em><strong>RUN</strong></em></a> Lightning on your own server,
we recommend using Docker.</li><li>If you want to <a href="#deploy-on-external-infrastructure"><em><strong>DEPLOY</strong></em></a> Lightning,
we recommend Docker builds and Kubernetes.</li><li>If you want to <a href="#contribute-to-this-project"><em><strong>CONTRIBUTE</strong></em></a> to the project,
we recommend
<a href="#run-lightning-locally">running Lightning on your local machine</a>.</li></ul><h3 id="run-via-docker" class="section-heading"><a href="#run-via-docker" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text"><strong>Run</strong> via Docker</span></h3><ol><li>Install the latest version of
<a href="https://docs.docker.com/engine/install/">Docker</a></li><li>Clone <a href="https://github.com/OpenFn/Lightning">this repo</a> using git</li><li>Setup PostgreSQL database with:
<code class="inline">docker compose build && docker compose run --rm web mix ecto.migrate</code></li><li>Run Lightning and PostgresSQL with: <code class="inline">docker compose up</code></li></ol><p>By default the application will be running at
<a href="http://localhost:4000/">localhost:4000</a>.</p><p>See <a href="#problems-with-docker">"Problems with Docker"</a> for additional
troubleshooting help. Note that you can also create your own
<code class="inline">docker-compose.yml</code> file, configuring a postgres database and using a
<a href="https://hub.docker.com/repository/docker/openfn/lightning">pre-built image</a>
from Dockerhub.</p><h3 id="deploy-on-external-infrastructure" class="section-heading"><a href="#deploy-on-external-infrastructure" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text"><strong>Deploy</strong> on external infrastructure</span></h3><p>Head to the <a href="https://docs.openfn.org/documentation/deploy/options">Deploy</a>
section of our docs site to get started.</p><p>For technical guidelines, see <a href="deployment.html">deployment considerations</a> for
more detailed information.</p><h3 id="dev-on-lightning-locally" class="section-heading"><a href="#dev-on-lightning-locally" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text"><strong>Dev</strong> on Lightning locally</span></h3><h4>Clone the repo and optionally set ENVs</h4><pre><code class="makeup sh" translate="no"><span class="">git clone git@github.com:OpenFn/Lightning.git # or from YOUR fork!
</span><span class="">cd Lightning
</span><span class="">cp .env.example .env # and adjust as necessary!
</span></code></pre><p>Take note of database names and ports in particular—they've got to match across
your Postgres setup and your ENVs. You can run lightning without any ENVs
assuming a vanilla postgres setup (see below), but you may want to make
adjustments.</p><h4>Database Setup</h4><p>If you're already using Postgres locally, create a new database called
<code class="inline">lightning_dev</code>, for example.</p><p>If you'd rather use Docker to set up a Postgres DB, create a new volume and
image:</p><pre><code class="makeup sh" translate="no"><span class="">docker volume create lightning-postgres-data
</span><span class="">
</span><span class="">docker create \
</span><span class=""> --name lightning-postgres \
</span><span class=""> --mount source=lightning-postgres-data,target=/var/lib/postgresql/data \
</span><span class=""> --publish 5432:5432 \
</span><span class=""> -e POSTGRES_PASSWORD=postgres \
</span><span class=""> postgres:15.3-alpine
</span><span class="">
</span><span class="">docker start lightning-postgres
</span></code></pre><h4>Elixir & Ecto Setup</h4><p>We use <a href="https://github.com/asdf-vm/asdf">asdf</a> to configure our local
environments. Included in the repo is a <code class="inline">.tool-versions</code> file that is read by
asdf in order to dynamically make the specified versions of Elixir and Erlang
available.</p><p>You'll need to install asdf plugins for Erlang, NodeJs, Elixir, and k6.
<strong>Important</strong>: Before installing these plugins, make sure to follow the
prerequisite installation steps for each plugin (linked below), as they often
require system dependencies to be installed first:</p><ul><li><a href="https://github.com/asdf-vm/asdf-erlang">Erlang</a> - <strong>Follow the "Before asdf
install" section</strong></li><li><a href="https://github.com/asdf-vm/asdf-nodejs">NodeJs</a></li><li><a href="https://github.com/asdf-vm/asdf-elixir">Elixir</a></li><li><a href="https://github.com/grimoh/asdf-k6">k6</a></li></ul><p>Then follow the installation instructions for each plugin from their respective
documentation pages.</p><p>We use <a href="https://doc.libsodium.org/">libsodium</a> for encoding values as required
by the
<a href="https://docs.github.com/en/rest/guides/encrypting-secrets-for-the-rest-api">Github API</a>.
You'll need to install <code class="inline">libsodium</code> in order for the application to compile.</p><p>For Mac Users:</p><pre><code class="makeup sh" translate="no"><span class="">brew install libsodium cmake
</span></code></pre><p>For Debian Users:</p><pre><code class="makeup sh" translate="no"><span class="">sudo apt-get install libsodium-dev cmake
</span></code></pre><p>You can find more on
<a href="https://doc.libsodium.org/installation">how to install libsodium here</a></p><pre><code class="makeup sh" translate="no"><span class="">asdf install # Install language versions
</span><span class="">./bin/bootstrap
</span></code></pre><h4>Run the app</h4><p>Lightning is a web app. To run it in interactive Elixir mode, start the
development server by running with your environment variables by running:</p><pre><code class="makeup sh" translate="no"><span class="">iex -S mix phx.server
</span></code></pre><p>or if you have set up custom environment variables, run:</p><pre><code class="makeup sh" translate="no"><span class="">env $(cat .env | grep -v "#" | xargs ) iex -S mix phx.server
</span></code></pre><p>Once the server has started, head to <a href="http://localhost:4000"><code class="inline">localhost:4000</code></a>
in your browser.</p><h4>Run the tests</h4><p>Before the first time running the tests, you need a test database setup.</p><pre><code class="makeup sh" translate="no"><span class="">MIX_ENV=test mix ecto.create
</span></code></pre><p>And then after that run the tests using:</p><pre><code class="makeup sh" translate="no"><span class="">MIX_ENV=test mix test
</span></code></pre><p>We also have <code class="inline">test.watch</code> installed which can be used to rerun the tests on file
changes.</p><h2 id="security-and-standards" class="section-heading"><a href="#security-and-standards" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Security and Standards</span></h2><p>We use a host of common Elixir static analysis tools to help us avoid common
pitfalls and make sure we keep everything clean and consistent.</p><p>In addition to our test suite, you can run the following commands:</p><ul><li><code class="inline">mix format --check-formatted</code> Code formatting checker, run again without the
<code class="inline">--check-formatted</code> flag to have your code automatically changed.</li><li><a href="https://hexdocs.pm/dialyxir/1.4.6/Mix.Tasks.Dialyzer.html"><code class="inline">mix dialyzer</code></a> Static analysis for type mismatches and other common warnings.
See <a href="https://github.com/jeremyjh/dialyxir">dialyxir</a>.</li><li><code class="inline">mix credo --strict --all</code> Static analysis for consistency, and coding
standards. See <a href="https://github.com/rrrene/credo">Credo</a>.</li><li><a href="https://hexdocs.pm/sobelow/0.14.0/Mix.Tasks.Sobelow.html"><code class="inline">mix sobelow</code></a> Check for commonly known security exploits. See
<a href="https://sobelow.io/">Sobelow</a>.</li><li><code class="inline">MIX_ENV=test mix coveralls</code> Test coverage reporter. This command also runs
the test suite, and can be used in place of <a href="https://hexdocs.pm/mix/Mix.Tasks.Test.html"><code class="inline">mix test</code></a> when checking
everything before pushing your code. See
<a href="https://github.com/parroty/excoveralls">excoveralls</a>.</li></ul><blockquote><p>For convenience there is a <code class="inline">verify</code> mix task that runs all of the above and
defaults the <code class="inline">MIX_ENV</code> to <code class="inline">test</code>.</p></blockquote><p>For more guidance on security best practices for workflow automation
implementations, check out OpenFn Docs:
<a href="https://docs.openfn.org/documentation/getting-started/security">docs.openfn.org/documentation/getting-started/security</a></p><h2 id="contribute-to-this-project" class="section-heading"><a href="#contribute-to-this-project" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text"><strong>Contribute</strong> to this project</span></h2><p>First, thanks for being here! You're contributing to a digital public good that
will always be free and open source and aimed at serving innovative NGOs,
governments, and social impact organizations the world over! You rock. ❤️</p><p>FYI, Lightning is built in <a href="https://elixir-lang.org/">Elixir</a>, harnessing the
<a href="https://www.phoenixframework.org/">Phoenix Framework</a>. Currently, the only
unbundled dependency is a <a href="https://www.postgresql.org/">PostgreSQL</a> database.</p><p>If you'd like to contribute to this projects, follow the steps below:</p><h3 id="pick-up-an-issue" class="section-heading"><a href="#pick-up-an-issue" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Pick up an issue</span></h3><p>Read through the existing <a href="https://github.com/OpenFn/Lightning/issues">issues</a>,
assign yourself to the issue you have chosen. Leave a comment on the issue to
let us know you'll be working on it, and if you have any questions of
clarifications that would help you get started ask them there - we will get back
to you as soon as possible.</p><p>If there isn't already an issue for the feature you would like to contribute,
please start a discussion in our
<a href="https://community.openfn.org/c/feature-requests/12">community forum</a>.</p><h3 id="open-a-pull-request" class="section-heading"><a href="#open-a-pull-request" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Open a pull request</span></h3><ol><li><p>Clone the Lightning repository, then
<a href="https://docs.github.com/en/get-started/quickstart/fork-a-repo">fork it</a>.</p></li><li><p>Run through <a href="#set-up-your-environment">setting up your environment</a> and make
your changes.</p></li><li><p>Make sure you have written your tests and updated /CHANGELOG.md (in the
'Unreleased' section, add a short description of the changes you are making,
along with a link to your issue).</p></li><li><p>Open a draft pull request by clicking "Contribute > Open Pull Request" from
your forked repository. Fill out the pull request template (this will <strong>be</strong>
added automatically for you), then make sure to self-review your code and go
through the 'Review checklist'. Don't worry about the QA checkbox, our
product manager Amber will tick that once she has reviewed your PR. You can
leave any notes for the reviewer in a comment.</p></li><li><p>Once you're ready to submit a pull request, you can mark your draft PR as
'Ready for review' and assign @stuartc or @taylordowns2000.</p></li></ol><h3 id="generate-the-docs-pages" class="section-heading"><a href="#generate-the-docs-pages" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Generate the docs pages</span></h3><p>You can generate the HTML and EPUB documentation locally using:</p><p><a href="https://hexdocs.pm/ex_doc/0.38.4/Mix.Tasks.Docs.html"><code class="inline">mix docs</code></a> and opening <code class="inline">doc/index.html</code> in your browser.</p><h2 id="server-specs-for-self-hosting" class="section-heading"><a href="#server-specs-for-self-hosting" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Server Specs for Self-Hosting</span></h2><p>For recommend server specifications for self-hosting of Lightning, head to the
<a href="https://docs.openfn.org/documentation/deploy/options">deployment planning</a>
section of the documentation or check out this
<a href="https://community.openfn.org/t/specs-for-self-hosting-lightning/292">self-hosting thread</a>
on our community forum.</p><h2 id="benchmarking" class="section-heading"><a href="#benchmarking" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Benchmarking</span></h2><p>We are using <a href="https://k6.io/">k6</a> to benchmark Lightning. Under <code class="inline">benchmarking</code>
folder you can find a script for benchmarking Webhook Workflows.</p><p>See <a href="benchmarking.md.html">Benchmarking</a> for more detailed information.</p><h2 id="troubleshooting" class="section-heading"><a href="#troubleshooting" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Troubleshooting</span></h2><h3 id="switching-branches-during-development" class="section-heading"><a href="#switching-branches-during-development" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Switching branches during development</span></h3><p>Branches can have different dependencies. Run the bootstrap script to sync
everything:</p><pre><code class="makeup sh" translate="no"><span class="">./bin/bootstrap
</span></code></pre><p>This handles Elixir dependencies, npm packages, native libraries, and database
migrations.</p><p>If the server was running, restart it after bootstrap completes.</p><h3 id="problems-with-environment-variables" class="section-heading"><a href="#problems-with-environment-variables" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Problems with environment variables</span></h3><p>For troubleshooting custom environment variable configuration it's important to
know how an Elixir app loads and modifies configuration. The order is as
follows:</p><ol><li>Stuff in <code class="inline">config.exs</code> is loaded.</li><li><em>That</em> is then modified (think: <em>overwritten</em>) by stuff your ENV-specific
config: <code class="inline">dev.exs</code>, <code class="inline">prod.exs</code> or <code class="inline">test.exs</code>.</li><li><em>That</em> is then modified by <code class="inline">runtime.exs</code> which is where you are allowed to
use <code class="inline">System.env()</code></li><li><em>Finally</em> <code class="inline">init/2</code> (if present in a child application) gets called (which
takes the config which has been set in steps 1-3) when that child application
is started during the parent app startup defined in <code class="inline">application.ex</code>.</li></ol><h3 id="problems-with-postgres" class="section-heading"><a href="#problems-with-postgres" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Problems with Postgres</span></h3><p>If you're having connection issues with Postgres, check the database section of
your <code class="inline">.env</code> to ensure the DB url is correctly set for your environment — note
that composing a DB url out of other, earlier declared variables, does not work
while using <code class="inline">xargs</code>.</p><h3 id="problems-with-debian" class="section-heading"><a href="#problems-with-debian" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Problems with Debian</span></h3><p>If you're getting this error on debian</p><pre><code class="makeup elixir" translate="no"><span class="o">==</span><span class="o">></span><span class="w"> </span><span class="n">earmark_parser</span><span class="w">
</span><span class="nc">Compiling</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">file</span><span class="w"> </span><span class="p" data-group-id="8195906449-1">(</span><span class="o">.</span><span class="n">yrl</span><span class="p" data-group-id="8195906449-1">)</span><span class="w">
</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">erlang</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">parsetools</span><span class="o">-</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">1</span><span class="o">/</span><span class="n">include</span><span class="o">/</span><span class="n">yeccpre</span><span class="o">.</span><span class="ss">hrl</span><span class="p">:</span><span class="w"> </span><span class="n">no</span><span class="w"> </span><span class="n">such</span><span class="w"> </span><span class="n">file</span><span class="w"> </span><span class="ow">or</span><span class="w"> </span><span class="n">directory</span><span class="w">
</span><span class="n">could</span><span class="w"> </span><span class="ow">not</span><span class="w"> </span><span class="n">compile</span><span class="w"> </span><span class="n">dependency</span><span class="w"> </span><span class="ss">:earmark_parser</span><span class="p">,</span><span class="w"> </span><span class="s">"mix compile"</span><span class="w"> </span><span class="n">failed</span><span class="o">.</span><span class="w"> </span><span class="nc">You</span><span class="w"> </span><span class="n">can</span><span class="w"> </span><span class="n">recompile</span><span class="w"> </span><span class="n">this</span><span class="w"> </span><span class="n">dependency</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="s">"mix deps.compile earmark_parser"</span><span class="p">,</span><span class="w"> </span><span class="n">update</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="s">"mix deps.update earmark_parser"</span><span class="w"> </span><span class="ow">or</span><span class="w"> </span><span class="n">clean</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="s">"mix deps.clean earmark_parser"</span></code></pre><p>You need to install erlang development environment <code class="inline">sudo apt install erlang-dev</code>
<a href="https://github.com/elixir-lang/ex_doc/issues/1441">refer to this issue</a></p><h3 id="problems-with-docker" class="section-heading"><a href="#problems-with-docker" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Problems with Docker</span></h3><h4>Versions</h4><p>The build may not work on old versions of Docker and Docker compose. It has been
tested against:</p><pre><code class="makeup elixir" translate="no"><span class="nc">Docker</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="mf">20.10</span><span class="o">.</span><span class="mi">17</span><span class="p">,</span><span class="w"> </span><span class="n">build</span><span class="w"> </span><span class="mi">100</span><span class="n">c701</span><span class="w">
</span><span class="nc">Docker</span><span class="w"> </span><span class="nc">Compose</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="n">v2</span><span class="o">.</span><span class="mf">6.0</span></code></pre><h4>Starting from scratch</h4><p>If you're actively working with docker, you start experiencing issues, and you
would like to start from scratch you can clean up everything and start over like
this:</p><pre><code class="makeup sh" translate="no"><span class=""># To remove any ignored files and reset your .env to its example
</span><span class="">git clean -fdx && cp .env.example .env
</span><span class=""># You can skip the line below if you want to keep your database
</span><span class="">docker compose down --rmi all --volumes
</span><span class="">
</span><span class="">docker compose build --no-cache web && \
</span><span class=""> docker compose create --force-recreate
</span><span class="">
</span><span class="">docker compose run --rm web mix ecto.migrate
</span><span class="">docker compose up
</span></code></pre><h4>Apple Silicon</h4><p>When running <code class="inline">docker compose up</code> on Apple Silicon (aarch64), you might encounter
the following error:</p><pre><code class="makeup elixir" translate="no"><span class="p" data-group-id="1899333135-1">[</span><span class="n">notice</span><span class="p" data-group-id="1899333135-1">]</span><span class="w"> </span><span class="nc">Application</span><span class="w"> </span><span class="n">ssl</span><span class="w"> </span><span class="ss">exited</span><span class="p">:</span><span class="w"> </span><span class="n">exited</span><span class="w"> </span><span class="ss">in</span><span class="p">:</span><span class="w"> </span><span class="nc">:ssl_app</span><span class="o">.</span><span class="n">start</span><span class="p" data-group-id="1899333135-2">(</span><span class="ss">:normal</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="1899333135-3">[</span><span class="p" data-group-id="1899333135-3">]</span><span class="p" data-group-id="1899333135-2">)</span><span class="w">
</span><span class="gt">** (EXIT) an exception was raised:
** (ArgumentError) could not call Module.put_attribute/3 because the module Lightning.MixProject is already compiled
(elixir 1.16.2) lib/module.ex:2360: Module.assert_not_readonly!/2
(elixir 1.16.2) lib/module.ex:2041: Module.__put_attribute__/5
(ssl 11.1.4) ssl_app.erl:35: :ssl_app.stop/1
(kernel 9.2.4) application_master.erl:293: :application_master.start_it_old/4</span><span class="w">
</span><span class="gt">** (MatchError) no match of right hand side value: {:error, {:ssl, {:bad_return, {{:ssl_app, :start, [:normal, []]}, {:EXIT, {%ArgumentError{message: "could not call Module.put_attribute/3 because the module Lightning.MixProject is already compiled"}, [{Module, :assert_not_readonly!, 2, [file: ~c"lib/module.ex", line: 2360]}, {Module, :__put_attribute__, 5, [file: ~c"lib/module.ex", line: 2041]}, {:ssl_app, :stop, 1, [file: ~c"ssl_app.erl", line: 35]}, {:application_master, :start_it_old, 4, [file: ~c"application_master.erl", line: 293]}]}}}}}}</span></code></pre><p>You can resolve this by setting <code class="inline">ERL_FLAGS="+JPperf true"</code> env to the failing
stage. You can follow this thread on our community forum for more info:
<a href="https://community.openfn.org/t/lightning-prebuilt-images-throw-no-matching-manifest-for-linux-arm64-v8-in-the-manifest-list-entries/465/15">no matching manifest for linux/arm64/v8 in the manifest list entries</a></p><h3 id="problems-with-rambo" class="section-heading"><a href="#problems-with-rambo" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Problems with Rambo</span></h3><p>When running <code class="inline">mix compile.rambo</code> on Apple Silicon (an Apple M1/M2, <code class="inline">macarm</code>,
<code class="inline">aarch64-apple-darwin</code>) and encountering the following error:</p><pre><code class="makeup elixir" translate="no"><span class="gt">** (RuntimeError) Rambo does not ship with binaries for your environment.</span><span class="w">
</span><span class="n">aarch64</span><span class="o">-</span><span class="n">apple</span><span class="o">-</span><span class="n">darwin22</span><span class="o">.</span><span class="mf">3.0</span><span class="w"> </span><span class="n">detected</span><span class="w">
</span><span class="nc">Install</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nc">Rust</span><span class="w"> </span><span class="n">compiler</span><span class="w"> </span><span class="n">so</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">binary</span><span class="w"> </span><span class="n">can</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">prepared</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">you</span><span class="o">.</span><span class="w">
</span><span class="n">lib</span><span class="o">/</span><span class="n">mix</span><span class="o">/</span><span class="n">tasks</span><span class="o">/</span><span class="n">compile</span><span class="o">.</span><span class="n">rambo</span><span class="o">.</span><span class="n">ex</span><span class="p">:</span><span class="mi">89</span><span class="p">:</span><span class="w"> </span><span class="nc">Mix.Tasks.Compile.Rambo</span><span class="o">.</span><span class="n">compile!</span><span class="o">/</span><span class="mi">0</span><span class="w">
</span><span class="n">lib</span><span class="o">/</span><span class="n">mix</span><span class="o">/</span><span class="n">tasks</span><span class="o">/</span><span class="n">compile</span><span class="o">.</span><span class="n">rambo</span><span class="o">.</span><span class="n">ex</span><span class="p">:</span><span class="mi">51</span><span class="p">:</span><span class="w"> </span><span class="nc">Mix.Tasks.Compile.Rambo</span><span class="o">.</span><span class="n">run</span><span class="o">/</span><span class="mi">1</span><span class="w">
</span><span class="p" data-group-id="8941309359-1">(</span><span class="n">mix</span><span class="w"> </span><span class="mf">1.14</span><span class="o">.</span><span class="mi">2</span><span class="p" data-group-id="8941309359-1">)</span><span class="w"> </span><span class="n">lib</span><span class="o">/</span><span class="n">mix</span><span class="o">/</span><span class="n">task</span><span class="o">.</span><span class="n">ex</span><span class="p">:</span><span class="mi">421</span><span class="p">:</span><span class="w"> </span><span class="n">anonymous</span><span class="w"> </span><span class="k" data-group-id="8941309359-2">fn</span><span class="o">/</span><span class="mi">3</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="nc">Mix.Task</span><span class="o">.</span><span class="n">run_task</span><span class="o">/</span><span class="mi">4</span><span class="w">
</span><span class="p" data-group-id="8941309359-3">(</span><span class="n">mix</span><span class="w"> </span><span class="mf">1.14</span><span class="o">.</span><span class="mi">2</span><span class="p" data-group-id="8941309359-3">)</span><span class="w"> </span><span class="n">lib</span><span class="o">/</span><span class="n">mix</span><span class="o">/</span><span class="n">cli</span><span class="o">.</span><span class="n">ex</span><span class="p">:</span><span class="mi">84</span><span class="p">:</span><span class="w"> </span><span class="nc">Mix.CLI</span><span class="o">.</span><span class="n">run_task</span><span class="o">/</span><span class="mi">2</span></code></pre><p>You can resolve this error by installing the Rust compiler using Homebrew. Run
the following command in your terminal: <code class="inline">brew install rust</code></p><p>If neither of the approaches above work, please raise an issue.</p><h2 id="support" class="section-heading"><a href="#support" class="hover-link"><i class="ri-link-m" aria-hidden="true"></i></a><span class="text">Support</span></h2><p>If you have any questions, feedback, or issues, please:</p><ul><li>Post on the OpenFn Community at
<a href="https://community.openfn.org">community.openfn.org</a></li><li>Open an issue directly on this GitHub Repo:
<a href="https://github.com/OpenFn/Lightning/issues">github.com/OpenFn/Lightning/issues</a></li></ul>
</div>
<div class="bottom-actions" id="bottom-actions">
<div class="bottom-actions-item">
</div>
<div class="bottom-actions-item">
<a href="runninglocal.html" class="bottom-actions-button" rel="next">
<span class="subheader">
Next Page →
</span>
<span class="title">
Running Locally
</span>
</a>
</div>
</div>
<footer class="footer">
<p>
<span class="line">
<button class="a-main footer-button display-quick-switch" title="Search HexDocs packages">
Search HexDocs
</button>
<a href="Lightning.epub" title="ePub version">
Download ePub version
</a>
</span>
</p>
<p class="built-using">
Built using
<a href="https://github.com/elixir-lang/ex_doc" title="ExDoc" target="_blank" rel="help noopener" translate="no">ExDoc</a> (v0.38.4) for the
<a href="https://elixir-lang.org" title="Elixir" target="_blank" translate="no">Elixir programming language</a>
</p>
</footer>
</div>
</main>
</div>
</body>
</html>
