Crystal is a transparent, dynamic and open Software-Defined Storage (SDS) system for OpenStack Swift.
The Crystal documentation is auto-generated after every commit and available online at http://crystal-controller.readthedocs.io/en/latest/
Crystal source code is structured in several components:
-
Controller: this project. The Crystal control plane that offers dynamic meta-programming facilities over the data plane.
-
Metric middleware: the middleware (data plane) that executes metrics that enable controllers to dynamically respond to workload changes in real time.
-
Filter middleware: the middleware (data plane) that executes storage filters that intercept object flows to run computations or perform transformations on them.
-
Dashboard: A user-friendly dashboard to manage policies, filters and workload metrics.
This repository contains the code of Crystal Controller, the Software-Defined-Storage (SDS) REST API in the IOStack architecture. It is a Django project that implements the REST API needed to handle filters, storlets and policies on top of Openstack Swift object-storage system. This API also includes a set of python processes who use the PyActive middleware, an Object Oriented implementation of the Actor model. This part allows to create simple policies using a DSL (integrated in the Crystal Controller API) and to deploy them as an actor process, who analyze the system data thanks to the monitoring system, and allows to set or remove filters to tenants depending on the established policy.
The repository is structured with the next folders:
-
doc: The doc folder includes the API specifications where you can find all the calls and parameters accepted by the SDS Controller.
-
puppet: In this folder you can find two subfolders more. The manifests folder, that contains all the config files of the puppet (To read more about puppet click here). On the other hand, the modules folder that contains all dependencies added by puppet. Remember that these modules only are a link to the original repository, so when you clone this repository you need to add the modules or cloning in recursive way. (To read more about submodules click here)
-
scripts: The scripts folder contains all the scripts needed for the project. The file vagrant-init.sh will be executed each time that you start the virtual machine using vagrant.
-
api: The folder contains the API source code. Its structure follows a standard Django project structure.
-
dynamic_policies The dynamic_policies contains the source code of this part.
-
Vagrantfile: This is the vagrant config file, where we define all the information that vagrant needs to start a virtual machine with all the requirements.
To build the APIs in an easy way we use Django REST Framework.
This project includes a Vagrant environment with all the dependencies (Django, pyactive, swiftclient, ...), so the only two requirements are:
- Install Virtualbox
- Install Vagrant
Once you have already installed the requirements, you only need to go to the project location using a terminal, and execute the command: vagrant up. First time, the process may take a few minutes because Vagrant downloads the Operative System to create the Virtual Machine. Next time the process will be faster.
The Virtual Machine that we started has all the tools that we need to run the server. To connect to this machine you only need to run the command vagrant ssh. The repository folder is synchronized between the local machine and the Virtual Machine, so you can develop the code in your local machine with your preferred IDE, and run the project in the Virtual Machine.
You can start the server using the command into the source folder (./api): python manage.py runserver 0.0.0.0:8000. After that, the server starts, and if you prefer to call Crystal controller from the host machine the port to use is 18000. For instance, to list the Storlets from the host machine the URL should be: localhost:18000/storlets.
If some problem appears, make sure that:
- redis-server service is running? Crystal Controller API stores the meta-data information in redis.
- is PyActive in the PYTHONPATH? At home folder you can find the pyactive folder, where you can find another install.txt, please follow this steps.
- review the settings file from Crystal Controller and make sure to write the correct IPs (Swift IP, Keystone IP, PyActive IP)
API usage is detailed in the API specification.
A convenient web dashboard is also available to simplify these API calls. Refer to the dashboard overview for detailed information.
Run unit tests from the source folder (./api) with the following command: python manage.py test
Please open an issue for support.
Please contribute using Github Flow. Create a branch, add commits, and open a pull request.
For more information, please visit crystal-sds.org.
There is an available development Virtual Machine which emulates running a Swift-all-in-one cluster together with Storlets and Crystal controller and middlewares. It also includes an extended version of the OpenStack Dashboard that simplifies the management of Crystal filters, metrics and policies.
- ftp://ast2-deim.urv.cat/s2caio_vm