> For the complete documentation index, see [llms.txt](https://altaisoft.gitbook.io/ysv/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://altaisoft.gitbook.io/ysv/technology-choices.md).

# Technology Choices

## Why Rust?

`ysv` is built to process tens of GBs of data fairly quickly - to prepare data for analysis, import into databases or cloud storage systems. Performance matters, and Rust, with its transportability and safety, seems to be a perfect fit for the job.

## UNIX spirit

Being a command line utility, `ysv` can be trivially integrated with other tools. For example, I am using it in scenarios like this:

```yaml
curl -s https://remote.host/data.csv.gz \
    | gunzip \
    | ysv transform.ysv \
    | pgloader load.pgloader
```

Here, we download data from a remote location, unarchive the data, transform it, and import into a PostgreSQL database. The dataset may be huge, but we do not even store it on local disk: everything happens on-the-fly.

## Configuration language

`ysv` configuration format aims to feel as a very specialized, but purely functional and strongly typed programming language. Every transformation except I/O is a monad. You may imagine an invisible `.map()` put between each two consecutive transformations.

## xsv2schema

`xsv2schema` is a tool written in Python and capable of generating stubs of `ysv` config files based on CSV data. Can save from tedious work writing confgs, and provides an example of how easy it is to generate `ysv` confgurations programmatically.

## Future

Plans:

* [ ] add a multitude of further transformations;
* [ ] implement monadic control structures (analogous to if-else);
* [ ] improve the logging system and teach it to output messages both in plain text and JSON (for integration with other software);
* [ ] improve the command line switches processing;
* [ ] implement concurrency;
* [ ] add further documentation and recipes of how to use `ysv` with various other tools;
* [ ] maybe build a web based or a desktop based UI :)