retoor/dataset
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove datafreeze component, fixes #217

Browse Source
This commit is contained in:
Friedrich Lindenberg
2017-09-09 18:24:34 +02:00
parent cd091eadca
commit a049691749
24 changed files with 136 additions and 895 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/conf.py
+2 -2
View File
@@ -48,9 +48,9 @@ copyright = u'2013-2015, Friedrich Lindenberg, Gregor Aisch, Stefan Wehrmeyer'
# built documents.
#
# The short X.Y version.
version = '0.6'
version = '1.0'
# The full version, including alpha/beta/rc tags.
release = '0.6.0'
release = '1.0.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
docs/freezefile.rst
-98
View File
@@ -1,98 +0,0 @@
Freezefiles and the ``datafreeze`` command
==========================================
``datafreeze`` creates static extracts of SQL databases for use in interactive
web applications. SQL databases are a great way to manage relational data, but
exposing them on the web to drive data apps can be cumbersome. Often, the
capacities of a proper database are not actually required, a few static JSON
files and a bit of JavaScript can have the same effect. Still, exporting JSON
by hand (or with a custom script) can also become a messy process.
With ``datafreeze``, exports are scripted in a Makefile-like description, making them simple to repeat and replicate.
Basic Usage
-----------
Calling DataFreeze is simple, the application is called with a
freeze file as its argument:
.. code-block:: bash
datafreeze Freezefile.yaml
Freeze files can be either written in JSON or in YAML. The database URI
indicated in the Freezefile can also be overridden via the command line:
datafreeze --db sqlite:///foo.db Freezefile.yaml
Example Freezefile.yaml
-----------------------
A freeze file is composed of a set of scripted queries and
specifications on how their output is to be handled. An example could look
like this:
.. code-block:: yaml
common:
database: "postgresql://user:password@localhost/operational_database"
prefix: my_project/dumps/
format: json
exports:
- query: "SELECT id, title, date FROM events"
filename: "index.json"
- query: "SELECT id, title, date, country FROM events"
filename: "countries/{{country}}.csv"
format: csv
- query: "SELECT * FROM events"
filename: "events/{{id}}.json"
mode: item
- query: "SELECT * FROM events"
filename: "all.json"
format: tabson
An identical JSON configuration can be found in this repository.
Options in detail
-----------------
The freeze file has two main sections, ``common`` and ``exports``. Both
accept many of the same arguments, with ``exports`` specifying a list of
exports while ``common`` defines some shared properties, such as the
database connection string.
The following options are recognized:
* ``database`` is a database URI, including the database type, username
and password, hostname and database name. Valid database types include
``sqlite``, ``mysql`` and ``postgresql`` (requires psycopg2).
* ``prefix`` specifies a common root directory for all extracted files.
* ``format`` identifies the format to be generated, ``csv``, ``json`` and
``tabson`` are supported. ``tabson`` is a condensed JSON
representation in which rows are not represented by objects but by
lists of values.
* ``query`` needs to be a valid SQL statement. All selected fields will
become keys or columns in the output, so it may make sense to define
proper aliases if any overlap is to be expected.
* ``mode`` specifies whether the query output is to be combined into a
single file (``list``) or whether a file should be generated for each
result row (``item``).
* ``filename`` is the output file name, appended to ``prefix``. All
occurences of ``{{field}}`` are expanded to a fields value to allow the
generation of file names e.g. by primary key. In list mode, templating
can be used to group records into several buckets, e.g. by country or
category.
* ``wrap`` can be used to specify whether the output should be wrapped
in a ``results`` hash in JSON output. This defaults to ``true`` for
``list``-mode output and ``false`` for ``item``-mode.
docs/index.rst
+15 -18
View File
@@ -10,22 +10,19 @@ dataset: databases for lazy people
:hidden:
Although managing data in relational database has plenty of benefits, they're rarely used in day-to-day work with small to medium scale datasets. But why is that? Why do we see an awful lot of data stored in static files in CSV or JSON format, even though they are hard
to query and update incrementally?
Although managing data in relational database has plenty of benefits, they're
rarely used in day-to-day work with small to medium scale datasets. But why is
that? Why do we see an awful lot of data stored in static files in CSV or JSON
format, even though they are hard to query and update incrementally?
The answer is that **programmers are lazy**, and thus they tend to prefer the easiest solution they find. And in **Python**, a database isn't the simplest solution for storing a bunch of structured data. This is what **dataset** is going to change!
The answer is that **programmers are lazy**, and thus they tend to prefer the
easiest solution they find. And in **Python**, a database isn't the simplest
solution for storing a bunch of structured data. This is what **dataset** is
going to change!
**dataset** provides two key functions that make using SQL databases in
Python a breeze:
* A simple abstraction layer removes most direct SQL statements without
the necessity for a full ORM model - essentially, databases can be
used like a JSON file or NoSQL store.
* Database contents can be exported (*frozen*) using a :doc:`sophisticated
plain file generator <freezefile>` with JSON and CSV support. Exports can be configured
to include metadata and dynamic file names depending on the exported
data. The exporter can also be used as a command-line tool, ``datafreeze``.
**dataset** provides a simple abstraction layer removes most direct SQL
statements without the necessity for a full ORM model - essentially, databases
can be used like a JSON file or NoSQL store.
A simple data loading script using **dataset** might look like this:
@@ -55,8 +52,6 @@ Features
* **Query helpers** for simple queries such as :py:meth:`all <dataset.Table.all>` rows in a table or
all :py:meth:`distinct <dataset.Table.distinct>` values across a set of columns.
* **Compatibility**: Being built on top of `SQLAlchemy <http://www.sqlalchemy.org/>`_, ``dataset`` works with all major databases, such as SQLite, PostgreSQL and MySQL.
* **Scripted exports**: Data can be exported based on a scripted
configuration, making the process easy and replicable.
Contents
--------
@@ -66,12 +61,14 @@ Contents
install
quickstart
freezefile
api
Contributors
------------
``dataset`` is written and maintained by `Friedrich Lindenberg <https://github.com/pudo>`_, `Gregor Aisch <https://github.com/gka>`_ and `Stefan Wehrmeyer <https://github.com/stefanw>`_. Its code is largely based on the preceding libraries `sqlaload <https://github.com/okfn/sqlaload>`_ and datafreeze. And of course, we're standing on the `shoulders of giants <http://www.sqlalchemy.org/>`_.
``dataset`` is written and maintained by `Friedrich Lindenberg <https://github.com/pudo>`_,
`Gregor Aisch <https://github.com/gka>`_ and `Stefan Wehrmeyer <https://github.com/stefanw>`_.
Its code is largely based on the preceding libraries `sqlaload <https://github.com/okfn/sqlaload>`_
and datafreeze. And of course, we're standing on the `shoulders of giants <http://www.sqlalchemy.org/>`_.
Our cute little `naked mole rat <http://www.youtube.com/watch?feature=player_detailpage&v=A5DcOEzW1wA#t=14s>`_ was drawn by `Johannes Koch <http://chechuchape.com/>`_.
docs/install.rst
+5 -2
View File
@@ -2,7 +2,8 @@
Installation Guide
==================
The easiest way is to install ``dataset`` from the `Python Package Index <https://pypi.python.org/pypi/dataset/>`_ using ``pip`` or ``easy_install``:
The easiest way is to install ``dataset`` from the `Python Package Index
<https://pypi.python.org/pypi/dataset/>`_ using ``pip`` or ``easy_install``:
.. code-block:: bash
@@ -16,4 +17,6 @@ To install it manually simply download the repository from Github:
$ cd dataset/
$ python setup.py install
Depending on the type of database backend, you may also need to install a database specific driver package. For MySQL, this is ``MySQLdb``, for Postgres its ``psycopg2``. SQLite support is integrated into Python.
Depending on the type of database backend, you may also need to install a
database specific driver package. For MySQL, this is ``MySQLdb``, for Postgres
its ``psycopg2``. SQLite support is integrated into Python.
docs/quickstart.rst
+6 -32
View File
@@ -30,8 +30,8 @@ so you can initialize database connection without explicitly passing an `URL`::
Depending on which database you're using, you may also have to install
the database bindings to support that database. SQLite is included in
the Python core, but PostgreSQL requires ``psycopg2`` to be installed.
MySQL can be enabled by installing the ``mysql-db`` drivers.
the Python core, but PostgreSQL requires ``psycopg2`` to be installed.
MySQL can be enabled by installing the ``mysql-db`` drivers.
Storing data
@@ -110,7 +110,7 @@ database:
Now, let's list all columns available in the table ``user``:
>>> print(db['user'].columns)
[u'id', u'country', u'age', u'name', u'gender']
[u'id', u'country', u'age', u'name', u'gender']
Using ``len()`` we can get the total number of rows in a table:
@@ -156,7 +156,7 @@ results will be returned::
db = dataset.connect('sqlite:///mydatabase.db', row_type=stuf)
Now contents will be returned in ``stuf`` objects (basically, ``dict``
objects whose elements can be acessed as attributes (``item.name``) as well as
objects whose elements can be acessed as attributes (``item.name``) as well as
by index (``item['name']``).
Running custom SQL queries
@@ -169,36 +169,10 @@ use the full power of SQL queries. Here's how you run them with ``dataset``::
for row in result:
print(row['country'], row['c'])
The :py:meth:`query() <dataset.Table.query>` method can also be used to
The :py:meth:`query() <dataset.Table.query>` method can also be used to
access the underlying `SQLAlchemy core API <http://docs.sqlalchemy.org/en/latest/orm/query.html#the-query-object>`_, which allows for the
programmatic construction of more complex queries::
table = db['user'].table
statement = table.select(table.c.name.like('%John%'))
result = db.query(statement)
Exporting data
--------------
While playing around with our database in Python is a nice thing, they are
sometimes just a processing stage until we go on to use it in another
place, say in an interactive web application. To make this seamless,
``dataset`` supports serializing rows of data into static JSON and CSV files
such using the :py:meth:`freeze() <dataset.freeze>` function::
# export all users into a single JSON
result = db['users'].all()
dataset.freeze(result, format='json', filename='users.json')
You can create one file per row by setting ``mode`` to "item"::
# export one JSON file per user
dataset.freeze(result, format='json', filename='users/{{ id }}.json', mode='item')
Since this is a common operation we made it available via command line
utility ``datafreeze``. Read more about the :doc:`freezefile markup <freezefile>`.
.. code-block:: bash
$ datafreeze freezefile.yaml
result = db.query(statement)