retoor/dataset
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
716 Commits 2 Branches 0 Tags
master
Commit Graph
134 Commits
Author SHA1 Message Date
retoor 73c4c0f4bc refactor: simplify database and table interactions 
feat: update sqlalchemy version to 2.0.0
refactor: improve table insert and update methods
refactor: streamline table select and distinct queries
refactor: simplify table delete and schema creation
feat: test: update tests to reflect api changes
 2025-11-09 17:02:19 +01:00
Simon Wörpel d61d309637 Respect _limit and _offset kwargs for select distinct 2023-08-08 15:42:38 +02:00
Friedrich Lindenberg b1638a41d6 Switch from nose to pytest 2021-10-30 12:07:06 +02:00
Friedrich Lindenberg 02d3d0333f Support disabling auto-increment on primary keys, fixes #351. 2020-12-29 20:03:45 +01:00
Friedrich Lindenberg 3fe0ae10aa Seems it was that lock. 2020-08-23 12:23:00 +02:00
Friedrich Lindenberg 28109d6826 Minor fixes 2020-08-23 12:10:10 +02:00
Friedrich Lindenberg a17f2c8d5c Apply black 2020-08-02 12:52:11 +02:00
Friedrich Lindenberg 6469ad5c42 Use default DB more 2020-06-29 08:33:41 +02:00
Ben Fasoli aeaab50043 Adds support to serialize dict as JSON 
There are likely some vendor-specific enhancements (such as `JSONB` in PostgreSQL) but the current method of type guessing doesn't know anything about the underlying database.
 2020-03-25 19:35:36 -07:00
Micha Gorelick ff98cfc5d4 Added ChunkedUpdate to leverage update_many 
The ChunkedInsert object is great for speeding up inserts. This commit
does the same for updates. It chunks up updates then sends them to the
table's `update_many` in bundles that contain updates to the same
fields.

In doing this the ChunkedInsert and new ChunkedUpdate were refactored to
inherit from a common `_Chunker` object that does most of the chunking
logic. This should also make it simple to add a ChunkedUpsert object.
 2020-03-13 19:00:54 +01:00
Micha Gorelick 03b7532997 fix flake8 errors 2020-03-13 15:05:07 +01:00
Micha Gorelick a9a15966e2 Add optional callback to ChunkedInsert 
This commit adds an optional `callback` argument to the `ChunkedInsert`
object. This callback is a callable object which gets called before the
chunked insert happens. This is useful for clearing any local caches
that may be in place to deal with the eventual consistency resulting
from the delayed nature of the chunked inserts.

For example,

```
cache = set()
chunked_table = ChunkedInsert(table, callback=lambda queue: cache.clear())
while True:
    data = get_data_id()
    key = data['key']
    if key in cache or table.find_one(key=key)
        continue
    cache.add(key)
    chunked_table.insert(data)
```
 2020-03-13 14:29:03 +01:00
Friedrich Lindenberg 72ecf561fe Make column lookups properly case-insensitive, refs #310. 2020-02-23 19:13:34 +01:00
Abdurrahmaan Iqbal d8cf515ffe Implement additional find operators (and relevant tests) 2019-08-08 12:55:29 +01:00
Friedrich Lindenberg 43e9431865 Merge pull request #296 from conorreid/bigint_check 
Add support for BigInteger
 2019-07-14 17:58:43 +02:00
conorreid 8409c40645 switch all ints to bigints 2019-07-13 10:42:20 -04:00
Friedrich Lindenberg 081cb5ec7a Merge pull request #298 from abmyii/master 
Implement update_many, upsert_many and refactor for a 2x speed-up of insert_many
 2019-07-13 14:47:04 +02:00
Friedrich Lindenberg 6baff6fa34 Merge pull request #284 from al42and/upsert_id 
Fix bug when UPSERTing a column named 'id'
 2019-07-13 14:28:32 +02:00
Abdurrahmaan Iqbal 82c6cdc990 Add whitespace around arithmetic operator so flake8 test passes 2019-07-09 09:41:21 +01:00
Abdurrahmaan Iqbal a9f3eb86b2 Add tests for new functions 2019-07-08 18:10:00 +01:00
conorreid 96f57f161a added tests for big int vs integer check 2019-06-21 11:49:01 -04:00
Stefan Wehrmeyer a008d120c7 Fix WS 2019-03-09 11:39:25 +01:00
Stefan Wehrmeyer 046f3434ff Remove polyfill for Python < 2.7 2019-03-09 11:39:00 +01:00
Andrey Alekseenko 25477717bc Fix bug when UPSERTing a column named 'id' 2019-02-14 22:35:47 -05:00
Friedrich Lindenberg 8324350f2e Try to fix up tests, Python 3.8 warnings 2019-01-31 14:56:19 +01:00
Simon Conseil c69827de7c Add support for advanced queries in distinct 
Fix #263
 2018-09-26 17:36:09 +02:00
Unknown f9bb652034 add find operators 2018-06-13 02:29:32 +02:00
Friedrich Lindenberg 2cf9da068a fix flake8 complaints 2017-12-05 17:21:08 +01:00
Friedrich Lindenberg 8226e51408 option for streamed responses. 2017-12-04 23:22:08 +01:00
Stefan Wehrmeyer e9aa9298f0 Drop unicode prefix 2017-09-29 14:10:17 +02:00
Friedrich Lindenberg b8598b230a Rename test, refs #217. 2017-09-09 18:37:37 +02:00
Friedrich Lindenberg fc59bc59e6 Fix up imports, refs #217. 2017-09-09 18:34:32 +02:00
Friedrich Lindenberg a049691749 Remove datafreeze component, fixes #217 2017-09-09 18:24:34 +02:00
Friedrich Lindenberg 4232606d27 Move to a model where the table is created lazily, with an initial set of columns 2017-09-03 23:23:57 +02:00
Friedrich Lindenberg e30cf24195 Rewrite data change functions on table. 2017-09-03 10:05:17 +02:00
Friedrich Lindenberg 37d7f47d39 Make table instances singleton-ish. 2017-09-02 22:35:29 +02:00
Friedrich Lindenberg bf17deeb7f forgot a print statement 2017-09-02 20:40:52 +02:00
Friedrich Lindenberg 13cbff37fe Fix postgres errors. 2017-09-02 20:33:11 +02:00
Friedrich Lindenberg a2748b7fde Reduce dependence on internal metadata caching, refs #208. 2017-09-02 19:35:01 +02:00
Friedrich Lindenberg 672b0bc8c4 test fixes 2017-09-02 17:31:05 +02:00
Friedrich Lindenberg 47cdf52323 fix mysql issue, test no-column table. 2017-09-02 17:25:52 +02:00
Friedrich Lindenberg cc7787036b still more cases of using the engine and not the executable in transaction 2017-09-02 17:17:24 +02:00
Friedrich Lindenberg a4c73a8fb8 Begin implementing a types handler instead of using plain text types. 
this is potentially BREAKING scripts which use the string syntax.
 2017-09-02 16:47:04 +02:00
Friedrich Lindenberg edc41e4d82 Merge branch 'master' into create_column_by_example 2017-09-02 08:22:22 +02:00
Stefan Wehrmeyer ffe7beae12 Make test read nicer 2017-05-16 16:59:18 +02:00
Stefan Wehrmeyer 3f330df20f Check for clauses and filters in delete 
Guard against wrong API use before dropping all data
 2017-05-16 16:57:07 +02:00
Paul Fitzpatrick 94bc6e09c0 add Table.create_column_by_example method 
This adds a method to create a column by giving an example of
the data it will contain rather than by specifying its
SQLAlchemy type.  This extends the range of operations that
can be performed without reading the SQLAlchemy docs.
 2016-10-26 09:29:55 -04:00
Stefan Wehrmeyer 6fc8bfed51 Skip invalid values test on mysql 2016-10-17 15:16:13 +02:00
Stefan Wehrmeyer a92a315b60 Test behaviour of _step, assert it still retrieves all data 2016-10-15 22:55:50 +02:00
Saun Shewanown c1e82bb9ae Added insert_ignore and changed upsert. 2016-04-22 11:35:14 -04:00