Commit Graph
334 Commits
Author SHA1 Message Date
J. Nick Koston c2b770bcb9 Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
J. Nick Koston 3533e77ec9 Fix using MariaDB slow range select workaround with Postgresql (#88459)
* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253

* Mark PostgreSQL range select as fast

Currently we were using the slow range select workaround for
PostgreSQL that was original developed for MariaDB but
its actually slower on PostgreSQ

fixes #83253
2023-02-19 21:08:43 -05:00
J. Nick Koston a9731a7b26 Make the recorder LRU adjustment threadsafe (#88443) 2023-02-19 19:30:08 +01:00
J. Nick Koston 9df00bd285 Adjust recorder LRU cache to align with the number of entities (#88350)
* Adjust size of recorder LRU based on number of entities

If there are a large number of entities the cache would
get thrashed as there were more state attributes being
recorded than the size of the cache. This meant we had
to go back to the database to do lookups frequently when
an instance has more than 2048 entities that change
frequently

* add a test

* do not actually record 4096 states

* patch target

* patch target
2023-02-19 00:02:36 -05:00
J. Nick Koston 3ca9f3c0d5 Only build the recorder SupportedDialect enum once (#88402)
Every call to dialect_name was creating a new enum object
in the recorder
2023-02-18 20:21:14 -05:00
Erik Montnemery 83e5bf7ae8 Use entity_sources to determine integration in recorder platforms (#88382) 2023-02-18 07:21:41 -06:00
J. Nick Koston 289bab6f87 Make recorder platform attribute exclude integration aware (#88357) 2023-02-18 10:08:59 +01:00
epenet a16e298599 Adjust json imports in tests (#88248) 2023-02-17 19:13:48 +01:00
epenet 185cd61cbd Add type hints to integration tests (recorder) (#88313) 2023-02-17 18:50:09 +01:00
J. Nick Koston b9f29c27ab Cleanup stale recorder code (#88275)
- Removes dead commit function
- Small cleanup to debug handling in execute
2023-02-16 23:12:11 -05:00
Franck Nijhof d3ed8de232 Update Ruff to v0.0.247 (#88210)
* Update Ruff to v0.0.247

* Fix new found violations
2023-02-15 16:46:03 -05:00
Franck Nijhof ed79265843 Enable Ruff PT006 (#88165)
* Enable Ruff PT006

* Adjust existing cases

* Fix tests

* Remove unneeded parentheses
2023-02-15 14:09:50 +01:00
epenetandJ. Nick Koston efef915fc7 Add mariadb 10.3.32 to database tests (#87741)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-13 16:28:39 -06:00
Erik MontnemeryandJ. Nick Koston 4ca0a24f87 Increase test scope of MariaDB + PostgreSQL tests (#87019)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-13 15:58:13 -06:00
J. Nick Koston 7b9293941e The recorder/statistic(s)_during_period websocket API now require the statistic_id(s) field (#87937) 2023-02-12 14:51:10 -06:00
J. Nick Koston c35661947a Optimize fetching statistics by avoiding timestamp conversions (#87845)
* Optimize fetching statistics by avoiding timestamp conversions

* remove accidential unrelated change

* fix test

* recreate so we handle timezone changes
2023-02-10 11:26:46 -05:00
cbaf4764e7 Ensure MySQL tests cleanup connections and raise an exception if they do not (#87767)
Co-authored-by: epenet <6771947+epenet@users.noreply.github.com>
Co-authored-by: Erik <erik@montnemery.com>
2023-02-09 16:27:53 -06:00
J. Nick Koston 509de02044 Ensure recorder still shuts down if the final commit fails (#87799) 2023-02-09 15:12:40 -06:00
J. Nick Koston abf0c87e40 Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
Erik Montnemery 2fae82a9ef Really ensure recorder test fixture is setup before hass fixture (#87796) 2023-02-09 11:43:14 -06:00
J. Nick Koston db503c0daa Always use UTC time for the MariaDB/MySQL session (#87583) 2023-02-09 11:39:45 -06:00
J. Nick Koston c05a7b29e6 Terminate stale MySQL connections at the end of test runs (#87794) 2023-02-09 11:13:13 -06:00
J. Nick Koston 03b47f1931 Disable test_entity_ids on MariaDB (#87766)
Dropping the database after this test will fail on
MySQL and hang forever because it causes an InnoDB deadlock

```
| 2042 | root   | localhost:52698 | NULL               | Query   |   41 | Waiting for table metadata lock | DROP DATABASE `homeassistant-test` |    0.000 |
```
2023-02-09 15:30:35 +01:00
J. Nick Koston f4a81a8b5b Disable test_migration_in_progress on MySQL (#87765)
This test was recently enabled on MySQL in
https://github.com/home-assistant/core/pull/87753

Since it the migration is still in progress in the background
when the test ends, it causes deadlock with InnoDB when its dropped
out from under it
2023-02-09 15:27:27 +01:00
J. Nick Koston 483b0cd017 Skip db locking test with postgresql/MySQL (#87760)
* Skip db locking test with postgresql/MySQL

https://github.com/home-assistant/core/pull/87756#issuecomment-1423828389

* switch to the guards instead
2023-02-09 13:58:32 +01:00
J. Nick Koston 77cdde94f3 Turn back off test_shutdown_before_startup_finishes test on MySQL (#87761) 2023-02-09 03:12:10 -06:00
Erik Montnemery 58618b2978 Improve recorder tests to use MariaDB / PostgreSQL (#87756) 2023-02-09 02:18:20 -06:00
Erik Montnemery b0f5bdd504 Improve migration tests to use MariaDB / PostgreSQL database (#87753) 2023-02-09 08:42:32 +01:00
epenet ba85fdcd61 Add type hints to integration tests (q-s) (#87706) 2023-02-08 08:12:54 -08:00
Erik MontnemeryandJ. Nick Koston 94519de8dd Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
720f51657d Add CI job which runs recorder tests on PostgreSQL (#80614)
Co-authored-by: Franck Nijhof <git@frenck.dev>
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-07 09:46:14 -06:00
epenet 59ca7780fa Add typing to tests with single hass argument (#87631) 2023-02-07 15:01:16 +01:00
J. Nick Koston 0a1d5c85ad Optimize history.get_last_state_changes query (#87554)
fixes undefined
2023-02-07 07:50:44 -06:00
epenet 4142f0d15d Add return type to tests without arguments (#87613)
* Add return type to tests without arguments

* Black

* Cancel fixture amends
2023-02-07 14:20:06 +01:00
epenet aa00114c2f Add return type to async tests without arguments (#87612) 2023-02-07 10:26:56 +01:00
Marc Mueller 342b406dc0 Add Self typing (1) [mypy 1.0] (#87598) 2023-02-06 22:29:47 -06:00
J. Nick Koston c7980bda82 Fix recorder run history during schema migration and startup (#87492)
Fix recorder run history during schema migration

RunHistory.get and RunHistory.current can be called before
RunHistory.start. We need to return a RecorderRuns object
with the recording_start time that will be used when start
it called to ensure history queries still work as expected.

fixes #87112
2023-02-05 22:12:41 -05:00
J. Nick Koston c11f3ffa17 Speed up purge time with newer MariaDB versions (#87409)
* Speed up purge time with newer MariaDB versions

* fix

* document

* document

* document

* rename

* self review

* Update homeassistant/components/recorder/util.py

* fixes
2023-02-05 17:58:34 -05:00
epenet 19e27f4f52 Ensure recorder api aligns with sensor converters (#87257) 2023-02-04 14:28:47 -06:00
J. Nick Koston 389fc515a1 Prevent overly large attributes from being stored in the database (#87105) 2023-02-02 11:49:33 -06:00
Franck Nijhof 5e81d28116 Update black to 23.1.0 (#87188) 2023-02-02 18:35:24 +01:00
epenet 58596d205c Ensure sensor and recorder converters align (#87138) 2023-02-02 10:36:03 +01:00
J. Nick Koston 1fd58b6cb7 Fix a few cartesian products in recorder tests (#87106)
Fixes
```
recorder/test_init.py:251: SAWarning: SELECT statement has a cartesian product between FROM element(s) "states" and FROM element "state_attributes".  Apply join condition(s) between each element to resolve.
```
2023-02-02 07:47:09 +01:00
J. Nick KostonandFranck Nijhof fb55933b0a Reduce chance of queue overflow during schema migration (#87090)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2023-02-01 18:34:40 +01:00
J. Nick Koston 5284837c8f Add a repair issue when using MariaDB is affected by MDEV-25020 (#87040)
closes https://github.com/home-assistant/core/issues/83787
2023-01-31 20:42:07 +01:00
Erik Montnemery 53c5f02ca2 Remove some dead code from recorder (#86697) 2023-01-30 18:37:48 +01:00
J. Nick Koston c612a92cfb Use python defaults for comparing State, LazyState, and Event objects (#86856)
* Speed up comparing State and Event objects

Use default python implementation for State and Event __hash__ and __eq__

The default implementation compared based on the id() of the object
which is effectively what we want here anyways. These overrides are
left over from the days when these used to be attrs objects

By avoiding implementing these ourselves all of the equality checks
can happen in native code

* tweak

* adjust tests

* write out some more

* fix test to not compare objects

* more test fixes

* more test fixes

* correct stats tests

* fix more tests

* fix more tests

* update sensor recorder tests
2023-01-29 13:31:43 -05:00
Franck Nijhof a79885ceaf Enable Ruff SIM117 (#86783) 2023-01-27 11:52:49 +01:00
Franck NijhofandMartin Hjelmare e4a78420b8 Enable Ruff PT013 (#86757)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2023-01-27 09:09:46 +01:00
Erik Montnemery fea30c1ce9 Terminate strings at NUL when recording states and events (#86687) 2023-01-26 11:11:03 +01:00