Commit Graph
63 Commits
Author SHA1 Message Date
J. Nick Koston c2b770bcb9 Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
epenet 185cd61cbd Add type hints to integration tests (recorder) (#88313) 2023-02-17 18:50:09 +01:00
epenetandJ. Nick Koston efef915fc7 Add mariadb 10.3.32 to database tests (#87741)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-13 16:28:39 -06:00
J. Nick Koston abf0c87e40 Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
720f51657d Add CI job which runs recorder tests on PostgreSQL (#80614)
Co-authored-by: Franck Nijhof <git@frenck.dev>
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-07 09:46:14 -06:00
J. Nick Koston b8a1537b58 Improve performance of fetching and storing history and events with the database (#84870) 2023-01-02 13:26:08 -10:00
Erik MontnemeryandFranck Nijhof f4951a4f31 Add CI job which runs recorder tests on MariaDB (#80586)
Co-authored-by: Franck Nijhof <git@frenck.dev>
2022-10-19 13:04:28 +02:00
Erik Montnemery 31a787558f Ensure recorder test fixture is setup before hass fixture (#80528)
* Ensure recorder test fixture is setup before hass fixture

* Adjust more tests
2022-10-19 07:58:47 +02:00
Erik MontnemeryandJ. Nick Koston 5f2b4001f3 Separate recorder database schema from other classes (#72977)
* Separate recorder database schema from other classes

* fix logbook imports

* migrate new tests

* few more

* last one

* fix merge

Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
J. Nick Koston d8b0376942 Fix purge of legacy database events that are not state changed (#72815) 2022-06-01 11:56:06 +02:00
J. Nick Koston a70e2a33dc Fixing purging legacy rows and improve performance (#71916) 2022-05-15 21:25:07 -07:00
J. Nick Koston a8aa0e1cca Add Estimated Database Size to the recorder system health (#71463) 2022-05-07 21:02:54 -07:00
J. Nick Koston e30940ef2a Move processing of recorder service call arguments into services.py (#71260) 2022-05-03 13:56:22 -07:00
J. Nick Koston e1be6dd34f Move recorder services to services.py (#71249) 2022-05-03 10:38:44 -07:00
J. Nick Koston 29bda196b5 Break apart recorder into tasks and core modules (#71222) 2022-05-02 21:53:56 -07:00
J. Nick KostonandPaulus Schoutsen 5db014666c Avoid recording state_changed events in the events table (#71165)
* squash

fix mypy

* Update homeassistant/components/recorder/models.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* pass all columns

* fix commented out code

* reduce logbook query complexity

* merge

* comment

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-02 02:10:34 -05:00
J. Nick Koston 6ce768465f Improve purge performance for non-sqlite databases (#70576) 2022-04-26 13:11:57 -07:00
Erik Montnemery 28ebab9c5a Simplify waiting for recorder in tests (#70647) 2022-04-25 12:04:47 +02:00
Erik MontnemeryandPaulus Schoutsen 982e314de6 Use recorder_mock in tests (#70363)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2022-04-22 19:29:44 -10:00
J. Nick Koston 3142a106fc Additional prep work for sqlalchemy 2.0 (#70358) 2022-04-21 12:39:18 -10:00
Erik Montnemery eb3458a3d2 Add MutexPool for recorder tests (#69410)
* Add MutexPool for recorder tests

* Fix get_schema_version

* Update test test_last_run_was_recently_clean

* Update test test_shutdown_before_startup_finishes

* Revert comments in test_write_lock_db

* Make the MutexPool lock a class variable

* Remove stale comment

* Move MutexPool

* Tweak debug prints
2022-04-12 07:41:46 -10:00
J. Nick Koston 3150915cb7 Convert unindexed domain queries to entity_id queries (#68404) 2022-03-20 01:28:17 -10:00
J. Nick Koston 7ee647cc78 Fix FOREIGN KEY constraint failed when removing state_attributes (#68364) 2022-03-19 14:21:28 -07:00
J. Nick Koston 9215702388 Separate attrs into another table (reduces database size) (#68224) 2022-03-18 00:23:13 -10:00
J. Nick Koston b18096fc54 Remove unused columns from states/events tables (#68078) 2022-03-13 23:44:55 -07:00
Erik Montnemery b301ab25a3 Purge short term statistics (#58028)
* Purge short term statistics

* Less meep

* Add tests
2021-10-20 16:00:59 +02:00
Erik MontnemeryandJ. Nick Koston 8567aa9e13 Evict purged states from recorder's old_state cache (#56877)
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-10-04 08:21:40 -07:00
PeteBa 42c4317628 Avoid drift in recorder purge cut-off (#52135) 2021-06-25 11:29:38 -10:00
PeteBa aa9b99713c Add purge_entities service call to recorder (#48069) 2021-05-22 10:30:05 -05:00
Erik Montnemery 89dd3292ba Initial draft of statistics (#49852) 2021-05-16 10:23:37 -07:00
jan iversen c351098f04 HomeAssistantType —> HomeAssistant for Integrations p* - s* (#49558) 2021-04-22 19:58:02 +02:00
J. Nick KostonandMartin Hjelmare 6d137d2316 Increase recorder test coverage (#49362)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
J. Nick KostonandPaulus Schoutsen 9368891b1b Live db migrations and recovery (#49036)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
Franck NijhofandMartin Hjelmare 08870690a6 Fix a collection of tests with missing asserts (#48127)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-03-20 17:23:55 +01:00
Franck Nijhof fb849b81b5 Rewrite of not a == b occurances (#48132) 2021-03-20 01:27:04 +01:00
J. Nick Koston 28c80c1133 Ensure recorder purge tests can handle multiple purge cycle (#47956)
Since a purge can generate another purge task, we now wait for
three recorder queue completions by default.
2021-03-15 11:19:19 -07:00
J. Nick Koston 61a2460c87 Improve error reporting in recorder purge test (#47929) 2021-03-14 19:46:21 -07:00
Marc Mueller 92852b9c10 Add apply_filter attribute to recorder.purge service (#45826) 2021-03-11 17:03:30 -10:00
Marc Mueller 10848b9bdf Recorder improvements (#47739) 2021-03-11 07:52:07 -10:00
J. Nick KostonandMarc Mueller a060acc2b1 Fix recorder with MSSQL (#46678)
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
2021-03-09 11:16:19 -10:00
Franck Nijhof 65cf2fcb6f Drop asynctest (#44746) 2021-01-01 22:31:56 +01:00
J. Nick Koston 2cbb93be43 Always keep the current recorder run when purging (#43733) 2020-11-30 09:13:50 +01:00
Ariana Hlavaty 1e9e40bf71 Rewrite recorder unittest tests to pytest style test function (#41264) 2020-10-06 21:24:13 +02:00
J. Nick Koston 7878d97588 Use SimpleQueue for recorder (#38967)
Now that python 3.7 is the minimum supported version, we can
use the more efficient SimpleQueue in the recorder as it does
not have to use threading.Lock
2020-08-21 13:54:13 +02:00
Franck Nijhof 79f131066c Ensure recorder data integrity and MySQL lock error handling (#37228) 2020-06-29 16:23:11 -07:00
Franck Nijhof 7d74b74570 Fix recorder purging by batch processing purges (#37140) 2020-06-26 10:27:45 -07:00
J. Nick KostonandPaulus Schoutsen ad6315be5c Ensure recorder runs are cleaned up during purge (#36989)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2020-06-22 20:10:05 -07:00
Paulus Schoutsen c33edbe5bb Use TestCase.addCleanup (#36560) 2020-06-08 12:26:40 -07:00
Paulus Schoutsen 2af984917e Use asynctest-mock in most places (#35109)
* Use asynctest-mock in most places

* Fix broken patch in pilight
2020-05-03 11:27:19 -07:00
Bas Nijholt 3d64405896 Sort imports according to PEP8 for recorder (#29652) 2019-12-08 18:48:18 +01:00