Commit Graph
63 Commits
Author SHA1 Message Date
J. Nick Koston c2b770bcb9 Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
Erik MontnemeryandJ. Nick Koston 94519de8dd Upgrade SQLAlchemy to 2.0.2 (#86436)
Co-authored-by: J. Nick Koston <nick@koston.org>
2023-02-08 08:17:32 -06:00
J. Nick Koston c11f3ffa17 Speed up purge time with newer MariaDB versions (#87409)
* Speed up purge time with newer MariaDB versions

* fix

* document

* document

* document

* rename

* self review

* Update homeassistant/components/recorder/util.py

* fixes
2023-02-05 17:58:34 -05:00
Franck Nijhof 5e81d28116 Update black to 23.1.0 (#87188) 2023-02-02 18:35:24 +01:00
Franck Nijhof c5dedb7a79 Code styling tweaks to the recorder integration (#86030) 2023-01-16 19:51:11 +01:00
J. Nick Koston b8a1537b58 Improve performance of fetching and storing history and events with the database (#84870) 2023-01-02 13:26:08 -10:00
Franck Nijhofandjjlawren 94755a5773 String formatting and max line length - Part 4 (#84445)
Co-authored-by: jjlawren <jjlawren@users.noreply.github.com>
2022-12-23 13:27:27 +01:00
Erik MontnemeryandJ. Nick Koston 5f2b4001f3 Separate recorder database schema from other classes (#72977)
* Separate recorder database schema from other classes

* fix logbook imports

* migrate new tests

* few more

* last one

* fix merge

Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
Marc Mueller 3c246b7800 Update mypy to 0.960 (#72481) 2022-05-25 11:42:14 -07:00
J. Nick Koston a70e2a33dc Fixing purging legacy rows and improve performance (#71916) 2022-05-15 21:25:07 -07:00
J. Nick Koston 896bf986eb Speed up nightly database purges with lambda_stmt (#71537) 2022-05-08 12:47:12 -07:00
J. Nick Koston a8aa0e1cca Add Estimated Database Size to the recorder system health (#71463) 2022-05-07 21:02:54 -07:00
J. Nick KostonandMartin Hjelmare eb77f8db85 Complete strict typing for recorder (#71274)
* Complete strict typing for recorder

* update tests

* Update tests/components/recorder/test_migrate.py

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>

* Update tests/components/recorder/test_migrate.py

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>

* Remove the asserts

* remove ignore comments

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-05-04 12:22:50 -05:00
J. Nick Koston 188040b8bb Use lambda_stmt for recorder queries and migrate them to queries module (#71219) 2022-05-02 15:17:21 -07:00
J. Nick KostonandPaulus Schoutsen 5db014666c Avoid recording state_changed events in the events table (#71165)
* squash

fix mypy

* Update homeassistant/components/recorder/models.py

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>

* pass all columns

* fix commented out code

* reduce logbook query complexity

* merge

* comment

Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2022-05-02 02:10:34 -05:00
J. Nick Koston c23866e5e5 De-duplicate event data into a new event_data table (#71135) 2022-05-01 21:01:17 -05:00
J. Nick Koston b9c7a89b47 Prevent sqlalchemy Transparent SQL Compilation Caching from filling up during purge (#71015) 2022-04-28 23:48:28 -07:00
J. Nick Koston 6ce768465f Improve purge performance for non-sqlite databases (#70576) 2022-04-26 13:11:57 -07:00
J. Nick KostonandErik Montnemery f073f17040 Refactor tracking of the recorder run history (#70456)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-26 09:59:43 -10:00
J. Nick Koston d75f577b88 Additional strict typing for recorder (#68860) 2022-03-30 09:20:44 -07:00
Marc Mueller 53110f8cb7 Update pylint to 2.13.2 (#68704) 2022-03-27 16:08:24 +02:00
Marc Mueller 911b159281 Cleanup after pylint update (#68657) 2022-03-26 00:34:12 +01:00
J. Nick Koston 7ee647cc78 Fix FOREIGN KEY constraint failed when removing state_attributes (#68364) 2022-03-19 14:21:28 -07:00
J. Nick Koston 9215702388 Separate attrs into another table (reduces database size) (#68224) 2022-03-18 00:23:13 -10:00
Marc Mueller cb736eaeaf Add type ignore error codes [recorder] (#66780) 2022-02-18 10:37:38 +01:00
Erik Montnemery b301ab25a3 Purge short term statistics (#58028)
* Purge short term statistics

* Less meep

* Add tests
2021-10-20 16:00:59 +02:00
Erik MontnemeryandJ. Nick Koston 8567aa9e13 Evict purged states from recorder's old_state cache (#56877)
Co-authored-by: J. Nick Koston <nick@koston.org>
2021-10-04 08:21:40 -07:00
Marc Mueller 565a9fea6b Import Callable from collections.abc (2) (#56776) 2021-09-29 14:06:51 +02:00
PeteBa 42c4317628 Avoid drift in recorder purge cut-off (#52135) 2021-06-25 11:29:38 -10:00
PeteBa aa9b99713c Add purge_entities service call to recorder (#48069) 2021-05-22 10:30:05 -05:00
Erik Montnemery 89dd3292ba Initial draft of statistics (#49852) 2021-05-16 10:23:37 -07:00
J. Nick KostonandMartin Hjelmare 6d137d2316 Increase recorder test coverage (#49362)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-04-19 08:22:38 -07:00
J. Nick KostonandPaulus Schoutsen 9368891b1b Live db migrations and recovery (#49036)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
Marc Mueller 92852b9c10 Add apply_filter attribute to recorder.purge service (#45826) 2021-03-11 17:03:30 -10:00
Marc Mueller 10848b9bdf Recorder improvements (#47739) 2021-03-11 07:52:07 -10:00
J. Nick KostonandMarc Mueller a060acc2b1 Fix recorder with MSSQL (#46678)
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
2021-03-09 11:16:19 -10:00
J. Nick Koston 2cbb93be43 Always keep the current recorder run when purging (#43733) 2020-11-30 09:13:50 +01:00
Franck Nijhof 1c2ebdf307 Upgrade black to 20.8b1 (#39287) 2020-08-27 13:56:20 +02:00
Franck Nijhof 53545c984b Log lines do not end with a full stop (#37527) 2020-07-05 23:04:19 +02:00
Franck Nijhof 79f131066c Ensure recorder data integrity and MySQL lock error handling (#37228) 2020-06-29 16:23:11 -07:00
Franck Nijhof 7d74b74570 Fix recorder purging by batch processing purges (#37140) 2020-06-26 10:27:45 -07:00
J. Nick Koston a4501b93c4 Fix repack when using pymysql (#37142) 2020-06-26 11:45:40 -05:00
J. Nick KostonandPaulus Schoutsen ad6315be5c Ensure recorder runs are cleaned up during purge (#36989)
Co-authored-by: Paulus Schoutsen <balloob@gmail.com>
2020-06-22 20:10:05 -07:00
Alex van den HoogenandPaulus Schoutsen ee816ed3dd Optimize recorder MySQL tables when repacking (#36762)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2020-06-22 17:28:03 -07:00
Franck Nijhof 187b6525b4 Removal of extraneous parenthesis (#33637) 2020-04-04 20:08:55 +02:00
Bas Nijholt 3d64405896 Sort imports according to PEP8 for recorder (#29652) 2019-12-08 18:48:18 +01:00
bastshoes 0226b76e0a Add support SQL VACUUM for PostgeSQL (#28106)
* Add support SQL VACUUM for PostgeSQL

VACUUM PostgreSQL DB if repack is true

* Update tests
2019-10-22 08:39:26 -07:00
Malte Franken 83a709b768 Move imports in recorder component (#27859)
* move imports to top-level in recorder init

* move imports to top-level in recorder migration

* move imports to top-level in recorder models

* move imports to top-level in recorder purge

* move imports to top-level in recorder util

* fix pylint
2019-10-18 10:14:54 -07:00
Paulus Schoutsen 4de97abc3a Black 2019-07-31 12:25:30 -07:00
Anders Melchiorsen e593383b4d Error handling for recorder purge (#20424) 2019-01-26 11:02:16 +01:00