Commit Graph
63 Commits
Author SHA1 Message Date
J. Nick Koston b9ac6b4a7c Improve reliability of context id migration (#89609)
* Split context id migration into states and events tasks

Since events can finish much earlier than states we
would keep looking at the table because states as not
done. Make them seperate tasks

* add retry dec

* fix migration happening twice

* another case
2023-03-12 21:41:48 -04:00
J. Nick Koston c41f91be89 Deduplicate entity_id in the states table (#89557) 2023-03-12 10:01:58 -10:00
J. Nick Koston 8bd43760b6 Deduplicate event_types in the events table (#89465)
* Deduplicate event_types in the events table

* Deduplicate event_types in the events table

* more fixes

* adjust

* adjust

* fix product

* fix tests

* adjust

* migrate

* migrate

* migrate

* more test fixes

* more test fixes

* fix

* migration test

* adjust

* speed up

* fix index

* fix more tests

* handle db failure

* preload

* tweak

* adjust

* fix stale docs strings, remove dead code

* refactor

* fix slow tests

* coverage

* self join to resolve query performance

* fix typo

* no need for quiet

* no need to drop index already dropped

* remove index that will never be used

* drop index sooner as we no longer use it

* Revert "remove index that will never be used"

This reverts commit 461aad2c52d7d6c2c6ca1df4cb30b77c52027d46.

* typo
2023-03-11 14:54:55 -05:00
J. Nick Koston 170a13302c Reduce overhead to store context ids in the database (#88942) 2023-03-08 14:51:45 -10:00
J. Nick Koston 1bd9767d8c Handle InnoDB deadlocks during migration (#89073)
* Handle slow InnoDB rollback when encountering duplicates during migration

fixes #89069

* adjust

* fix mock

* tests

* return on success
2023-03-03 22:00:13 -05:00
J. Nick Koston c2b770bcb9 Load pending state attributes and event data ids at startup (#88444)
* Load pending state attributes and event data ids at startup

Since we queue all events to be processed after startup
we can have a thundering herd of queries to prime the
LRUs of event data and state attributes ids. Since we
know we are about to process a chunk of events we can
fetch all the ids in two queries

* lru

* fix hang

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* Fix recorder LRU being destroyed if event session is reopened

We would clear the LRU in _close_event_session but
it would never get replaced with an LRU again so
it would leak memory if the event session is reopened

* cleanup
2023-02-19 21:26:38 -05:00
epenet 185cd61cbd Add type hints to integration tests (recorder) (#88313) 2023-02-17 18:50:09 +01:00
Franck Nijhof ed79265843 Enable Ruff PT006 (#88165)
* Enable Ruff PT006

* Adjust existing cases

* Fix tests

* Remove unneeded parentheses
2023-02-15 14:09:50 +01:00
cbaf4764e7 Ensure MySQL tests cleanup connections and raise an exception if they do not (#87767)
Co-authored-by: epenet <6771947+epenet@users.noreply.github.com>
Co-authored-by: Erik <erik@montnemery.com>
2023-02-09 16:27:53 -06:00
J. Nick Koston abf0c87e40 Migrate statistics to use timestamp columns (#87321) 2023-02-09 12:24:19 -06:00
Erik Montnemery 2fae82a9ef Really ensure recorder test fixture is setup before hass fixture (#87796) 2023-02-09 11:43:14 -06:00
J. Nick Koston f4a81a8b5b Disable test_migration_in_progress on MySQL (#87765)
This test was recently enabled on MySQL in
https://github.com/home-assistant/core/pull/87753

Since it the migration is still in progress in the background
when the test ends, it causes deadlock with InnoDB when its dropped
out from under it
2023-02-09 15:27:27 +01:00
Erik Montnemery b0f5bdd504 Improve migration tests to use MariaDB / PostgreSQL database (#87753) 2023-02-09 08:42:32 +01:00
epenet ba85fdcd61 Add type hints to integration tests (q-s) (#87706) 2023-02-08 08:12:54 -08:00
epenet 59ca7780fa Add typing to tests with single hass argument (#87631) 2023-02-07 15:01:16 +01:00
epenet 4142f0d15d Add return type to tests without arguments (#87613)
* Add return type to tests without arguments

* Black

* Cancel fixture amends
2023-02-07 14:20:06 +01:00
Franck Nijhof 5e81d28116 Update black to 23.1.0 (#87188) 2023-02-02 18:35:24 +01:00
epenet 30bf0634fe Add per-file-ignore to pylint (#86289) 2023-01-22 17:26:24 +01:00
Erik Montnemery 466c4656ca Refactor recorder migration (#80175)
* Refactor recorder migration

* Improve test coverage
2022-10-13 08:11:54 +02:00
Erik 4a1c40f09b Revert "Refactor recorder migration"
This reverts commit 69e10e5982.
2022-10-12 15:12:12 +02:00
Erik 69e10e5982 Refactor recorder migration 2022-10-12 15:00:28 +02:00
Erik Montnemery 2de273500e Remove state_unit_of_measurement from metadata DB table (#79370)
* Remove state_unit_of_measurement from metadata DB table

* Adjust test
2022-10-01 18:55:00 +02:00
Erik Montnemery dd20a7ea62 Display statistics in the source's unit (#78031) 2022-09-15 12:01:24 -04:00
Erik Montnemery 0a1fd36e03 Catch up with statistics after DB migration is done (#78469)
* Catch up with statistics after DB migration is done

* Don't access the database from the event loop

* Fix deadlocking test

* Fix test
2022-09-15 07:40:56 +02:00
Erik Montnemery fd6ffef52f Support non-live database migration (#72433)
* Support non-live database migration

* Tweak startup order, add test

* Address review comments

* Fix typo

* Clarify comment about promoting dependencies

* Tweak

* Fix merge mistake

* Fix some tests

* Fix additional test

* Fix additional test

* Adjust tests

* Improve test coverage
2022-07-22 15:11:34 +02:00
Erik Montnemery 606d544157 Use recorder get_instance function to improve typing (#75567) 2022-07-22 11:58:26 +02:00
Erik MontnemeryandJ. Nick Koston 5f2b4001f3 Separate recorder database schema from other classes (#72977)
* Separate recorder database schema from other classes

* fix logbook imports

* migrate new tests

* few more

* last one

* fix merge

Co-authored-by: J. Nick Koston <nick@koston.org>
2022-06-07 14:41:43 +02:00
J. Nick KostonandMartin Hjelmare eb77f8db85 Complete strict typing for recorder (#71274)
* Complete strict typing for recorder

* update tests

* Update tests/components/recorder/test_migrate.py

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>

* Update tests/components/recorder/test_migrate.py

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>

* Remove the asserts

* remove ignore comments

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-05-04 12:22:50 -05:00
J. Nick Koston 13ce0a7d6a Fix history using pre v25 queries during v26 migration (#71295) 2022-05-04 08:56:50 -07:00
J. Nick Koston 29bda196b5 Break apart recorder into tasks and core modules (#71222) 2022-05-02 21:53:56 -07:00
J. Nick Koston 1c4a785fb3 Prevent autoflush from happening during attrs lookup (#70768) 2022-04-26 13:04:58 -07:00
J. Nick KostonandErik Montnemery f073f17040 Refactor tracking of the recorder run history (#70456)
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2022-04-26 09:59:43 -10:00
J. Nick Koston 89e3a48751 Use database executor in migration test (#70774) 2022-04-26 07:42:28 -07:00
Erik Montnemery 28ebab9c5a Simplify waiting for recorder in tests (#70647) 2022-04-25 12:04:47 +02:00
J. Nick Koston 66f0a3816a Reduce memory pressure during database migration (#69628) 2022-04-07 18:29:31 -10:00
Erik Montnemery 949b0e1b65 Don't allow in-memory SQLite database (#69616) 2022-04-08 00:43:09 +02:00
J. Nick Koston e911936a0d Remove direct usage of concurrent.futures from recorder (#68593) 2022-03-24 09:48:49 -10:00
Erik Montnemery 41ab12cb88 Don't use shared session during recorder migration (#65672) 2022-02-04 09:55:11 -08:00
Erik Montnemery e5ee38d3b0 Adjust recorder tests according to #59087 (#59143) 2021-11-05 10:49:58 +01:00
Erik Montnemery 470b01e4ce Move async_migration_in_progress (#59087) 2021-11-05 10:40:56 +01:00
Erik Montnemery 185f7beafc Improve recorder migration tests (#59075) 2021-11-04 20:21:38 -07:00
Erik Montnemery 4c5aca93df Add recorder status WS API (#58989)
* Add recorder status WS API

* Rename recorder/status to recorder/info

* Silence pylint

* Improve tests

* Address review comments

* Tweak

* Try to fix tests

* Try to debug flaky tests

* Try to fix tests

* Revert changes to async_migration_in_progress

* Try to fix tests

* Remove debug prints

* Apply suggestions from code review
2021-11-04 16:46:45 +01:00
Paulus SchoutsenandMartin Hjelmare a4d9019ffc Refactor persistent notification to no longer route all data via a service (#57157)
* Convert persistent notification tests to async

* Create/dismiss persistent notifications in exposed functions, not service calls

* Fix notify persistent_notification

* Remove setting up persistent_notification

* Drop more setups

* Empty methods

* Undeprecate sync methods because too big task

* Fix setup clearing notifications

* Fix a bunch of tests

* Fix more tests

* Uno mas

* Test persistent notification events

* Clean up stale comment

Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2021-10-07 12:58:00 +02:00
Erik Montnemery 64393b462d Add migration for 5-minute statistics (#56585)
* Add migration for 5-minute statistics

* Tweaks
2021-09-24 09:19:22 +02:00
J. Nick Koston 1c8d9ca68b Check exception causes for matching strings during recorder migration (#49999) 2021-05-02 20:57:42 -07:00
J. Nick Koston c10836fcee Upgrade to sqlalchemy 1.4.11 (#49538) 2021-04-21 20:29:36 -10:00
J. Nick Koston 53853f035d Prevent calling stop or restart services during db upgrade (#49098) 2021-04-12 17:18:38 -07:00
J. Nick KostonandPaulus Schoutsen 9368891b1b Live db migrations and recovery (#49036)
Co-authored-by: Paulus Schoutsen <paulus@home-assistant.io>
2021-04-11 20:43:54 -10:00
Laszlo Magyar c7e4857d2c Let recorder deal with event names longer than 32 chars (#47748) 2021-04-08 07:08:49 -10:00
Erik Montnemery 2178e27fb4 Fix unclean shutdown of recorder test (#47791) 2021-03-12 14:17:27 -08:00