* Unified Storage: First iteration Dual Write Syncer
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
Co-authored-by: Leonor Oliveira <9090754+leonorfmartins@users.noreply.github.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
* Itroduce watcher and mode4
* Logging
* Mode4 should be initialized from the dual writer for observability
* Comment watch while it's not implemented
* Lint
* Use mode log when dual writer is initiated
* Use error from logger
* Compare results when reading/writing between unified_storage and legacy
* Always use name when comparing objects
* Compare on get method
* Update pkg/apiserver/rest/dualwriter.go
Co-authored-by: Dan Cech <dcech@grafana.com>
* Add new metric to count how many times we read from legacy in mode 2
* Move counter
* Add name in mode1
---------
Co-authored-by: Dan Cech <dcech@grafana.com>
* WIP implement generic compare interface
* Use global compare fn for all entities
* Lint
* Update pkg/apiserver/rest/dualwriter.go
Co-authored-by: Dan Cech <dcech@grafana.com>
* Don't need to hash, just compare bytes
* Fix tests
---------
Co-authored-by: Dan Cech <dcech@grafana.com>
* Pass prometheus registerer to the dual writer
* Fix tests
* Remove unused var
* Fix tests
* Uncomment test
* Remove leading line
* Fix tests. Reuse registerer if there's already one
* Lint
* Improve double registering logic
* Rebase main
* Also call storage on mode1. Add metrics
* Update comment
* Don't use compare function for now
* Remove very important space
* Improve code readability on modes 1 and 2
* Fix tests
* Return error from legacy write
* Renume useless defer
* Start adding logging for mode2
* Retrurn errors is storage read fails
* Finish adding latency metrics to mode2
* Emit kind to metrics instead
* Fix var name
* Return if legacy delete list fails
* Unify latency calls
* Fix get function
* Extract returned object hydration into function
* Finish writing tests for utils func
* Lint
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Better var naming
* Remove duplicated logic
* Lint
* Fix test
* Lint
* Make type private
* Fix one more test
* Fix test
---------
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Add skeleton implementation for mode reconciliation between 1 and 2
* Track mode for each dual writer
* Add test for setting dual writer
* Include context when setting dual writing mode
---------
Co-authored-by: Dan Cech <dcech@grafana.com>
* Also call storage on mode1. Add metrics
* Update comment
* Don't use compare function for now
* Remove very important space
* Finish add logging in mode2.
Also call US in mode1 in a non blocking way
* Improve code readability on modes 1 and 2
* Fix tests
* Rename vars
* Lint
* Return error from legacy write
* Renume useless defer
* [REVIEW] improvements
* Pass kind instead of name
* Use kind instead of name in metrics
* Only call latency metrics once
* Return error on writes to legacystore in mode1
* Move accesssor logic into the goroutine as well
* Choose mode. Add log field on top level dualwriter
* Add logs
* Turn DualWriter into a full interface. Fix tests
* Lint
* Use struct for dualWriter interface
* Use struct
* Default should be legacyStore for all entities
* Fix test. Get rid of extra concrete type
* Remove comment
* Add comment
* Temp set dualwriter mode 2 for playlists while configs are not in place
* Add modes type + add comment on what each mode does
* Don't require watcher interface for now
* Use storage implementation on mode 2
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
* Pass log values to the context
* test
* Update pkg/apiserver/rest/dualwriter_mode3.go
Co-authored-by: Dan Cech <dcech@grafana.com>
---------
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
* Fix mode 2 List test
* Set origin timestamp during conversion to k8s resource
* Add instructions for updating a playlist
* Handle partial deletions of a collection in mode 2
* Add origin keys to List request
* Check origin keys requirement in the sql entity server's List method
* Refactor mode 2 List
* Check origin keys before making Storage List call
---------
Co-authored-by: Dan Cech <dcech@grafana.com>
* Add update methods for the dual writer
* improve errors
* [WIP] add tests for the update method
* Move example package to its own package so it can be used by the rest package. Finish tests
* Add codeowners
* Use Pod as a dummy resource
* :int
* Lint
* [REVIEW] rename var
* [REVIEW] don't rely on legacy storage at all in mode4
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* [REVIEW] improve comment
* Update pkg/apiserver/rest/dualwriter_mode1.go
Co-authored-by: Dan Cech <dcech@grafana.com>
* [REVIEW] improve mode3
* Lint
* Move test files
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Lint
* Update pkg/apiserver/rest/dualwriter_mode4_test.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Fix error
* Lint
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Don't set the flag to true as updatedObj creates an object in case it's not found
* Lint
* Lint
* Add tests on update
* Lint
---------
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
* WIP: add mocks for dual writer
* Test dualwriter mode1
* Re-add non implementation errors
* Use testify assert
* Write tests for mode2.
* Lint
* Remove comment
* Update pkg/apiserver/rest/dualwriter_mode2_test.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* [REVIEW improve readability and call fn legacy fn
* [REVIEW] rename mocks and setupFns
* [REVIEW add missing test case
* [REVIEW] add test case with getting object from storage and not legacy
* Add deleteCollection
* Test deleteCollection
* Rename file
---------
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Add Get method in dual writer
* Update pkg/apiserver/rest/dualwriter_mode3.go
Co-authored-by: Dan Cech <dcech@grafana.com>
* Actually run the tests
* no need for t.Run()
* Update pkg/apiserver/rest/dualwriter_mode3.go
Co-authored-by: Dan Cech <dcech@grafana.com>
* Update pkg/apiserver/rest/dualwriter_mode3.go
Co-authored-by: Dan Cech <dcech@grafana.com>
* Update pkg/apiserver/rest/dualwriter_test.go
Co-authored-by: Dan Cech <dcech@grafana.com>
* Log improvements
* Fix method name
---------
Co-authored-by: Dan Cech <dcech@grafana.com>
* Make Legacy a public field
* Remove duplicated Create method
* Add logger to dualwriter
* Use klog
* Add comment about selecting the dual writer
* Update pkg/apiserver/rest/dualwriter_mode1.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Update pkg/apiserver/rest/dualwriter_mode3.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Update pkg/apiserver/rest/dualwriter_mode3.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Create error var
* Lint
---------
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
Ryan McKinley
Leonor Oliveira
maicon
Arati R