Skip to content
This repository was archived by the owner on Aug 2, 2021. It is now read-only.

Conversation

@zelig
Copy link
Member

@zelig zelig commented Apr 1, 2019

This PR implements push syncing protocol in a new package swarm/storage/pusher

The PR uses the localstore integration branch until it is merged.

The entry point to push sync is the RunPushSync function which will be called on swarm.go level.
It uses two interfaces:

  • PushSyncIndex
    • to iterate over chunks to be push synced
    • to mark items synced
    • the new localstore SubscribePush and Set methods implements this.
  • PubSub
    • to communicate
    • pss.PubSub implements this interface

storer nodes which accept push synced content will run RunStorer on swarm.go level

It obsoletes the earlier #971
and greatly simplifies the code.

It needs:

  • integration test with localstore and pss pubsub
  • simulation test with many nodes
  • benchmarks

Copy link
Contributor

@nolash nolash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The review contains some comments from parts of the diff that is out of scope of the push sync stuff; I didn't realize the scope until later on. Handling them is optional then, I guess.

Validate(ch Chunk) bool
}

// ValidatorStore encapsulates Store by decorting the Put method
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does decorting mean?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a typo, it should be decorating.

SubscribePush(context.Context) (<-chan storage.Chunk, func())
// called to set a chunk as synced - and allow it to be garbage collected
// TODO this should take ... last argument to delete many in one batch
Set(context.Context, chunk.ModeSet, storage.Address) error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's set the interface correctly right away, but comment it only handles the first element for now.

@nolash
Copy link
Contributor

nolash commented Apr 12, 2019

needs rebase @zelig

@zelig zelig mentioned this pull request Apr 15, 2019
20 tasks
@zelig zelig requested review from acud and nonsense as code owners May 8, 2019 07:34
@zelig zelig changed the base branch from master to swarm-rather-stable May 8, 2019 09:33
nonsense and others added 17 commits May 10, 2019 10:21
* swarm/shed: remove metrics fields from DB struct

* swarm/schunk: add String methods to modes

* swarm/storage/localstore: add metrics and traces

* swarm/chunk: unknown modes without spaces in String methods

* swarm/storage/localstore: remove bin number from pull subscription metrics

* swarm/storage/localstore: add resetting time metrics and code improvements
* swarm/chunk: add tags backend to chunk package
swarm/api: integrate tags to count chunks being split and stored
swarm/api/http: integrate tags in middleware for HTTP `POST` calls and assert chunks being calculated and counted correctly
swarm: remove deprecated and unused code, add swarm hash to DoneSplit signature, remove calls to the api client from the http package
@zelig
Copy link
Member Author

zelig commented May 10, 2019

reopen

@zelig zelig reopened this May 10, 2019
@nonsense nonsense force-pushed the swarm-rather-stable branch 4 times, most recently from ad91427 to 9b1543c Compare May 10, 2019 10:29
@nonsense nonsense mentioned this pull request Jun 18, 2019
2 tasks
@acud
Copy link
Contributor

acud commented Jun 18, 2019

superseded by #1392

@acud acud closed this Jun 18, 2019
@nonsense nonsense deleted the pushsync branch September 30, 2019 10:43
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants