localstore/db how to insert items in GC index

[adamschmideg]

Related issues:

1. Garbage Collection and Syncing and Proof of Burn - Garbage Collection and Syncing and Proof of Burn #1017 ; https://hackmd.io/t-OQFK3mTsGfrpLCqDrdlw
2. Spam protection of Swarm through modifying GC rules - Spam protection of Swarm through modifying GC rules #757
3. Implement DB store API - Implement DB store API #990 - https://hackmd.io/ffBjSu8RTyKikvRO7bYrzA?both#operations

---

anteva:
It seems to me that this is the main entrypoint issue for the new GC effort.

Whoever works on this and has the most knowledge on this topic: please add the following sections to this issue:

• the current state of the GC in Swarm as of v0.3 (and soon v0.4)
• what problems the current implementation of the GC has. a few use-cases and scenarios where the GC creates problems would be helpful.
• what the new GC solution proposes and whether it addresses the problems listed above.
• what other GC solutions have been considered and why they are dropped.

Without such a history it will be very difficult to understand any of the decisions made in this effort a few months from now.

cc @zelig @nagydani @homotopycolimit @janos

[cobordism]

Honestly, I have no idea what "add interval to accesstimestamp" means.

@janos and I were discussing the following idea the other day:

For each chunk in the database we store 2 extra hashes; something like nexthash and prehash - effectively giving us a double linked list.

We also maintain a list of insertion points [quantile] -> [hash].

When a chunk is added to the database, and we want to add it at the 80% mark for example, we can look up the hash of the chunk that is currently sitting at that insertion point, we add our new chunk, and we update the insertion point hash as well as the prevhash and nexthash of the neighbouring chunks.

When a chunk is served to a retrieval request, it can be inserted at the top of the queue and the [head] pointer will point to it. The prevhash and nexthash of the surrounding chunks are updated to reflect that the chunks has been removed from its previous location.

For bootstrapping, all quantiles (except [head] of course) will map to the end of the chunk queue. This would happen until the total number of chunks has reached 50% of the total storage size. At that point the 50% marker can be set, while the others (60%, 70%... whatever) continue pointing to the last chunk.

GC happens from the tail of the queue, proceeding to prevhash every time.

One main benefit of this idea is that we can maintain multiple insertion points, and we don't have to update every chunk's entry when the queue changes because position-in-the-queue is a property and not an attribute.

[zelig]

@homotopycolimit
the trouble with this idea was that it necessitates a global lock on chunk insertion and thus imposes a performance penalty. Maybe @janos can explain in more detail.

"add interval to accesstimestamp" is meant to refer to @janos solution that at each retrieval access an interval is added to the last accesstimestamp, thereby boosting its rank (add more protection against GC).
The actual interval added can depend on proximity, current accesstimestamp and mode of access (sync/upload/retrieval for peer/retrieval for self)
With this method exponential aging is implemented.
parameters can be learnt by fitting a model to expectations on GC order given real world usecases. The problem is i am not sure we can formulate expectations of the "right" GC order.

[cobordism]

Ah ok.
What we discussed the other day was strictly the quantile insertion, and not the timestamp ranking.

[cobordism]

the trouble with this idea was that it necessitates a global lock on chunk insertion and thus imposes a performance penalty.

why global?

[janos]

why global?

@homotopycolimit relations in double linked list require a lock to ensure that no other parallel goroutine is changing the list. If we want to insert a hash at some percentile, we would need to lock two hashes in between which the new one is inserted, but also their neighbours, as some other chunk may need to be inserted among them at the same time, and so on, which at the end results in the global lock.

[cobordism]

it still feels like a local lock to me :)

but I trust your verdict

[janos]

Descriptions of different Garabge collection implementations

Here are summarized garbage collection suggestions based on meetings with @zelig, @homotopycolimit and @nagydani, and https://hackmd.io/t-OQFK3mTsGfrpLCqDrdlw document.

Note: Chunks are added to garbage collection index only after they have been synced.

Suggestion 1: Quantiles with accessTimestamp index

GC index encoding would be accessTimestamp|hash->nil. This is the same index as the one for the current algorithm, where every time a chunk is accessed it is put on the top of the queue. The addition is to keep track which keys are at specific quantiles. Every new chunk will be added to the the queue at some quantile depending on the origin of the chunk (uploaded, synced, retrieved), and with every new access, chunk will be moved to the top of the queue. Adding a chunk to the quantile would require to identify the key where the quantile is, then to increment that key's accessTimestamp by one and use it as the accessTimestamp for the newly added chunk. That accessTimestamp would not be the access timestamp, but the timestamp at which the chunk would be inserted at the desired quantile. In order to keep exact track of quantiles, a global lock is required.

Pros

Simple algorithm. Fast garbage collection.

Cons

Requires global lock. Additional chunk index for tracking chunk origins hash->origin is required.

Suggestion 2: Quantiles in linked list

GC index structure would be defined hash->leftHash|rightHash, where key is only the chunk hash and value contains hashes of its left and right neighbours in double linked list structure. Additional fields would be required to keep track of at least head and tail of the list, and various quantile position hashes. Every new chunk will be added to the the list at some quantile depending on the origin of the chunk (uploaded, synced, retrieved), and with every new access, chunk will be moved to the start. Inserting a chunk to the list would require identifying the hash at a specific quantile and setting appropriate index values for that hash, the next one and the newly added hash, in a way that they construct unbroken double linked list. Garbage collection would start from the end of the list, deleting the last chunk and identifying the next one from its value as the neighbour chunk.

Pros

Very simple algorithm to understand and test.

Cons

Any operations on double linked list and for keeping track for quantiles a global lock is required. Garbage collection iteration is slow and expensive as it requires to use ldbstore get instead of ldbstore iterator. Tracking quantiles require additional fields. Adding a new quantile or potential calibration is an expensive operation as it requires walking through the whole list with ldbstore get. Additional chunk index for tracking chunk origins hash->origin is required.

Suggestion 3: Ranking function

GC index encoding would be rank+accessTimestamp|hash->nil, where rank is calculated by the function that accepts parameters

• popularity - number of chunk accesses (integer)
• proximity from the node base address (integer)
• chunk origin (origin enum: locally uploaded, synced, retrieved)

and added as integer to the accessTimestamp. Garbage collection would just remove the chunks from the tail of the queue as chunks will be ordered with least important at the end. The rank is actually increasing accessTimestamp by some value calculated by this parameters. The increment is greater as the popularity raises, for more proximate chunks and by chunk origin. Chunks with greater rank will be placed higher in the queue and later garbage collected. The returned value from the rank function would place the chunk at the same time in the future as it would place a chunk that is accessed for the first time at the same proximity and with the same origin. The problem is that the ranking function can be derived only by guesses and empirical data, based on the dynamics of chunks that we predict. Every parameter may have a linear relation to the rank or non linear. With linear relation we are risking that some chunks may be placed in the queue so high that even when not accessed for a long time, they are not garbage collected. So a non linear dependance would be better for popularity parameter, for example a logarithmic. Every parameter needs to have one or more constants that need to define their relation to accessTimestamp. The first order linear constant is in the same unit as accessTimestamp is, nanoseconds. If more proximate chunks need to be prioritized in the queue, the most simple way would be to add more time to the result of the ranking function. For example, the constant for proximity would be 10 minutes (very arbitrary guess) which would be multiplied by the proximity level. If the proximity value raises for more closer chunks, then the ranking function would multiply proximity with the constant of 10 minutes. For popularity, it would multiply logarithm of chunk accesses number with its constant which may be 1h, for example. Chunk origin enumeration would prioritize ranking by adding different fixed constants based on enum value. The rank function can be as simple as linear function of independent parameters rank(pop, prox, orig) = c1*pop + c2*prox + c2*orig, or more complex non linear rank(pop, prox, orig) = c1*log(pop) + c2*prox + c2*orig, to a very complex function of non-orthogonal parameters rang(pop, prox, orig) = f1(pop, prox) + f2(prox, orig) + f3(pop, orig).

Pros

Chunks in garbage collection index are completely independent and can be inserted or moved in queue without a global lock. Iteration on garbage collection index is the least expensive operation for iterating over leveldb key/value pairs, as keys are already sorted on the database level.

Cons

Constants and the formula for the ranking function can be defined only by empirical data and our assumptions about chunk prioritization in time, which may be very subjective, making a consensus on it very difficult, or impossible to achieve. At least two additional indexes need to be added. One for popularity counters hash->popularity for every chunk and one for the chunk origin hash->origin. These indexes are needed for rank calculation.

Suggestion 4: Self adjusting quantiles or ranking function

This would be an alternative to the previous suggestions where quantiles or ranking function constants would be calibrated at runtime. This would be a very advanced approach, but requires discussion and formalization.