Add rateLimiter and memoryManager to do memory control and rate limit.

[Yuukadesu]

Purpose

Add two functions separately, RareLimiter to control speed and MemoryManager to control memory size, ultimately preventing excessive CPU usage and memory overload.

Programs

SpRateLimiter
Write a limiter using the speed limiting algorithm encapsulated in Guava, which provides configurable parameters, configurable initialization parameters, and core limit methods

private static final double DEFAULT_PERMITS_PER_SECOND = 100.0;
private static final long DEFAULT_WARMUP_PERIOD = 1000L;
private static final TimeUnit DEFAULT_TIME_UNIT = TimeUnit.MILLISECONDS;
void limit()

SpMemoryManager
Set the memory size (such as 10GB) during initialization, check the memory in a loop during allocation, and if it is insufficient, block and wait for memory allocation; After the event allocated to memory is released, the memory is released.

SpMemoryManager() {
        this.freeMemory = 10L * 1024 * 1024 * 1024; // Initialize with 10 GB of free memory
        log.info("SpMemoryManager initialized with {} bytes of free memory", freeMemory);
    }
void allocate()
void free()

Tracking point
Finally, embed points in the process and onEvent methods of classes such as StandaloneEventSinkRuntime to enable StreamPipes to perform memory control and rate limiting.

SpRateLimiter.INSTANCE.limit();
SpMemoryManager.INSTANCE.allocate(event.length);
SpMemoryManager.INSTANCE.free(event.length);