DynamoDB TTL Experience: "get" method must return null when item has expired

[hffmnn]

Hi,

I am trying to add a rate limiter to an app with the RateLimiterDynamo class. The key and expire get written to the table but expired items don't get deleted. When searching for TTL and DynamoDB it brings up a lot of threads that say, that one shouldn't rely on TTL values of DynamoDB, e.g. this on Stackoverflow.

I wonder:

• if anyone else has experienced the same and found a workaround
• is anyone useing RateLimiterDynamo in production

[animir]

@hffmnn Hi, what are the oldest not deleted items?

[hffmnn]

In my (test-)case I had a TTL of 5 minutes set but it took 15 Minutes until it got deleted. And it was a test db with 1 entry.

My current workaround is to delete the item manually if msBeforeNext of the rate limiter result is 0.

[animir]

@hffmnn This matters only if you pay for storage. In case of billions of items stored in DynamoDB it may be expensive. Is it your concern?

[hffmnn]

No, this isn't my concern. It only shows that the non-enforce TTL has nothing to do with the current load on the table.

My concern is that if I have a window of 5 minutes set but DynamoDB decides to delete an item after "some" arbitrary time the TTL setting is rather useless (for DynamoDB).

I tested the same setup with the Redis limiter and it works just as expected: After the 5 minutes redis deletes the entry and the rate limiting is gone.
On DynamoDB the rate liimiting is gone after some arbitrary time.

[animir]

@hffmnn rate-limiter-flexible logic doesn't rely on the existence of items. Instead, it checks expire value and compares it with the current time.
Maybe you found a bug?
Could you share your code?

[hffmnn]

@animir My code is mostly the same as in the official example found here

rate-limiter-flexible logic doesn't rely on the existence of items. Instead, it checks expire value and compares it with the current time.

This isn't the case (at least not for the DynomoDB part of things):

When calling get on the limiter it calls _get of RateLimiterDynamo which only returns null if the item doesn't exist. There is no check on expired.

Also when converting the item to a RateLimiterRes: no check if the expired property is still valid

And because of that, the if condition in the following code snippet is always true and only can get false if DynamoDB deletes the item.

const rlResUsername = await limiterConsecutiveFailsByUsername.get(username);

  if (rlResUsername !== null && rlResUsername.consumedPoints > maxConsecutiveFailsByUsername) {
    const retrySecs = Math.round(rlResUsername.msBeforeNext / 1000) || 1;
    res.set('Retry-After', String(retrySecs));
    res.status(429).send('Too Many Requests');
  }

[animir]

@hffmnn Thanks!
It looks like get method is buggy.
As a workaround msBeforeNext could be checked. It should return correct 0 or -1 value when an item has expired.

[hffmnn]

Hi

As a workaround msBeforeNext could be checked. It should return correct 0 or -1 value when an item has expired.

yes, this is exactly what I am doing already:

My current workaround is to delete the item manually if msBeforeNext of the rate limiter result is 0.

[anasdevv]

Is this bug still active, and may I open a PR?

[hffmnn]

I didn't check again, but I guess you can check the linked code to see if it still does what I describe.

[animir]

@anasdevv Nobody fixed it yet. PR is most welcomed.