Topics

moderated Deleting attachments when out of space #update


Bruce Bowman
 

Thanks for the update, Mark. As you might imagine, we're getting some follow-up inquiries in GMF.

If I may, please ensure that the nightly attachment trim is the last function performed in whatever nightly maintenance is scheduled by groups.io. It would be unfortunate if an unexpected bolus of incoming attachments on a given day resulted in some of them being purged before they went out in the daily digest.

Thanks again,
Bruce


 

On Wed, Jan 29, 2020 at 12:49 PM Bruce Bowman <bruce.bowman@...> wrote:

If I may, please ensure that the nightly attachment trim is the last function performed in whatever nightly maintenance is scheduled by groups.io. It would be unfortunate if an unexpected bolus of incoming attachments on a given day resulted in some of them being purged before they went out in the daily digest.


First, as a word nerd, I need to say that I really appreciate the use of the word bolus.

I haven't yet set up the time for when the enforcer process will run, but I'm not sure I understand your request. Are you envisioning a group that is continually running up against its storage limit not because of attachments, but because of files/photos? And so the enforcer process would end up deleting all attachments in the group each time it runs? And given that, there'd be a timing issue with digests going out? 

Assuming I understand the request, I guess that's a possibility, but I'm not sure there's anything I can do about it. I currently force digests every night (with the caveat that if someone had recently gotten a digest, a new one doesn't go out). But otherwise digests can go out at any time, depending on messages sent to a group. And I've been contemplating changing the forcing of digests so that it would work more like summaries do, so that a digest would be forced at 6am based on a user's timezone, instead of everyone getting forced at 10:30pm pacific time.

Thanks,
Mark


Bruce Bowman
 

On Wed, Jan 29, 2020 at 05:01 PM, Mark Fletcher wrote:
Are you envisioning a group that is continually running up against its storage limit not because of attachments, but because of files/photos? And so the enforcer process would end up deleting all attachments in the group each time it runs?
Yes, that's the kind of scenario that I was thinking about.

And given that, there'd be a timing issue with digests going out? 
Perhaps. I'm mildly concerned that the enforcer [insert "hasta la vista, baby" voice here] -- if run before the daily digest compiler/sender -- then some attachments may never make it to the digests containing their corresponding messages. I hope that made sense.

Assuming I understand the request, I guess that's a possibility, but I'm not sure there's anything I can do about it. I currently force digests every night (with the caveat that if someone had recently gotten a digest, a new one doesn't go out). But otherwise digests can go out at any time, depending on messages sent to a group.
This should not be a problem for "intermediate" digests because the nightly enforcer [ditto] presumably hasn't run yet.

Of course, this all assumes that the attachment purge and digest compilation activities share some sort of time-space with respect to system maintenance...see below...

And I've been contemplating changing the forcing of digests so that it would work more like summaries do, so that a digest would be forced at 6am based on a user's timezone, instead of everyone getting forced at 10:30pm pacific time.
Yep...it would be nice for everyone to get up in the morning and find their daily digest available at that time, regardless of their time zone. That said, they might not be so happy if it meant that some attachments that were supposed to come over with that digest didn't make it.

I have no inclination to micromanage on this end...just please give some thought to this, lest it become the source of a bug report somewhere down the line.

Thanks,
Bruce


Bruce Bowman
 

On Tue, Jan 7, 2020 at 06:58 PM, Mark Fletcher wrote:
Attachments will be deleted from the oldest messages first. The text in the messages will not be deleted, just the attachments themselves. I will delete enough attachments to get the group to 95% of their storage limit. After that, the process will run nightly to keep each group under their storage limit.
Mark -- Just following up on this. Per Friday's changelog, the attachment purges appear to have been implemented, but it's not happening in one of my groups. 

I have 13% attachments, 38% photos and 58% files for a total of 109%. "Storage Limit Reached" (used to be called Out Of Space) is set to Delete Old Attachments. I am unable to upload additional files and photos, which is good. But the purge of oldest attachments has not happened, and I'm at a loss to explain why.

It's not causing me any actual hardship, but I thought you'd want to know. Please contact me off-list if you need the group name.

Regards,
Bruce


 

On Sun, Feb 16, 2020 at 7:11 AM Bruce Bowman <bruce.bowman@...> wrote:
On Tue, Jan 7, 2020 at 06:58 PM, Mark Fletcher wrote:
Attachments will be deleted from the oldest messages first. The text in the messages will not be deleted, just the attachments themselves. I will delete enough attachments to get the group to 95% of their storage limit. After that, the process will run nightly to keep each group under their storage limit.
Mark -- Just following up on this. Per Friday's changelog, the attachment purges appear to have been implemented, but it's not happening in one of my groups. 


I'm working through the groups and it's taking some time. It's not that it's a lot of groups, but the process takes some time for each message. Also, I'm going from most space used to least, so you'll be near the end of the list. 

Thanks,
Mark


Bruce Bowman
 

On Sun, Feb 16, 2020 at 10:36 AM, Mark Fletcher wrote:
Also, I'm going from most space used to least, so you'll be near the end of the list. 
I'm seeing two log entries now where it says "Attachments up to message #xxx were deleted to free up space via system"

However, when I go to Admin>Upgrade, it still shows this group as 9% over quota (13% as attachments), and the file and photo upload buttons remain grayed out. I expected all attachments to be deleted to get me to 96%.

So somethin' still ain't right.

FYI,
Bruce


 

On Mon, Feb 24, 2020 at 1:30 PM Bruce Bowman <bruce.bowman@...> wrote:

However, when I go to Admin>Upgrade, it still shows this group as 9% over quota (13% as attachments), and the file and photo upload buttons remain grayed out. I expected all attachments to be deleted to get me to 96%.

So somethin' still ain't right.


You were right.  When figuring out if we had deleted enough attachments, we were not taking into account space used by files and photos. So we were deleting fewer attachments than we should have. I've fixed the bug and am currently doing a run.

I have not yet figured out how often to run the process that deletes attachments. I'm thinking probably once a week, unless anyone has a reason it should be run more often.

Thanks,
Mark


Bruce Bowman
 

On Mon, Feb 24, 2020 at 04:54 PM, Mark Fletcher wrote:
I have not yet figured out how often to run the process that deletes attachments. I'm thinking probably once a week, unless anyone has a reason it should be run more often.
Mark -- Weekly works for me. A couple more things (there's always at least one more thing isn't there?)...

1) When the last attachment is deleted from a message, it still displays a paperclip icon in the Messages/Topics view. This happens whether they are deleted manually or via the attachment purge job.

2) If an attachment is deleted from a message manually (either via editing the message or deleting an image from Emailed Photos), an audit trail entry is created and the message displays an Edited tag. Attachments deleted via the attachment purge job do not.

These are just observations...I'm not proposing that they need to be acted upon.

Thanks,
Bruce


Benoît Dumeaux
 

For image attachement it could be interesting to have eidtion option for reduce size.