Topics

moderated Deleting attachments when out of space #update


Benoît Dumeaux
 

For image attachement it could be interesting to have eidtion option for reduce size.


Bruce Bowman
 

On Mon, Feb 24, 2020 at 04:54 PM, Mark Fletcher wrote:
I have not yet figured out how often to run the process that deletes attachments. I'm thinking probably once a week, unless anyone has a reason it should be run more often.
Mark -- Weekly works for me. A couple more things (there's always at least one more thing isn't there?)...

1) When the last attachment is deleted from a message, it still displays a paperclip icon in the Messages/Topics view. This happens whether they are deleted manually or via the attachment purge job.

2) If an attachment is deleted from a message manually (either via editing the message or deleting an image from Emailed Photos), an audit trail entry is created and the message displays an Edited tag. Attachments deleted via the attachment purge job do not.

These are just observations...I'm not proposing that they need to be acted upon.

Thanks,
Bruce


 

On Mon, Feb 24, 2020 at 1:30 PM Bruce Bowman <bruce.bowman@...> wrote:

However, when I go to Admin>Upgrade, it still shows this group as 9% over quota (13% as attachments), and the file and photo upload buttons remain grayed out. I expected all attachments to be deleted to get me to 96%.

So somethin' still ain't right.


You were right.  When figuring out if we had deleted enough attachments, we were not taking into account space used by files and photos. So we were deleting fewer attachments than we should have. I've fixed the bug and am currently doing a run.

I have not yet figured out how often to run the process that deletes attachments. I'm thinking probably once a week, unless anyone has a reason it should be run more often.

Thanks,
Mark


Bruce Bowman
 

On Sun, Feb 16, 2020 at 10:36 AM, Mark Fletcher wrote:
Also, I'm going from most space used to least, so you'll be near the end of the list. 
I'm seeing two log entries now where it says "Attachments up to message #xxx were deleted to free up space via system"

However, when I go to Admin>Upgrade, it still shows this group as 9% over quota (13% as attachments), and the file and photo upload buttons remain grayed out. I expected all attachments to be deleted to get me to 96%.

So somethin' still ain't right.

FYI,
Bruce


 

On Sun, Feb 16, 2020 at 7:11 AM Bruce Bowman <bruce.bowman@...> wrote:
On Tue, Jan 7, 2020 at 06:58 PM, Mark Fletcher wrote:
Attachments will be deleted from the oldest messages first. The text in the messages will not be deleted, just the attachments themselves. I will delete enough attachments to get the group to 95% of their storage limit. After that, the process will run nightly to keep each group under their storage limit.
Mark -- Just following up on this. Per Friday's changelog, the attachment purges appear to have been implemented, but it's not happening in one of my groups. 


I'm working through the groups and it's taking some time. It's not that it's a lot of groups, but the process takes some time for each message. Also, I'm going from most space used to least, so you'll be near the end of the list. 

Thanks,
Mark


Bruce Bowman
 

On Tue, Jan 7, 2020 at 06:58 PM, Mark Fletcher wrote:
Attachments will be deleted from the oldest messages first. The text in the messages will not be deleted, just the attachments themselves. I will delete enough attachments to get the group to 95% of their storage limit. After that, the process will run nightly to keep each group under their storage limit.
Mark -- Just following up on this. Per Friday's changelog, the attachment purges appear to have been implemented, but it's not happening in one of my groups. 

I have 13% attachments, 38% photos and 58% files for a total of 109%. "Storage Limit Reached" (used to be called Out Of Space) is set to Delete Old Attachments. I am unable to upload additional files and photos, which is good. But the purge of oldest attachments has not happened, and I'm at a loss to explain why.

It's not causing me any actual hardship, but I thought you'd want to know. Please contact me off-list if you need the group name.

Regards,
Bruce


Bruce Bowman
 

On Wed, Jan 29, 2020 at 05:01 PM, Mark Fletcher wrote:
Are you envisioning a group that is continually running up against its storage limit not because of attachments, but because of files/photos? And so the enforcer process would end up deleting all attachments in the group each time it runs?
Yes, that's the kind of scenario that I was thinking about.

And given that, there'd be a timing issue with digests going out? 
Perhaps. I'm mildly concerned that the enforcer [insert "hasta la vista, baby" voice here] -- if run before the daily digest compiler/sender -- then some attachments may never make it to the digests containing their corresponding messages. I hope that made sense.

Assuming I understand the request, I guess that's a possibility, but I'm not sure there's anything I can do about it. I currently force digests every night (with the caveat that if someone had recently gotten a digest, a new one doesn't go out). But otherwise digests can go out at any time, depending on messages sent to a group.
This should not be a problem for "intermediate" digests because the nightly enforcer [ditto] presumably hasn't run yet.

Of course, this all assumes that the attachment purge and digest compilation activities share some sort of time-space with respect to system maintenance...see below...

And I've been contemplating changing the forcing of digests so that it would work more like summaries do, so that a digest would be forced at 6am based on a user's timezone, instead of everyone getting forced at 10:30pm pacific time.
Yep...it would be nice for everyone to get up in the morning and find their daily digest available at that time, regardless of their time zone. That said, they might not be so happy if it meant that some attachments that were supposed to come over with that digest didn't make it.

I have no inclination to micromanage on this end...just please give some thought to this, lest it become the source of a bug report somewhere down the line.

Thanks,
Bruce


 

On Wed, Jan 29, 2020 at 12:49 PM Bruce Bowman <bruce.bowman@...> wrote:

If I may, please ensure that the nightly attachment trim is the last function performed in whatever nightly maintenance is scheduled by groups.io. It would be unfortunate if an unexpected bolus of incoming attachments on a given day resulted in some of them being purged before they went out in the daily digest.


First, as a word nerd, I need to say that I really appreciate the use of the word bolus.

I haven't yet set up the time for when the enforcer process will run, but I'm not sure I understand your request. Are you envisioning a group that is continually running up against its storage limit not because of attachments, but because of files/photos? And so the enforcer process would end up deleting all attachments in the group each time it runs? And given that, there'd be a timing issue with digests going out? 

Assuming I understand the request, I guess that's a possibility, but I'm not sure there's anything I can do about it. I currently force digests every night (with the caveat that if someone had recently gotten a digest, a new one doesn't go out). But otherwise digests can go out at any time, depending on messages sent to a group. And I've been contemplating changing the forcing of digests so that it would work more like summaries do, so that a digest would be forced at 6am based on a user's timezone, instead of everyone getting forced at 10:30pm pacific time.

Thanks,
Mark


Bruce Bowman
 

Thanks for the update, Mark. As you might imagine, we're getting some follow-up inquiries in GMF.

If I may, please ensure that the nightly attachment trim is the last function performed in whatever nightly maintenance is scheduled by groups.io. It would be unfortunate if an unexpected bolus of incoming attachments on a given day resulted in some of them being purged before they went out in the daily digest.

Thanks again,
Bruce


 

On Sat, Jan 18, 2020 at 04:33 PM, Bruce Bowman wrote:

Mark -- The previously stated, two-week timeline for the auto-deletion of attachments is fast becoming imminent; so I thought it would be a good idea to revisit this now.

I just sent out the two week warning messages.

-- On what date/time is the "great attachment purge" actually scheduled to occur (~Jan 21)?

It will start two weeks from tomorrow.

-- What was the final decision regarding the resulting entry[s] in the Activity Log?

I will change it so that there will be one log entry per "purge", and it'll list the high water mark message number.

Thanks, Mark


Bruce Bowman
 

Mark -- The previously stated, two-week timeline for the auto-deletion of attachments is fast becoming imminent; so I thought it would be a good idea to revisit this now.

-- Can you confirm that the notification to groups above 80% was actually turned on (~Jan 9), such that folks in that situation have received ample warning?
-- On what date/time is the "great attachment purge" actually scheduled to occur (~Jan 21)?
-- What was the final decision regarding the resulting entry[s] in the Activity Log?

Thanks,
Bruce


Duane
 

On Thu, Jan 9, 2020 at 12:06 PM, SP4149 wrote:
and when found it is impossible for an owner/moderator to strip the attachment and save the message.
Quite easy actually.  Follow the instructions for deleting attachments at https://groups.io/g/GroupManagersForum/wiki/Deleting%20attachments,%20files%20and%20photos

Duane


SP4149
 

For some lists I monitor the attachment image storage for storage that has gotten out of control.  It is not possible to review and search all attachments conveniently.

For example, video files eat up a lot of space and are not subject to the size limitations for image attachments, they don't show-up in the e-mail photo album, very cumbersome to find

and when found it is impossible for an owner/moderator to strip the attachment and save the message.

Before taking Draconian messages, it would be prudent to analyze what files are found stored as attachments and perhaps ban/restrict certain file types that are currently stored in attachments,

but are not controlled by current image size restrictions.

ken clark

www.shastasprings.com



 

On Wed, Jan 8, 2020 at 02:17 AM, Duane wrote:

Not to get too far offtrack, but it might be better if the Out Of Space choice said "Bounce Messages With Attachments" to make it clear.  As is, "Bounce Messages", sounds like all messages would be bounced (though I know they wouldn't.)

I've made this change.

Thanks, Mark


Leeni
 

If attachments are going to be deleted when allotted space is met, is there a way that inserted images can retain the name of the image that is being inserted? Right now if you insert an image in the body of the message and then go to retrieve it from the group's site, it shows up as a number and not the name of the image. The attachments I am showing in this email shows what I am talking about.
 
The image with the 0 as it's name was inserted in the body of the email and saved from the inserted graphic. The image with the tags name was retrieved from the attachment.
 
In many cases the artist of these images want the file name to stay as it was written. But saving it from the body of the email right now doesn't do that.
 
Leeni
 
 
 
 
 

-------Original Message-------
 
Date: 1/8/2020 11:28:14 AM
Subject: Re: [beta] Deleting attachments when out of space
 

On Wed, Jan 8, 2020 at 05:29 AM, Bruce Bowman wrote:

Mark -- Thanks for the heads-up...such is generally my understanding of how the "Out of Space" setting was intended to function. I'm assuming the extra 5% is to allow file and photo uploads to continue to work?

Yes, that's correct.

What happens if 95% of your quota is being used by other items -- would ALL of the group's attachments then be deleted? Not saying that's wrong behavior, but want to clarify.

Yes, all attachments would be deleted.

Also, will upload restrictions now be enforced at the quota level (exactly 1 GB, 10 GB, etc)? In the past, some have reported substantial "grace storage" before such uploads were actually inhibited.

Yes. The "grace storage" was only for attachments, because this process wasn't working. It's always been the case that if your group is over the limit, you could not upload additional photos or files.

Thanks, Mark

 


 

On Wed, Jan 8, 2020 at 09:21 AM, D R Stinson wrote:
In my experience, this is the source of much of the wasted space. We have members who read and share by email, and they just click on reply. The result is the duplication of the same image a number of times. Even if the system just checked for the same image more than once in a thread would be an improvement.
Yes!
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


 

I would really not like to have log entries for individual attachments deleted, ever. It would serve no purpose in my group and could just deluge the log. So if individual entries could not happen at all, or be a group option, that would be great.
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


 

On Wed, Jan 8, 2020 at 05:29 AM, Bruce Bowman wrote:

Mark -- Thanks for the heads-up...such is generally my understanding of how the "Out of Space" setting was intended to function. I'm assuming the extra 5% is to allow file and photo uploads to continue to work?

Yes, that's correct.

What happens if 95% of your quota is being used by other items -- would ALL of the group's attachments then be deleted? Not saying that's wrong behavior, but want to clarify.

Yes, all attachments would be deleted.

Also, will upload restrictions now be enforced at the quota level (exactly 1 GB, 10 GB, etc)? In the past, some have reported substantial "grace storage" before such uploads were actually inhibited.

Yes. The "grace storage" was only for attachments, because this process wasn't working. It's always been the case that if your group is over the limit, you could not upload additional photos or files.

Thanks, Mark


 

How about deduplication first? Eliminate multiple instances of
the same file . . .
Right, that too would be very valuable. If I understand it correctly
your idea is roughly equivalent to (3) here:
https://beta.groups.io/g/main/message/15545
In my experience, this is the source of much of the wasted space. We have members who read and share by email, and they just click on reply. The result is the duplication of the same image a number of times. Even if the system just checked for the same image more than once in a thread would be an improvement.

I also agree as well about posters who put a small image in their signature. By itself they add very little, but they can add up quickly and some posters don't understand they're not saved with the message, but separately. I'm wondering if any image in a signature shouldn't be stored in a members profile where it could be accessed by the system, and all images otherwise in signatures be deleted automatically.

Another 2¢ worth.

Dano


 

Larry,

How about deduplication first? Eliminate multiple instances of the
same file--perhaps even annotating the messages to indicate where the
file has been retained--reducing space with no loss of information.
Right, that too would be very valuable. If I understand it correctly your idea is roughly equivalent to (3) here:
https://beta.groups.io/g/main/message/15545

The only downside I can think of is what Bruce mentioned: it may be too much to implement and test before this sweep.

Shal