Topics

moderated Deleting attachments when out of space #update


 

Hi All,

The process that deletes attachments from messages when a group runs out of space has had a bug in it for quite some time, preventing it from working (insert Homer Simpson DOH! here). I have fixed that bug and will start up the process again soon, but I wanted to outline it before I did so.

There are two parts to this system, the one time notification part, and the nightly deletion part.

Starting in the next day or so, I will turn on the notification part. This part looks at all groups every night and the *first time* a group goes over 80% of their storage limit, it sends the following email to the group owners telling them this and outlining the options:

This email message is a notification to let you know that your group GROUPNAME is using over 80% of its available storage of XXGBs. The storage space is used by attachments, files, photos, database images and wiki images. You can view the usage statistics for your group here.
Based on your group settings, when the storage limit is reached, messages to the group that contain attachments will be bounced./the oldest attachments will be automatically deleted to make space.
Note: If your group is already over its storage allotment, we will delete attachments starting in two weeks from when you received this notice.
If you would like to increase the amount of storage space available for your group, please visit:
GROUPURL/plans
Alternatively, you can set your group to bounce messages that contain attachments.
If you have any questions, please visit our help center or contact us at support@groups.io.


In two weeks, for those groups that are over their limit, attachments will be deleted to get each group under the storage limit. Attachments will be deleted from the oldest messages first. The text in the messages will not be deleted, just the attachments themselves. I will delete enough attachments to get the group to 95% of their storage limit. After that, the process will run nightly to keep each group under their storage limit.

There is a new activity log entry, "Attachments deleted for space." That will log each message whose attachments have been deleted. My thought is for the initial deletion that I will not log each of those messages, to avoid a mass spamming of everyone's activity log. But after that I will turn it on.

To view how much space your group is using, click on the Billing or Upgrade tab under Admin for your group.

Please let me know if you have any questions or suggestions.

Thanks,
Mark


Gerald Boutin <groupsio@...>
 

On Tue, Jan 7, 2020 at 07:58 PM, Mark Fletcher wrote:
Please let me know if you have any questions or suggestions.
This almost sounds cumbersome.

What if part two was when the 100% limit was reached and new uploads were no longer allowed? This would give group owners control on prioritization as to what got deleted.

--
Gerald


 

On Tue, Jan 7, 2020 at 7:57 PM Gerald Boutin <groupsio@...> wrote:

What if part two was when the 100% limit was reached and new uploads were no longer allowed? This would give group owners control on prioritization as to what got deleted.


That is indeed an existing group setting, at the bottom of the `Message Policies` section of the Group Settings page.

Thanks,
Mark 


 

Mark,

My thought is for the initial deletion that I will not log each of
those messages, to avoid a mass spamming of everyone's activity log.
Would it be possible during the initial deletion to have a single log entry showing the high-water mark of the deletion (date and/or message number) and the number of affected messages?

Possibly this could be used in lieu of individual entries any time a deletion batch exceeds some (large) number of affected messages.

Also, bump: https://beta.groups.io/g/main/message/14338

Shal


Michael Pavan
 

Mark,

On Jan 8, 2020, at 1:39 AM, Shal Farley <@Shal> wrote:

Mark,

My thought is for the initial deletion that I will not log each of
those messages, to avoid a mass spamming of everyone's activity log.
Would it be possible during the initial deletion to have a single log entry showing the high-water mark of the deletion (date and/or message number) and the number of affected messages?
Yes, please.

Possibly this could be used in lieu of individual entries any time a deletion batch exceeds some (large) number of affected messages.
Perhaps with a link (or drop down menu) to the list of individual deletions...


Also, bump: https://beta.groups.io/g/main/message/14338
Agreed.
It would be great to be able to move attachments from messages to Files or Photos, where they could be rationally organized - not only would this help to preserve valued attachments, but make them easier to find or access. It would be a bonus if a link to the attachment remained with the message should someone happen to be looking at old messages.

Attachments when moved to Files or Photos could be titled intuitively, rather than having to edit message Subject lines which often aren't (and the notifications that message editing can trigger).

Michael


Chris Jones
 

On Wed, Jan 8, 2020 at 08:56 AM, Michael Pavan wrote:
It would be great to be able to move attachments from messages to Files or Photos
I think that would be a very popular option! However, IIRC non - photo format attachments cannot be found in the emailed photos album (although they are in there somewhere) so there would IMHO need to be a way of finding them more easily.

Chris


Duane
 

On Tue, Jan 7, 2020 at 11:25 PM, Mark Fletcher wrote:
That is indeed an existing group setting, at the bottom of the `Message Policies` section of the Group Settings page.
 
Not to get too far offtrack, but it might be better if the Out Of Space choice said "Bounce Messages With Attachments" to make it clear.  As is, "Bounce Messages", sounds like all messages would be bounced (though I know they wouldn't.)

Having the 'Keep' option Shal mentioned in his other message, that moderators could use to prevent an attachment from being deleted during the purge, might be less work than moving?  This way there'd be no need to keep things related, as with a move, but I can see situations where being able to save the information elsewhere would be useful.  I don't think it should be a member option because most likely everyone would feel that their attachment needed to be saved.

Thanks,
Duane


Laurence Marks <marks@...>
 

How about deduplication first? Eliminate multiple instances of the same file--perhaps even annotating the messages to indicate where the file has been retained--reducing space with no loss of information.

To carry this thought a little further (and this may be out of scope), there are schemes where files differing only slightly have the common parts deduplicated. Riverbed was one of the first firms to provide hardware appliances that do this, but now it's a complete industry. See   Deduplication Appliance Buying Guide


Larry


--
Larry Marks


Bruce Bowman
 

On Tue, Jan 7, 2020 at 06:58 PM, Mark Fletcher wrote:
In two weeks, for those groups that are over their limit, attachments will be deleted to get each group under the storage limit. Attachments will be deleted from the oldest messages first. The text in the messages will not be deleted, just the attachments themselves. I will delete enough attachments to get the group to 95% of their storage limit. After that, the process will run nightly to keep each group under their storage limit.
Mark -- Thanks for the heads-up...such is generally my understanding of how the "Out of Space" setting was intended to function. I'm assuming the extra 5% is to allow file and photo uploads to continue to work?

What happens if 95% of your quota is being used by other items -- would ALL of the group's attachments then be deleted? Not saying that's wrong behavior, but want to clarify.

Also, will upload restrictions now be enforced at the quota level (exactly 1 GB, 10 GB, etc)? In the past, some have reported substantial "grace storage" before such uploads were actually inhibited.

In this context I hope we can focus on completion of the bug-fix (and minimizing its "surprise factor" on existing groups currently over quota) before attempting to introduce new, indirectly-related features. It seems as if this thing could get complicated enough without going down that road.

My $0.02,
Bruce


 

Larry,

How about deduplication first? Eliminate multiple instances of the
same file--perhaps even annotating the messages to indicate where the
file has been retained--reducing space with no loss of information.
Right, that too would be very valuable. If I understand it correctly your idea is roughly equivalent to (3) here:
https://beta.groups.io/g/main/message/15545

The only downside I can think of is what Bruce mentioned: it may be too much to implement and test before this sweep.

Shal


 

How about deduplication first? Eliminate multiple instances of
the same file . . .
Right, that too would be very valuable. If I understand it correctly
your idea is roughly equivalent to (3) here:
https://beta.groups.io/g/main/message/15545
In my experience, this is the source of much of the wasted space. We have members who read and share by email, and they just click on reply. The result is the duplication of the same image a number of times. Even if the system just checked for the same image more than once in a thread would be an improvement.

I also agree as well about posters who put a small image in their signature. By itself they add very little, but they can add up quickly and some posters don't understand they're not saved with the message, but separately. I'm wondering if any image in a signature shouldn't be stored in a members profile where it could be accessed by the system, and all images otherwise in signatures be deleted automatically.

Another 2¢ worth.

Dano


 

On Wed, Jan 8, 2020 at 05:29 AM, Bruce Bowman wrote:

Mark -- Thanks for the heads-up...such is generally my understanding of how the "Out of Space" setting was intended to function. I'm assuming the extra 5% is to allow file and photo uploads to continue to work?

Yes, that's correct.

What happens if 95% of your quota is being used by other items -- would ALL of the group's attachments then be deleted? Not saying that's wrong behavior, but want to clarify.

Yes, all attachments would be deleted.

Also, will upload restrictions now be enforced at the quota level (exactly 1 GB, 10 GB, etc)? In the past, some have reported substantial "grace storage" before such uploads were actually inhibited.

Yes. The "grace storage" was only for attachments, because this process wasn't working. It's always been the case that if your group is over the limit, you could not upload additional photos or files.

Thanks, Mark


 

I would really not like to have log entries for individual attachments deleted, ever. It would serve no purpose in my group and could just deluge the log. So if individual entries could not happen at all, or be a group option, that would be great.
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


 

On Wed, Jan 8, 2020 at 09:21 AM, D R Stinson wrote:
In my experience, this is the source of much of the wasted space. We have members who read and share by email, and they just click on reply. The result is the duplication of the same image a number of times. Even if the system just checked for the same image more than once in a thread would be an improvement.
Yes!
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


Leeni
 

If attachments are going to be deleted when allotted space is met, is there a way that inserted images can retain the name of the image that is being inserted? Right now if you insert an image in the body of the message and then go to retrieve it from the group's site, it shows up as a number and not the name of the image. The attachments I am showing in this email shows what I am talking about.
 
The image with the 0 as it's name was inserted in the body of the email and saved from the inserted graphic. The image with the tags name was retrieved from the attachment.
 
In many cases the artist of these images want the file name to stay as it was written. But saving it from the body of the email right now doesn't do that.
 
Leeni
 
 
 
 
 

-------Original Message-------
 
Date: 1/8/2020 11:28:14 AM
Subject: Re: [beta] Deleting attachments when out of space
 

On Wed, Jan 8, 2020 at 05:29 AM, Bruce Bowman wrote:

Mark -- Thanks for the heads-up...such is generally my understanding of how the "Out of Space" setting was intended to function. I'm assuming the extra 5% is to allow file and photo uploads to continue to work?

Yes, that's correct.

What happens if 95% of your quota is being used by other items -- would ALL of the group's attachments then be deleted? Not saying that's wrong behavior, but want to clarify.

Yes, all attachments would be deleted.

Also, will upload restrictions now be enforced at the quota level (exactly 1 GB, 10 GB, etc)? In the past, some have reported substantial "grace storage" before such uploads were actually inhibited.

Yes. The "grace storage" was only for attachments, because this process wasn't working. It's always been the case that if your group is over the limit, you could not upload additional photos or files.

Thanks, Mark

 


 

On Wed, Jan 8, 2020 at 02:17 AM, Duane wrote:

Not to get too far offtrack, but it might be better if the Out Of Space choice said "Bounce Messages With Attachments" to make it clear.  As is, "Bounce Messages", sounds like all messages would be bounced (though I know they wouldn't.)

I've made this change.

Thanks, Mark


SP4149
 

For some lists I monitor the attachment image storage for storage that has gotten out of control.  It is not possible to review and search all attachments conveniently.

For example, video files eat up a lot of space and are not subject to the size limitations for image attachments, they don't show-up in the e-mail photo album, very cumbersome to find

and when found it is impossible for an owner/moderator to strip the attachment and save the message.

Before taking Draconian messages, it would be prudent to analyze what files are found stored as attachments and perhaps ban/restrict certain file types that are currently stored in attachments,

but are not controlled by current image size restrictions.

ken clark

www.shastasprings.com



Duane
 

On Thu, Jan 9, 2020 at 12:06 PM, SP4149 wrote:
and when found it is impossible for an owner/moderator to strip the attachment and save the message.
Quite easy actually.  Follow the instructions for deleting attachments at https://groups.io/g/GroupManagersForum/wiki/Deleting%20attachments,%20files%20and%20photos

Duane


Bruce Bowman
 

Mark -- The previously stated, two-week timeline for the auto-deletion of attachments is fast becoming imminent; so I thought it would be a good idea to revisit this now.

-- Can you confirm that the notification to groups above 80% was actually turned on (~Jan 9), such that folks in that situation have received ample warning?
-- On what date/time is the "great attachment purge" actually scheduled to occur (~Jan 21)?
-- What was the final decision regarding the resulting entry[s] in the Activity Log?

Thanks,
Bruce


 

On Sat, Jan 18, 2020 at 04:33 PM, Bruce Bowman wrote:

Mark -- The previously stated, two-week timeline for the auto-deletion of attachments is fast becoming imminent; so I thought it would be a good idea to revisit this now.

I just sent out the two week warning messages.

-- On what date/time is the "great attachment purge" actually scheduled to occur (~Jan 21)?

It will start two weeks from tomorrow.

-- What was the final decision regarding the resulting entry[s] in the Activity Log?

I will change it so that there will be one log entry per "purge", and it'll list the high water mark message number.

Thanks, Mark