Differential backups are forced in ShadowProtect SPX

jmt356
SilverLounger
Posts: 2419
Joined: 28 Mar 2010, 01:49

Differential backups are forced in ShadowProtect SPX

Post by jmt356 »

I am using ShadowProtect SPX to create incremental backups of my laptop. SPX is set to take incremental backups at scheduled increments. However, SPX forces differential backups for nearly half of the scheduled backups. ShadowProtect support is saying that differential backups are forced following the installation of Windows Updates. In other words, if a Windows Update is installed, the scheduled incremental backup will be displaced and a differential backup will instead be forced. I get a Windows Update downloaded and installed on my system every few days. This leads to a large number of incremental backups to be displaced.

I do not find ShadowProtect's explanation be credible. Virtually every Windows 10 system has updates constantly being downloaded and installed. I think many people would get frustrated if every time an update was installed, the incremental backup, which should take a few minutes, is displaced by a differential backup, which takes hours to complete.

Has anyone else had an issue with scheduled SPX incremental backups being displaced by differential backups? If so, was the cause Windows Updates?
Regards,

JMT

User avatar
BobArch2
BronzeLounger
Posts: 1253
Joined: 25 Jan 2010, 22:25
Location: Pickering, Ontario, Canada

Re: Differential backups are forced in ShadowProtect SPX

Post by BobArch2 »

jmt356 wrote:Has anyone else had an issue with scheduled SPX incremental backups being displaced by differential backups? If so, was the cause Windows Updates?
I run full SPX backups on Monday and incrementals once daily for the remaining days in the week. The schedule for both are set for 8:30 a.m.

I find that when the monthly Window 10 updates are performed on the second Tuesday of each month I will get a differential backup on the next day, Wednesday. The only other Win10 updates that are performed daily are the definition files for Defender and they do not invoke differential backups.

The other feature that can cause differentials is the Fast Start feature in Windows 10. Turning off Fast Start resolves the problem.

My full weekly backups of C: (SSD) and D: (HDD) take about 45 minutes. File sizes of SPF about 45GB (C:) and 250GB (D:). Incrementals are 1-2 minutes. When differentials occur the timing jumps from 1-2 minutes to 6-7 minutes ... tolerable.

Turning off Fast Start
Regards,
Bob

jmt356
SilverLounger
Posts: 2419
Joined: 28 Mar 2010, 01:49

Re: Differential backups are forced in ShadowProtect SPX

Post by jmt356 »

I don't believe I get monthly Windows 10 updates. I get updates every few days, and that's counting only Quality Updates. Last week, Cumulative Update for .NET Frameworks 3.5 and 4.8 for Windows 10 Version 1903 for x64 was installed on my system. The next incremental backup I tried taking after that Quality Update was forced into a differential backup. If counting Driver Updates and Other Updates, the frequency is about every 2-3 days. If counting Definition Updates, the frequency is daily.

Should Quality Updates, Driver Updates and Other Updates be triggering differential backups?

Is there a way to get all of these updates to download and install only once a month? It seems I can delay these updates for only up to 8 days.

I wish I could get differential backups in 6-7 minutes. If I did, I would not be posting here. They usually take me about 2 hours.

I will also look into whether turning off Fast Startup prevents forced differential backups.
Regards,

JMT

User avatar
HansV
Administrator
Posts: 79669
Joined: 16 Jan 2010, 00:14
Status: Microsoft MVP
Location: Wageningen, The Netherlands

Re: Differential backups are forced in ShadowProtect SPX

Post by HansV »

As Bob wrote, Windows Defender definition updates should not trigger a differential backup.
Here is my list of Quality Updates (using dd-mm-yyyy format):
S3046.png
You can click "Pause updates for 7 days" up to 5 times, for a total of 35 days:
S3047.png
You do not have the required permissions to view the files attached to this post.
Best wishes,
Hans

jmt356
SilverLounger
Posts: 2419
Joined: 28 Mar 2010, 01:49

Re: Differential backups are forced in ShadowProtect SPX

Post by jmt356 »

Hans, how do you get your quality updates to install only on the second Tuesday of each month? I get updates every few days. See attached. I'd like to display a more complete history, but View update history is only showing the last 4 Quality Updates.

Is there a way to pause updates for 35 days in one shot? The problem with pausing for 7 days 5 times is it requires remembering on 5 occasions to pause on day 7. Otherwise, I assume updates are automatically downloaded and forced to install.
You do not have the required permissions to view the files attached to this post.
Regards,

JMT

User avatar
HansV
Administrator
Posts: 79669
Joined: 16 Jan 2010, 00:14
Status: Microsoft MVP
Location: Wageningen, The Netherlands

Re: Differential backups are forced in ShadowProtect SPX

Post by HansV »

1) I do something that many users (Woody in particular) abhor: I click 'Check for Updates' on the 2nd Tuesday of the month, after the local equivalent of 9 PM Seattle time, then install what is found.

2) You can click 'Pause updates for 7 days' up to 5 times in a row.
Best wishes,
Hans

User avatar
BobArch2
BronzeLounger
Posts: 1253
Joined: 25 Jan 2010, 22:25
Location: Pickering, Ontario, Canada

Re: Differential backups are forced in ShadowProtect SPX

Post by BobArch2 »

jmt356 wrote:I wish I could get differential backups in 6-7 minutes. If I did, I would not be posting here. They usually take me about 2 hours.

I will also look into whether turning off Fast Startup prevents forced differential backups.
Any relief with turning off Fast Start?

I have been doing some research on “differentials”. It seems some backup systems that invoke differentials (when deemed needed) will compare the last created full backup to the current environment and create a file containing all new or changed files. As compared to other backup systems that will create new backups of only the differences between current environment and the last incremental. Based on my experience with SPX, differential created files resemble the latter. That is, the files are small, with no evidence of being created by the comparison to the last full backup. :scratch:

I am puzzled why your differentials take about 2 hours. How long does it take you to do a full image and how large are the resulting files? Lastly, what schedule are you following to create full and incrementals?
Regards,
Bob

User avatar
BobArch2
BronzeLounger
Posts: 1253
Joined: 25 Jan 2010, 22:25
Location: Pickering, Ontario, Canada

Re: Differential backups are forced in ShadowProtect SPX

Post by BobArch2 »

jmt356 wrote:Has anyone else had an issue with scheduled SPX incremental backups being displaced by differential backups? If so, was the cause Windows Updates?
Update: I have taken a closer look at my three weekly cycles of SPX backups and have determined that I am also experiencing excessive differentials.

First some background info ...
1) My system drive (C:) is a 500GB Samsung 860 EVO SSD drive which replaced a 5 year old 250GB 840 EVO SSD drive December 27, 2019.
2) The Power Options are those as recommended by Samsung.
3) The Fast Start feature is not turned on.
4) My data drive is a 2TB WD Black internal hard drive
5) I power down the system before going to bed and there does not appear to be any problems with the power down process (no "dirty" shut downs).

The snapshot shown below reflects the backup sizes and timings for the last 3 weeks. As can be seen, the differentials are excessive. The only one I would have expected is the one the day after my Monthly Quality Win10 updates (which I manually check for on the second Tuesday of the month - late afternoon).

I have also included in the snapshot a reference to the differences between incrementals and differentials along with a diagram from EaseUS depicting the process. These references are different, I believe, from how SPX determines the need for a differential because the size of my differentials does not follow the pattern depicted in those references.

As a side note, I have PerfectDisk installed as a defragmentation tool. I first acquired it about 15 years ago but have not run the process for at least 5 years. In examining my system in the last couple of days I determined that PD is run at start up and it has a feature to prevent defragmentation I disabled PD earlier today and we'll see if that has any impact.

As a last comment, I filed a support request with StorageCraft this morning explaining all of the above. I will keep you informed of any of their replies.
BackupInfoEdited.jpg
[Edited 2020.01.27 to correct a typo where an incremental file size was shown as 837.3GB which should be 837.3MB]
You do not have the required permissions to view the files attached to this post.
Regards,
Bob

User avatar
BobArch2
BronzeLounger
Posts: 1253
Joined: 25 Jan 2010, 22:25
Location: Pickering, Ontario, Canada

Re: Differential backups are forced in ShadowProtect SPX

Post by BobArch2 »

Just a fast reply ... I heard back from StorageCraft yesterday and they were not able to spot any issues (other than VDIFF being invoked) in the SPX log files I had sent them. They did make a couple of suggestions, which I am going to ignore, and am trying some other modifications to my environment to see if those mods have any impact. I tried 3 mods yesterday and had success in eliminating the differential. But, one day of testing is not sufficient in my mind. I'll see what happens over the next 3 days of normal activities. If successful, I’ll post what changes I made.

I'll keep you informed.

P.S. their suggestion was to turn off start up services/apps and through process of reintroduction, see if that finds the trigger that damages the incremental record. A fast check of services and start up apps indicate that it would take weeks to try the combinations. No way Jose!!!
Regards,
Bob

User avatar
PaulB
SilverLounger
Posts: 1619
Joined: 26 Jan 2010, 20:28
Location: Ottawa ON

Re: Differential backups are forced in ShadowProtect SPX

Post by PaulB »

Bob, my backup methodology mirrors yours (weekly fulls, daily incrementals) and all I ever see are .spf (full) and .spi (incremental) file extensions, along with their .mp5 files. Fulls (:C and :D in my case) take slightly less than 1 hour while incrementals usually take less that 1 or 2 minutes. I do occasionally have a :C incremental take 5-15 minutes. I attribute this to the large number of files changed with the Patch Tuesday updates.

How do you identify a SPX differential backup file in File Explorer?
Regards,
Paul

The pessimist complains about the wind. The optimist expects it to change. The realist adjusts his sails.

User avatar
BobArch2
BronzeLounger
Posts: 1253
Joined: 25 Jan 2010, 22:25
Location: Pickering, Ontario, Canada

Re: Differential backups are forced in ShadowProtect SPX

Post by BobArch2 »

PaulB wrote:How do you identify a SPX differential backup file in File Explorer?
Hi Paul,

Differentials do not have a different extension. A differential will have the SPI extension. You can tell that a differential has taken place if the time to perform an expected incremental takes longer than normal. And, if you open SPX and check the job schedule you will see that it is colour coded brown. Full backups are blue, incrementals are green, failures are red and I think there is a grey type. (I am responding via my iPad at the moment and am not in front of the PC.)

I have noticed that the day after the monthly updates I invariably get a differential and not an incremental.
Regards,
Bob

User avatar
PaulB
SilverLounger
Posts: 1619
Joined: 26 Jan 2010, 20:28
Location: Ottawa ON

Re: Differential backups are forced in ShadowProtect SPX

Post by PaulB »

I checked the job schedule as you indicated and found that indeed I did have a differential backup on 15 Jan. This was the date that the Patch Tuesday updates were applied. I will follow this thread with greater attention now!
Regards,
Paul

The pessimist complains about the wind. The optimist expects it to change. The realist adjusts his sails.

jmt356
SilverLounger
Posts: 2419
Joined: 28 Mar 2010, 01:49

Re: Differential backups are forced in ShadowProtect SPX

Post by jmt356 »

Is there a way to get Windows updates to install only once a month on Patch Tuesday rather than every few days other than having to constantly pause updates?
Regards,

JMT

User avatar
HansV
Administrator
Posts: 79669
Joined: 16 Jan 2010, 00:14
Status: Microsoft MVP
Location: Wageningen, The Netherlands

Re: Differential backups are forced in ShadowProtect SPX

Post by HansV »

Don't click "Check for updates".
Best wishes,
Hans

jmt356
SilverLounger
Posts: 2419
Joined: 28 Mar 2010, 01:49

Re: Differential backups are forced in ShadowProtect SPX

Post by jmt356 »

I don't, but I still have frequently gotten updates installed between Patch Tuesdays.
Regards,

JMT

User avatar
HansV
Administrator
Posts: 79669
Joined: 16 Jan 2010, 00:14
Status: Microsoft MVP
Location: Wageningen, The Netherlands

Re: Differential backups are forced in ShadowProtect SPX

Post by HansV »

Weird...
Best wishes,
Hans

User avatar
BobArch2
BronzeLounger
Posts: 1253
Joined: 25 Jan 2010, 22:25
Location: Pickering, Ontario, Canada

Re: Differential backups are forced in ShadowProtect SPX

Post by BobArch2 »

Just a fast update...

What we know ... if the incremental tracking record in SPX gets reset for reasons as noted below, the next scheduled incremental backup will force a differential.

1) a “dirty” or hard shutdown will reset the incremental tracking record.
2) the use of Fast Start will reset the incremental tracking record.
3) major updates to the Windows 10 operating system will reset the incremental tracking record.

The above 3 reasons have been known for awhile and there does not appear to be a remedy because vital information stored in the tracking record gets corrupted.

My testing over the last 1.5 weeks has added a 4th reason. I completed the testing last night and will be reporting my findings later today to StorageCraft support. But, my first allegiance is to Lounge members.

4) updating any application software which requires the system to be restarted to complete the installation will reset the incremental tracking record.

To be noted ... any updates to the operating system or application software that does NOT require a reboot, will not reset the incremental tracking record.

As stated, I will report my findings to StorageCraft and will post here when they reply.
Regards,
Bob

User avatar
HansV
Administrator
Posts: 79669
Joined: 16 Jan 2010, 00:14
Status: Microsoft MVP
Location: Wageningen, The Netherlands

Re: Differential backups are forced in ShadowProtect SPX

Post by HansV »

Thanks!
Best wishes,
Hans

User avatar
BobArch2
BronzeLounger
Posts: 1253
Joined: 25 Jan 2010, 22:25
Location: Pickering, Ontario, Canada

Re: Differential backups are forced in ShadowProtect SPX

Post by BobArch2 »

Some additional observations about differentials:

Yesterday, March 17th, 2020, I returned home after a 33 day stay in the hospital. (I had fallen and broken my upper right arm in 2 places. Typing this with my non-dominant left hand so please excuse any typos etc. :grin: )

My standard practice is to shut down my system on a daily basis. When I was taken to the hospital on February 14, my system was still powered up. After about 25 days in the hospital I suddenly realized that my system was never turned off. I used my phone to check the email account used to advise of backup notifications. Every single one of those notifications related to daily incrementals was less than 1.5 minutes. Meaning, not a single differential had taken place.

When I got home yesterday one of the tasks was to check the state of the system and activities automatically performed during the past 33 days.

1. SPX had automatically performed full weekly backups without any problems.
2. SPX had automatically performed daily incrementals without problems (no differentials)
3. SPX had automatically performed the full monthly backup successfully.
4. Windows 10 had automatically updated Defender definitions on a daily basis.
5. Windows 10 had automatically performed the monthly cumulative updates successfully.
6. Software updates to Office 365 and a couple of other apps were automatically updated successfully.

With all of the above performed over the 33 day period and not a single differential.

After checking all of the above, I performed about 6 software updates. I left the system powered up when I retired last night. A Normal incremental backup was successfully performed this morning.

Leading me to believe, that contrary to some of my comments in previous notes above, powering down the system under certain circumstances affects the incremental tracking record.

I just might leave my system on 24x7. :grin:
Regards,
Bob

User avatar
HansV
Administrator
Posts: 79669
Joined: 16 Jan 2010, 00:14
Status: Microsoft MVP
Location: Wageningen, The Netherlands

Re: Differential backups are forced in ShadowProtect SPX

Post by HansV »

Welcome back home!
Best wishes,
Hans