Azure VM’s and their Temporary Storage

Every virtual machine deployed in Azure automatically gets an operating system disk and a temporary (virtual hard) disk. As the temporary disk is labeled as the D:\ drive by default, it causes some confusion or problems when used for the wrong purpose. But what is the right purpose? Let me start by sharing some background information, why there is a need to have this temporary disk and how to make the best use of it.

The Temporary Storage

In Azure, every VM – regardless if Linux or Windows – gets a temporary disk assigned automatically. This temporary disk is located on the physical server (the hypervisor) where the Azure VM is hosted and is non-persistent. Disks used by the operating system or additionally added data disks are persistent disks and stored in Azure Storage.

Azure VM’s can be moved from its current host to new host at any time due to maintenance, hardware failures or other reasons. In such an event, the data from the temporary storage will not preserve or moved to the new host. Apart from the hardware failures, there are many other reasons data from the temporary disk will be lost:

  • Resizing of the VM
  • Restarting of the VM
  • Moving from one host to another
  • Updating/upgrading of host

Really, the temporary disk should never be used for data that has to be persistent. To avoid misconfiguration, the disk also has the drive label “Temporary Storage” and includes a text file “DATALOSS_WARNING_README.txt”. It couldn’t be more obvious…

Azure VM’s and their Temporary Storage

For Windows Server, the temporary disk is mounted as D:\. Linux based VM’s have the temporary disk mounted as “/dev/sdb1”. Of course, the same principles apply, your risk losing data that can’t be recovered when storing data on this disk.

The size of the temporary disk varies, based on the size of the virtual machine (and it’s available physical memory). For more information, see Sizes for Windows virtual machines. Look for the column “Temp storage (SSD) GiB” to understand how big the temporary disk will be. Good to know, the temporary storage provided with each Azure VM has no extra cost associated.

Memory Extension and the pagefile.sys

As the temporary disk can’t be used to store any persistent data, why should you care? Let’s take it a step further and have a look at the page file. Windows Memory Management is based on virtual memory, where every process has its own private virtual address space. In case the operating system runs low on memory, it will move the least used memory pages from the physical memory to a page file (pagefile.sys). This ensures the processes still have access to resources. This process is called “paging”. The main purpose of the page file is the following:

  • Extend the physical memory;
  • Store information in case of a system crash.

In the old days, sysadmins liked to tweak the size and location of this pagefile.sys. However, those old rules of thumb (e.g., page file size equals physical memory * 1.5) makes no sense anymore when using the modern operating system. Since Windows Server 2012 the Microsoft recommendation is to leave the option system managed. This still is a valid recommend practice, unless recommended different for a specific application. You can find some further reads in the Microsoft KB 2860880.

Good to know, newly created VM’s already have the pagefile.sys configured to be located on this temporary disk.

pagefile.sys for Azure VM’s

Putting it into the context of Azure – the temporary disk (D:\) is used by default to store the pagefile.sys. As applications often are installed on D:\ drive, this might cause some problems and change corporate policies might not be the firsts (or most comfortable) choice. To remap the temporary disk to a different drive letter, follow the instructions in this article Change the drive letter of the Windows temporary disk.

As most of the deployments these days are performed using ARM templates, it would be a good opportunity to extend the script to move the temporary disk already during the deployment. There is a project on GitHub (MoveAzureTempDrive) which is a great starting point. You only have to modify the azuredeploy.parameters.json file with the vmName and desired tempDriveLetter to be used.

There’s just one more thing…

The temporary storage drive is non-persisted storage, and you shouldn’t store any data on this drive. However, we always have this one exception…. SQL Server. When working with SQL Server, you can get significant improvements of I/O throughputs when storing the TempDB on the temporary disk – only the TempDB, no user database or transaction log files.

For SQL Server using D-series, Dv2-series, and G-series VM’s, the temporary disk is SSD-based. If the workload makes heavy use of TempDB, storing this file on the D:\ drive could result in higher throughput and lower latency. For VMs that support Premium Storage (DS-series, DSv2-series, and GS-series), it is recommended storing TempDB on a disk that supports Premium Storage with read caching enabled.

Tags

14 thoughts on “Azure VM’s and their Temporary Storage”

  1. Excellent article.

    I have a question, is there any possibility of recovering lost data, in temporary storage? creating a ticket, I lost valuable data this week, when I did a size reconfiguration from a VM A0 to an A1.

    Thank you!

    Sergio

    Reply
  2. My VM is being used as a webserver. I would like to put the IIS Temporary Cache and the ASP Compiled Templates on this drive. But the folder structure also gets wiped out. Is it possible to script the creation of those folders on system startup? And would that ensure those folders exist for all of the scenarios you list that cause data on this drive to be wiped out?

    Reply
    • Scripting this wouldn’t be a problem, but I would ensure to add some checks in case the script fails. Maybe check if you get such a performance benefit using this storage, do some tests to compare and staying in a “supported scenario” would always be my preference.

      Reply
  3. Would you recommend using the Temporary storage for TEMPDB on a virtual machine of size E16s v3?

    I’m getting quite a lot these messages in SQL Server log:
    “SQL Server has encountered x occurrences of I/O requests taking longer then 15 seconds to complete on” <— tempdb + dempdev files.

    Reply
  4. Is it really wise to use the default system managed pagefile on small azure machines?

    Just recently a problem with an insufficiently sized pagefile. It was system managed on the Temporary Storage D: drive which was a lousy 15GB. Combined with the hard limit of 1/8th of drive size this results in a mere 1.8GB of swapfile.

    Should be better to manually set it to a proper size.

    Reply
  5. Do Ephemeral OS disks use that same Temporary Drive or is it using a Separate Drive that are only available on certain VM’s that support ephemeral OS disks?

    Reason I ask is because I am using a WVD with Ephemeral OS disks but yet I still have a full sized temp drive which leads me to believe they are not the same.

    Reply
  6. Tempdb in SQLServer on the D-drive is almost a best-practice these days. But when I read the data on this disk is gone after a maintenance event (that does not always cause a restart of the vm!), I get worried. When a vm is moved by eg live-migration or an resize-operation, the disk may be emptied. And that will cause major issues in SQL Server. How is that prevented as you too mention that SQL Server is the one exception where it is OK to use the temp drive?!

    Reply
  7. Back in 2014 the best practice advice from Microsoft was to make sure the pagefile was stored on the temporary disk. In fact, it did that by default if you create it from Azure (rather than upload a VHD or migrate from on-prem).

    I can’t see any pagefile advice since Microsoft have taken away the temp disk on certain VM families. Should I be creating my own disk manually just to store the pagefile, or should I just leave it on the OS disk (e.g. C:\ drive). If I should be creating one then is there any advice on the type/size of disk to use? I’m not even sure that would be of benefit or cost-effective compared to just using the sizes that include the temp disk!

    Reply
  8. I have a VM where I had renamed the temporary drive to with letter T: following the official documentation and I enabled custom page filing for one of the drives G:. When I used Win32_PageFileUsage class, I got both T: and G: in responses with no different fields to differentiate the system managed one from custom one. Is there a way I can fetch the drive name of only the system managed page file drive? Any C# module or WMI class?

    Reply

Leave a Reply to Michel Luescher Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.