Quantcast
Channel: Jose Barreto's Blog
Viewing all 182 articles
Browse latest View live

PowerShell Examples: Adventure House Game

$
0
0

This blog is part of a series that shows example PowerShell code for those learning the language.

This time we’re using PowerShell create a text adventure game that requires to find your way out of a house using simple commands. There is a locked door and you need to find a key.

This example explores the use of arrays (an array of arrays, actually), string manipulation and the switch statement.

 

#
# Adventure House Game v2
#
#

[Array] $Rooms = (
#    Name                 Description                           N  S  E  W   U   D
(00, "Exit!",             "Exit!"                               ,99,99,99,01,99,99),
(01, "Entrance Hall",     "entrance hall. The door is locked"   ,10,02,99,99,99,99),
(02, "Downstairs Hall",   "hall at the bottom of the stairs"    ,01,04,03,99,11,99),
(03, "Guest Bathroom",    "small guest bathroom downstaris"     ,99,99,05,02,99,99),
(04, "Living Room",       "living room on the southeast side"   ,02,99,99,05,99,99),
(05, "Family Room",       "family room with a large TV"         ,06,99,02,99,99,99),
(06, "Nook",              "nook with a small dining table"      ,07,05,99,24,99,99),
(07, "Kitchen",           "kitchen with a large granite counter",08,06,99,99,99,99),
(08, "Trash Hall",        "small hall with two large trash cans",99,07,10,09,99,99),
(09, "Garage",            "garage, the big door is closed"      ,99,99,08,99,99,99),
(10, "Dining Room",       "dining room on the northeast side"   ,99,01,99,08,99,99),
(11, "Upstairs Hall",     "hall at the top of the stairs"       ,99,12,16,13,99,02),
(12, "Upper East Hall",   "hall with two tables and computers"  ,11,15,99,99,99,99),
(13, "Upper North Hall",  "hall with a large closet"            ,18,14,11,17,99,99),
(14, "Upper West Hall",   "hall with a small closet"            ,13,23,99,22,99,99),
(15, "Guest Room",        "guest room with a queen size bed"    ,12,99,99,99,99,99),
(16, "Laundry",           "laundry room with a washer and dryer",99,99,99,11,99,99),
(17, "Main Bathroom",     "main bathroom with a bathtub"        ,99,99,13,99,99,99),
(18, "Master Bedroom",    "master bedroom with a king size bed" ,21,13,19,99,99,99),
(19, "Master Closet",     "long and narrow walk-in closet"      ,99,99,99,18,20,99),
(20, "Attic",             "attic, it is dark in here"           ,99,99,99,99,99,19),
(21, "Master BathRoom",   "master bedroom with a shower and tub",99,18,99,99,99,99),
(22, "Children's Room",   "children's room with twin beds"      ,99,99,14,99,99,99),
(23, "Entertainment Room","play room with games and toys"       ,14,99,99,99,99,99),
(24, "Patio",             "wooden patio. A key on the floor"    ,99,99,06,99,99,99)

[Int] $Room = 20
[String] $Message = "Find the way out of this house."
[Array] $Inventory = ("BREAD","BUGLE")

While ($Room -ne 0) {
    $Name  = $Rooms[$Room][1]
    $Desc  = $Rooms[$Room][2]
    $North = $Rooms[$Room][3]
    $South = $Rooms[$Room][4]
    $East  = $Rooms[$Room][5]
    $West  = $Rooms[$Room][6]
    $Up    = $Rooms[$Room][7]
    $Down  = $Rooms[$Room][8] 

    $Available = "[Q]uit, [I]nventory, [L]ook, [A]pply"
    If ($North -ne 99) { $Available += ", [N]orth" }
    If ($South -ne 99) { $Available += ", [S]outh" }
    If ($East  -ne 99) { $Available += ", [E]ast" }
    If ($West  -ne 99) { $Available += ", [W]est" }
    If ($Up    -ne 99) { $Available += ", [U]p" }
    If ($Down  -ne 99) { $Available += ", [D]own" }
    If ($Room -eq 24 -and -not $Inventory.Contains("KEY")) { $Available += ", [K]ey" } 

    Clear-Host
    Write-Host $Message
    $Message="" 

    Write-Host
    Write-Host -ForegroundColor Yellow $Name
    Write-Host "You are at the $Desc"
    Write-Host

    Write-Host -ForegroundColor Green "Commands : $Available ? " -NoNewline
    $Cmd = Read-Host
    $Cmd = $Cmd.ToUpper()
    $Action = $Cmd[0]
    $Item = $Cmd.Split(" ")[1] 

    Switch ($Action) {
    "N" { If ($North -ne 99) { $Room = $North } }
    "S" { If ($South -ne 99) { $Room = $South } }
    "E" { If ($East  -ne 99) { $Room = $East  } }
    "W" { If ($West  -ne 99) { $Room = $West  } }
    "U" { If ($Up    -ne 99) { $Room = $Up    } }
    "D" { If ($Down  -ne 99) { $Room = $Down  } }

    "K" { If ($inventory.Contains("KEY")) {
             $Message = "You already have the key"
          } ElseIf ($Room -eq 24) {
             $Message = "You grabbed the key."
             $Rooms[24][2] = "wooden patio on the west side."
             $Inventory += "KEY"
          } Else {
            $Message = "I don't see a key here."
          } #end if
        } #end "K"

    "I" { $Message = "You have : "
            0..($Inventory.Length - 2) | % { $Message += $Inventory[$_] + ", " }
            $Message += $Inventory[$Inventory.Length - 1]
        }

    "L" { if ($item -eq "" -or $item -eq $null) { $Message = "Look what?" }
          elseif (-not $inventory.contains($item)) { $Message = "There isn't any '$Item' in your inventory." }
          else { Switch ($Item) {
                  "KEY"   { $Message = "A shiny, aesthetically pleasing key. Must open something, right?" }
                  "BUGLE" { $Message = "You were never very good with instruments." }
                  "BREAD" { $Message = "A small loaf of bread. Not quite a lunch, but way too big for a snack. Quite a predicament, really." }
                  default { $Message = "You have no clue what $item is." }
                 } #end switch $item
          } #end if
        } #end "X"

    "A" { if ($item -eq "" -or $item -eq $null) { $Message = "Apply what?" }
          elseif (-not $inventory.contains($item)) { $Message = "There isn't any '$Item' in your inventory." }
          else { Switch ($Item) {
                 "KEY"   { if ($Room -ne 01) { $Message = "The key doesn't fit anywhere here." }
                           elseif ($Rooms[01][5] -eq 99 ) {
                               $Message = "The key fits perfectly and the door unlocked with some effort."
                               $Rooms[01][2] = "hall by the entrance. The key unlocked the door."
                               $Rooms[01][5] = 0 }
                           else { $Message = "You already unlocked it just a moment ago." }
                          }
                 "BUGLE" { $Message = "You try to no avail to produce something that could constitute music." }
                 "BREAD" { $Message = "It's too big for a snack. Maybe later, for lunch." }
                 default { $Message = "You have no idea what to do with $item." }
                 } #end switch $item
          } #end if
        } #end "A"

    "Q" { $Room = 0 }

     } #end switch $Action
} #end while

if ($Action -ne "Q") { Write-Host "You found the way out. Congratulations!" }
                else { Write-Host "Better luck next time..." }

 


PowerShell Examples – Random words and their popularity via Bing

$
0
0

This blog is part of a series that shows example PowerShell code for those learning the language.

This time we’re using PowerShell to generate random words and check if they are popular by using a Bing search. The words are color-coded as white (not found on the web), green (between 1 and 1,000 hits on the web), yellow (between 1,000 and 1,000,000 hits) and red (more than 1,000,000 hits). This could be useful if you need to create a username on a popular website and everything you can think of is already taken :-).

This example explores using the Internet Explorer objects and searching through web pages by ID, along with more common items like arrays, string manipulation and functions.

 

#
# Random words and their popularity with Bing
#

#
# Defines array with common vowels, consonants and endings
#

[array] $Vowels = "a;a;a;a;e;e;e;e;i;i;i;o;o;o;u;u;y" -split ";"
[array] $Consonants = "b;b;br;c;c;c;ch;cr;d;f;g;h;j;k;l;m;m;m;n;n;p;p;ph;qu;r;r;r;s;s;s;sh;t;tr;v;w;x;z" -split ";"
[array] $Endings = "r;r;s;r;l;n;n;n;c;c;t;p" -split ";"

#
# Functions for random vowels, consonants, endings and words
#

function Get-RandomVowel
{ return $Vowels[(Get-Random($Vowels.Length))] }

function Get-RandomConsonant
{ return $Consonants[(Get-Random($Consonants.Length))] }

function Get-RandomEnding
{ return $Endings[(Get-Random($Endings.Length))] }

function Get-RandomSyllable ([int32] $PercentConsonants, [int32] $PercentEndings)

   [string] $Syllable = ""
   if ((Get-Random(100)) -le $PercentConsonants)
   { $Syllable+= Get-RandomConsonant }
   $Syllable+= Get-RandomVowel
   if ((Get-Random(100)) -le $PercentEndings)
   { $Syllable+= Get-RandomEnding }
   return $Syllable
}

function Get-RandomWord ([int32] $MinSyllables, [int32] $MaxSyllables)

   [string] $Word = ""
   [int32] $Syllables = ($MinSyllables) + (Get-Random(($MaxSyllables - $MinSyllables + 1)))
   for ([int32] $Count=1; $Count -le $Syllables; $Count++)
   { $Word += Get-RandomSyllable 70 20 } <# Consonant 70% of the time, Ending 20% #>
   return $Word
}

#
# Function to see how many pages Bing finds for a given term
#

Function Get-BingCount([string] $Term) {

    # Navigate to the Bing page to query the $term
    $ie.Navigate("
http://bing.com/search?q=%2B"+$term);

    # Wait for the page to load
    $timeout = 0
    while ($ie.Busy) {
        # Write-Host "Waiting for Bing page for $term to load"
        Start-Sleep -Milliseconds 100
        $timeout++
        If ($timeout  -gt 100) {
            return "L-Error"
        }
    }    

    # Wait for the document to be ready
    $timeout = 0
    $element1 = $ie.Document.IHTMLDocument3_getElementById("b_tween").innertext
    $element2 = $ie.Document.IHTMLDocument3_getElementById("b_content").innertext
    While ($element1 -eq $null -and $element2 -eq $null) {
        # Write-Host "Waiting for Bing document for $term to be ready"
        Start-Sleep -Milliseconds 100
        $timeout++
        If ($timeout  -gt 100) {
            return "D-Error"
        }
        $element1 = $ie.Document.IHTMLDocument3_getElementById("b_tween").innertext
        $element2 = $ie.Document.IHTMLDocument3_getElementById("b_content").innertext
    }

    # Get the count of pages
    If ($element1 -ne $null) { $result = $element1.split(" ")[0] }
                       else  { $result = "0" }
    # Return the count
    return $result
}

#
# Main code
#

# Create Internet Explorer object
$ie = New-Object -ComObject "InternetExplorer.Application"     

# Show initial message
Write-Host
Write-Host "Here are 20 random words and their popularity"
Write-Host

1..20 | % {
    # Get a random word
    $word = Get-RandomWord 2 5

    # Check the popularity with Bing
    $count = ([string] (Get-BingCount $word)).Padleft(12)
    $countint = [int] $count

    # Select Color based on popularity.
    if     ($countint -eq 0)       { $color = "white"  }
    elseif ($countint -lt 1000)    { $color = "green"  }
    elseif ($countint -lt 1000000) { $color = "yellow" }
    else                           { $color = "red"    } 

    # Write the info with the right color
    Write-Host "$count --> $word" -ForegroundColor $color
}

# Quit Internet Explorer
$ie.quit();

 

In case you were wondering what the output would look like, here is a sample:

image

PowerShell Examples: Using Bing to measure the popularity of cmdlets in a specific PowerShell module

$
0
0

This blog is part of a series that shows example PowerShell code for those learning the language.

This time we’re using PowerShell to find out which PowerShell cmdlet in a specific module is the most popular. To do this, we’re using a Bing search and producing a list with the name and number of web pages found for each one. This is returned as an object, so you can pipe the output to sort or filter. The name of the module is hard coded to “SmbShare”, but you can replace it with any valid module name. You could also make a parameter.

This example explores using the Web client in the .NET System.Web assembly, searching through the resulting web page using string functions, showing cmdlet progress and also a trick to create an object on the fly. It also shows more common items like arrays and functions.

 

#
# Popularity (via Bing) of the PowerShell cmdlets in a module
#

#
# Function to see how many pages Bing finds for a given term
#

# Adds assembly to support URL encoding and the web client

Add-Type -Assembly System.Web
$WebClient = New-Object system.Net.WebClient

Function Get-BingCount([string] $Term) {

    # Add plus and quotes, encodes for URL
    $Term = '+"' + $Term + '"'
    $Term = [System.Web.HttpUtility]::UrlEncode($Term)

    # Load the page as a string
    $URL = "http://www.bing.com/search?q=" + $Term
    $Page = $WebClient.DownloadString($URL)

    # searches for the string before the number of hits on the page
    $String1 = '<span class="sb_count">'
    $Index1 = $Page.IndexOf($String1)

    # if found the right string, finds the exact end of the number
    If ($Index1 -ne -1) {
        $Index1 += $String1.Length
        $Index2 = $Page.IndexOf(" ", $Index1)
        $result = $Page.Substring($Index1, $Index2 - $index1)
    } else { $result = "0" }

    # Return the count
    return $result
}

#
# Main code
#

$CmdletList = Get-Command -Module SmbShare | Select Name
$CmdletCount = $CmdletList.Count -1

0..$CmdletCount | % {

    # Tracks progress
    Write-Progress -Activity "Checking cmdlet popularity" -PercentComplete ($_ * 100 / $CmdletCount)

    # Check the popularity with Bing
    $cmdlet = $CmdletList[$_].Name
    $count = [int] (Get-BingCount $cmdlet)

    # Format as a row, output it
    $Row = "" | Select CmdletName, BingCount
    $Row.CmdletName  = $cmdlet
    $Row.BingCount = $count
    $Row
}

Write-Progress -Activity "Checking cmdlet popularity" -Completed

# Releases resources used by the web client
$WebClient.Dispose()

 

In case you were wondering what the output would look like, here is a sample:

CmdletPopularity

Storage Sessions at Microsoft Ignite. Make sure to update your schedule!

$
0
0

Hi!

If you’re planning to attend Ignite, here is a list of sessions related to Storage at the event:

Session titlePresenters
Deploying Highly Scalable Clusters with Dell Servers and Microsoft StorageClaus Joergensen, Shai Ofek, Syama Poluri
Enabling New On-premises Scale-Out File Server with Direct-Attached StorageClaus Joergensen, Michael Gray
Exploring Storage Replica in Windows Server vNextNed Pyle
Hyper-V Storage Performance with Storage Quality of ServiceJose Barreto, Senthil Rajaram
Managing Storage with System Center Virtual Machine Manager: A Deep DiveHector Linares
Overview of Microsoft Azure Storage and Key Usage ScenariosVamshidhar Kommineni
Overview of the Microsoft Cloud Platform SystemVijay Tewari, Wassim Fayed
Platform Vision & Strategy (4 of 7): Storage OverviewJose Barreto, Siddhartha Roy
Spaces-Based, Software-Defined Storage: Design and Configuration Best PracticesAllen Stewart, Jason Gerend, Joshua Adams
StorSimple: Extending Your Datacenter into Microsoft Azure with Hybrid Cloud StorageBadri Venkatachari, Meghan Liese
Stretching Failover Clusters and Using Storage Replica in Windows Server vNextElden Christensen, Ned Pyle
System Center Virtual Machine Manager: Technical Overview and RoadmapEric Winner, Jonobie Ford
The Power of the Windows Server Software Defined datacenter in actionPhillip Moss
Upgrading your private cloud to Windows Server 2012 R2 and beyond!Ben Armstrong, Rob Hindman

There are other sessions on Storage and other sessions on Windows Server, but this list tries to cover the intersection of the two, plus some bonus Azure and Hybrid sessions. My apologies in advance if I missed anything and feel free to add comments about other sessions that you’re planning to attend at Ignite.

With hundreds of sessions in the event, I suggest that you get started building your schedule…

Windows PowerShell equivalents for common networking commands (IPCONFIG, PING, NSLOOKUP)

$
0
0

Network troubleshooting is part any System Administrator’s life. Maybe you need to check the IP address of a machine or test if its networking connection is working. Maybe you need to see if DNS is properly configured or check the latency between two hosts.

If you have been in this field long enough, you probably have a few favorite commands that you learned years ago and use on a regular basis, like IPCONFIG or PING.

There are literally hundreds of networking-related PowerShell cmdlets in Windows these days. Just try out this command on your machine: Get-Command -Module Net* | Group Module

But more important than knowing every one of them, is to know the most useful cmdlets that have the potential to replace those old commands that you can’t live without.

And it’s when you combine the many networking cmdlets in ways that only PowerShell can do that you’ll find amazing new troubleshooting abilities…

 

IPCONFIG

Description: This command has many options, but the most common usage is just to show the IP address, subnet mask and default gateway for each network adapter in a machine.

PowerShell: Get-NetIPConfiguration or Get-NetIPAddress

Sample command lines:

  • Get-NetIPConfiguration
  • Get-NetIPAddress | Sort InterfaceIndex | FT InterfaceIndex, InterfaceAlias, AddressFamily, IPAddress, PrefixLength -Autosize
  • Get-NetIPAddress | ? AddressFamily -eq IPv4 | FT –AutoSize
  • Get-NetAdapter Wi-Fi | Get-NetIPAddress | FT -AutoSize

Sample output:

PS C:\> Get-NetIPConfiguration

InterfaceAlias       : Wi-Fi
InterfaceIndex       : 3
InterfaceDescription : Dell Wireless 1703 802.11b|g|n (2.4GHz)
NetProfile.Name      : HomeWifi
IPv6Address          : fded:b22c:44c4:1:88f2:9970:4082:4118
IPv4Address          : 192.168.1.2
IPv6DefaultGateway   :
IPv4DefaultGateway   : 192.168.1.1
DNSServer            : 192.168.1.1

InterfaceAlias       : Bluetooth Network Connection
InterfaceIndex       : 6
InterfaceDescription : Bluetooth Device (Personal Area Network)
NetAdapter.Status    : Disconnected

InterfaceAlias       : Ethernet
InterfaceIndex       : 4
InterfaceDescription : Realtek PCIe GBE Family Controller
NetAdapter.Status    : Disconnected

PS C:\> Get-NetIPAddress | Sort InterfaceIndex | FT InterfaceIndex, InterfaceAlias, AddressFamily, IPAddress, PrefixLength –Autosize

InterfaceIndex InterfaceAlias                                AddressFamily IPAddress                            PrefixLength
-------------- --------------                                ------------- ---------                            -------
             1 Loopback Pseudo-Interface 1                            IPv6 ::1                                      128
             1 Loopback Pseudo-Interface 1                            IPv4 127.0.0.1                                  8
             3 Wi-Fi                                                  IPv6 fe80::88f2:9970:4082:4118%3               64
             3 Wi-Fi                                                  IPv6 fded:b22c:44c4:1:f188:1e45:58e3:9242     128
             3 Wi-Fi                                                  IPv6 fded:b22c:44c4:1:88f2:9970:4082:4118      64
             3 Wi-Fi                                                  IPv4 192.168.1.2                               24
             4 Ethernet                                               IPv6 fe80::ce6:97c9:ae58:b393%4                64
             4 Ethernet                                               IPv4 169.254.179.147                           16
             6 Bluetooth Network Connection                           IPv6 fe80::2884:6750:b46b:cec4%6               64
             6 Bluetooth Network Connection                           IPv4 169.254.206.196                           16
             7 Local Area Connection* 3                               IPv6 fe80::f11f:1051:2f3d:882%7                64
             7 Local Area Connection* 3                               IPv4 169.254.8.130                             16
             8 Teredo Tunneling Pseudo-Interface                      IPv6 2001:0:5ef5:79fd:1091:f90:e7e9:62f0       64
             8 Teredo Tunneling Pseudo-Interface                      IPv6 fe80::1091:f90:e7e9:62f0%8                64
             9 isatap.{024820F0-C990-475F-890B-B42EA24003F1}          IPv6 fe80::5efe:192.168.1.2%9                 128

PS C:\> Get-NetIPAddress | ? AddressFamily -eq IPv4 | FT –AutoSize

ifIndex IPAddress       PrefixLength PrefixOrigin SuffixOrigin AddressState PolicyStore
------- ---------       ------------ ------------ ------------ ------------ -----------
7       169.254.8.130             16 WellKnown    Link         Tentative    ActiveStore
6       169.254.206.196           16 WellKnown    Link         Tentative    ActiveStore
3       192.168.1.2               24 Dhcp         Dhcp         Preferred    ActiveStore
1       127.0.0.1                  8 WellKnown    WellKnown    Preferred    ActiveStore
4       169.254.179.147           16 WellKnown    Link         Tentative    ActiveStore

PS C:\> Get-NetAdapter Wi-Fi | Get-NetIPAddress | FT -AutoSize

ifIndex IPAddress                            PrefixLength PrefixOrigin        SuffixOrigin AddressState PolicyStore
------- ---------                            ------------ ------------        ------------ ------------ -----------
3       fe80::88f2:9970:4082:4118%3                    64 WellKnown           Link         Preferred    ActiveStore
3       fded:b22c:44c4:1:f188:1e45:58e3:9242          128 RouterAdvertisement Random       Preferred    ActiveStore
3       fded:b22c:44c4:1:88f2:9970:4082:4118           64 RouterAdvertisement Link         Preferred    ActiveStore
3       192.168.1.2                                    24 Dhcp                Dhcp         Preferred    ActiveStore

 

PING

Description: Checks connectivity to a specific host. Commonly used to check for liveliness, but also used to measure network latency.

PowerShell: Test-NetConnection

Sample command lines:

  • Test-NetConnection www.microsoft.com
  • Test-NetConnection -ComputerName www.microsoft.com -InformationLevel Detailed
  • Test-NetConnection -ComputerName www.microsoft.com | Select -ExpandProperty PingReplyDetails | FT Address, Status, RoundTripTime
  • 1..10 | % { Test-NetConnection -ComputerName www.microsoft.com -RemotePort 80 } | FT -AutoSize

Sample output

PS C:\> Test-NetConnection www.microsoft.com

ComputerName           : www.microsoft.com
RemoteAddress          : 104.66.197.237
InterfaceAlias         : Wi-Fi
SourceAddress          : 192.168.1.2
PingSucceeded          : True
PingReplyDetails (RTT) : 22 ms

PS C:\> Test-NetConnection -ComputerName www.microsoft.com -InformationLevel Detailed

ComputerName             : www.microsoft.com
RemoteAddress            : 104.66.197.237
AllNameResolutionResults : 104.66.197.237
                           2600:1409:a:396::2768
                           2600:1409:a:39b::2768
InterfaceAlias           : Wi-Fi
SourceAddress            : 192.168.1.2
NetRoute (NextHop)       : 192.168.1.1
PingSucceeded            : True
PingReplyDetails (RTT)   : 14 ms

PS C:\> Test-NetConnection -ComputerName www.microsoft.com | Select -ExpandProperty PingReplyDetails | FT Address, Status, RoundTripTime -Autosize

Address         Status RoundtripTime
-------         ------ -------------
104.66.197.237 Success            22

PS C:\> 1..10 | % { Test-NetConnection -ComputerName www.microsoft.com -RemotePort 80 } | FT -AutoSize

ComputerName      RemotePort RemoteAddress  PingSucceeded PingReplyDetails (RTT) TcpTestSucceeded
------------      ---------- -------------  ------------- ---------------------- ----------------
www.microsoft.com 80         104.66.197.237 True          17 ms                  True
www.microsoft.com 80         104.66.197.237 True          16 ms                  True
www.microsoft.com 80         104.66.197.237 True          15 ms                  True
www.microsoft.com 80         104.66.197.237 True          18 ms                  True
www.microsoft.com 80         104.66.197.237 True          20 ms                  True
www.microsoft.com 80         104.66.197.237 True          20 ms                  True
www.microsoft.com 80         104.66.197.237 True          20 ms                  True
www.microsoft.com 80         104.66.197.237 True          20 ms                  True
www.microsoft.com 80         104.66.197.237 True          15 ms                  True
www.microsoft.com 80         104.66.197.237 True          13 ms                  True

 

NSLOOKUP

Description: Name server lookup. Mostly used to find the IP address for a given DNS name (or vice-versa). Has many, many options.

PowerShell: Resolve-DnsName

Sample command lines:

  • Resolve-DnsName www.microsoft.com
  • Resolve-DnsName microsoft.com -type SOA
  • Resolve-DnsName microsoft.com -Server 8.8.8.8 –Type A

Sample output

PS C:\> Resolve-DnsName www.microsoft.com

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
www.microsoft.com              CNAME  6     Answer     toggle.www.ms.akadns.net
toggle.www.ms.akadns.net       CNAME  6     Answer     www.microsoft.com-c.edgekey.net
www.microsoft.com-c.edgekey.ne CNAME  6     Answer     www.microsoft.com-c.edgekey.net.globalredir.akadns.net
t
www.microsoft.com-c.edgekey.ne CNAME  6     Answer     e10088.dspb.akamaiedge.net
t.globalredir.akadns.net

Name       : e10088.dspb.akamaiedge.net
QueryType  : AAAA
TTL        : 6
Section    : Answer
IP6Address : 2600:1409:a:39b::2768

Name       : e10088.dspb.akamaiedge.net
QueryType  : AAAA
TTL        : 6
Section    : Answer
IP6Address : 2600:1409:a:396::2768

Name       : e10088.dspb.akamaiedge.net
QueryType  : A
TTL        : 6
Section    : Answer
IP4Address : 104.66.197.237

PS C:\> Resolve-DnsName microsoft.com -type SOA

Name                        Type TTL   Section    PrimaryServer               NameAdministrator           SerialNumber
----                        ---- ---   -------    -------------               -----------------           ------------
microsoft.com               SOA  2976  Answer     ns1.msft.net                msnhst.microsoft.com        2015041801

PS C:\> Resolve-DnsName microsoft.com -Server 8.8.8.8 –Type A

Name                                           Type   TTL   Section    IPAddress
----                                           ----   ---   -------    ---------
microsoft.com                                  A      1244  Answer     134.170.188.221
microsoft.com                                  A      1244  Answer     134.170.185.46

 

ROUTE

Description: Shows the IP routes in a given system (also used to add and delete routes)

PowerShell: Get-NetRoute (also New-NetRoute and Remove-NetRoute)

Sample command lines:

  • Get-NetRoute -Protocol Local -DestinationPrefix 192.168*
  • Get-NetAdapter Wi-Fi | Get-NetRoute

Sample output:

PS C:\WINDOWS\system32> Get-NetRoute -Protocol Local -DestinationPrefix 192.168*

ifIndex DestinationPrefix NextHop RouteMetric PolicyStore
------- ----------------- ------- ----------- -----------
2       192.168.1.255/32  0.0.0.0         256 ActiveStore
2       192.168.1.5/32    0.0.0.0         256 ActiveStore
2       192.168.1.0/24    0.0.0.0         256 ActiveStore

PS C:\WINDOWS\system32> Get-NetAdapter Wi-Fi | Get-NetRoute

ifIndex DestinationPrefix                        NextHop     RouteMetric PolicyStore
------- -----------------                        -------     ----------- -----------
2       255.255.255.255/32                       0.0.0.0             256 ActiveStore
2       224.0.0.0/4                              0.0.0.0             256 ActiveStore
2       192.168.1.255/32                         0.0.0.0             256 ActiveStore
2       192.168.1.5/32                           0.0.0.0             256 ActiveStore
2       192.168.1.0/24                           0.0.0.0             256 ActiveStore
2       0.0.0.0/0                                192.168.1.1           0 ActiveStore
2       ff00::/8                                 ::                  256 ActiveStore
2       fe80::d1b9:9258:1fa:33e9/128             ::                  256 ActiveStore
2       fe80::/64                                ::                  256 ActiveStore
2       fded:b22c:44c4:1:d1b9:9258:1fa:33e9/128  ::                  256 ActiveStore
2       fded:b22c:44c4:1:c025:aa72:9331:442/128  ::                  256 ActiveStore
2       fded:b22c:44c4:1::/64                    ::                  256 ActiveStore

 

TRACERT

Description: Trace route. Shows the IP route to a host, including all the hops between your computer and that host.

PowerShell: Test-NetConnection –TraceRoute

Sample command lines:

  • Test-NetConnection www.microsoft.com –TraceRoute
  • Test-NetConnection outlook.com -TraceRoute | Select -ExpandProperty TraceRoute | % { Resolve-DnsName $_ -type PTR -ErrorAction SilentlyContinue }

Sample output:

PS C:\> Test-NetConnection www.microsoft.com–TraceRoute

ComputerName           : www.microsoft.com
RemoteAddress          : 104.66.197.237
InterfaceAlias         : Wi-Fi
SourceAddress          : 192.168.1.2
PingSucceeded          : True
PingReplyDetails (RTT) : 16 ms
TraceRoute             : 192.168.1.1
                         10.0.0.1
                         TimedOut
                         68.86.113.181
                         69.139.164.2
                         68.85.240.94
                         68.86.93.165
                         68.86.83.126
                         104.66.197.237

PS C:\> Test-NetConnection outlook.com -TraceRoute | Select -ExpandProperty TraceRoute | % { Resolve-DnsName $_ -type PTR -ErrorAction SilentlyContinue }

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
125.144.85.68.in-addr.arpa     PTR    7200  Answer     te-0-1-0-10-sur02.bellevue.wa.seattle.comcast.net
142.96.86.68.in-addr.arpa      PTR    4164  Answer     be-1-sur03.bellevue.wa.seattle.comcast.net
6.164.139.69.in-addr.arpa      PTR    2469  Answer     be-40-ar01.seattle.wa.seattle.comcast.net
165.93.86.68.in-addr.arpa      PTR    4505  Answer     be-33650-cr02.seattle.wa.ibone.comcast.net
178.56.167.173.in-addr.arpa    PTR    7200  Answer     as8075-1-c.seattle.wa.ibone.comcast.net
248.82.234.191.in-addr.arpa    PTR    3600  Answer     ae11-0.co2-96c-1a.ntwk.msn.net

 

NETSTAT

Description: Shows current TCP/IP network connections.

PowerShell: Get-NetTCPConnection

Sample command lines:

  • Get-NetTCPConnection | Group State, RemotePort | Sort Count | FT Count, Name –Autosize
  • Get-NetTCPConnection | ? State -eq Established | FT –Autosize
  • Get-NetTCPConnection | ? State -eq Established | ? RemoteAddress -notlike 127* | % { $_; Resolve-DnsName $_.RemoteAddress -type PTR -ErrorAction SilentlyContinue }

Sample output:

PS C:\> Get-NetTCPConnection | Group State, RemotePort | Sort Count | FT Count, Name -Autosize

Count Name
----- ----
    1 SynSent, 9100
    1 Established, 40028
    1 Established, 65001
    1 Established, 27015
    1 Established, 5223
    1 Established, 49227
    1 Established, 49157
    1 Established, 49156
    1 Established, 12350
    1 Established, 49200
    2 Established, 5354
    2 TimeWait, 5357
    2 Established, 80
    3 Established, 443
   36 Listen, 0

PS C:\> Get-NetTCPConnection | ? State -eq Established | FT -Autosize

LocalAddress LocalPort RemoteAddress   RemotePort State       AppliedSetting
------------ --------- -------------   ---------- -----       --------------
127.0.0.1    65001     127.0.0.1       49200      Established Internet
192.168.1.2  59619     91.190.218.57   12350      Established Internet
192.168.1.2  57993     213.199.179.175 40028      Established Internet
192.168.1.2  54334     17.158.28.49    443        Established Internet
192.168.1.2  54320     96.17.8.170     80         Established Internet
192.168.1.2  54319     23.3.105.144    80         Established Internet
192.168.1.2  54147     65.55.68.119    443        Established Internet
192.168.1.2  49257     17.143.162.214  5223       Established Internet
127.0.0.1    49227     127.0.0.1       27015      Established Internet
127.0.0.1    49200     127.0.0.1       65001      Established Internet
192.168.1.2  49197     157.56.98.92    443        Established Internet
127.0.0.1    49157     127.0.0.1       5354       Established Internet
127.0.0.1    49156     127.0.0.1       5354       Established Internet
127.0.0.1    27015     127.0.0.1       49227      Established Internet
127.0.0.1    5354      127.0.0.1       49157      Established Internet
127.0.0.1    5354      127.0.0.1       49156      Established Internet

PS C:\> Get-NetTCPConnection | ? State -eq Established | ? RemoteAddress -notlike 127* | % { $_; Resolve-DnsName $_.RemoteAddress -type PTR -ErrorAction SilentlyContinue }

LocalAddress                        LocalPort RemoteAddress                       RemotePort State       AppliedSetting
------------                        --------- -------------                       ---------- -----       --------------
192.168.1.2                         59619     91.190.218.57                       12350      Established Internet
192.168.1.2                         57993     213.199.179.175                     40028      Established Internet
192.168.1.2                         54334     17.158.28.49                        443        Established Internet
192.168.1.2                         54320     96.17.8.170                         80         Established Internet

Name      : 170.8.17.96.in-addr.arpa
QueryType : PTR
TTL       : 86377
Section   : Answer
NameHost  : a96-17-8-170.deploy.akamaitechnologies.com

192.168.1.2                         54319     23.3.105.144                        80         Established Internet

Name      : 144.105.3.23.in-addr.arpa
QueryType : PTR
TTL       : 7
Section   : Answer
NameHost  : a23-3-105-144.deploy.static.akamaitechnologies.com

192.168.1.2                         54147     65.55.68.119                        443        Established Internet

Name      : 119.68.55.65.in-addr.arpa
QueryType : PTR
TTL       : 850
Section   : Answer
NameHost  : snt404-m.hotmail.com

192.168.1.2                         49257     17.143.162.214                      5223       Established Internet

192.168.1.2                         49197     157.56.98.92                        443        Established Internet

Name      : 92.98.56.157.in-addr.arpa
QueryType : PTR
TTL       : 3600
Section   : Answer
NameHost  : bn1wns1011516.wns.windows.com

Note: Including a PDF version of the output in case you can't see it too well on the web with the lines wrapping and all. See below.

The Deprecation of SMB1 – You should be planning to get rid of this old SMB dialect

$
0
0

I regularly get a question about when will SMB1 be completely removed from Windows. This blog post summarizes the current state of this old SMB dialect in Windows client and server.

 

1) SMB1 is deprecated, but not yet removed

We already added SMB1 to the Windows Server 2012 R2 deprecation list in June 2013. That does not mean it’s fully removed, but that the feature is “planned for potential removal in subsequent releases”. You can find the Windows Server 2012 R2 deprecation list at https://technet.microsoft.com/en-us/library/dn303411.aspx.

 

2) Windows Server 2003 is going away

The last supported Windows operating system that can only negotiate SMB1 is Windows Server 2003. All other currently supported Windows operating systems (client and server) are able to negotiate SMB2 or higher. Windows Server 2003 support will end on July 14 of this year, as you probably heard.

 

3) SMB versions in current releases of Windows and Windows Server

Aside from Windows Server 2003, all other versions of Windows (client and server) support newer versions of SMB:

  • Windows Server 2008 or Windows Vista – SMB1 or SMB2
  • Windows Server 2008 R2 or Windows 7 – SMB1 or SMB2
  • Windows Server 2012 and Windows 8 – SMB1, SMB2 or SMB3
  • Windows Server 2012 R2 and Windows 8.1 – SMB1, SMB2 or SMB3

For details on specific dialects and how they are negotiated, see this blog post on SMB dialects and Windows versions.

image 

4) SMB1 removal in Windows Server 2012 R2 and Windows 8.1

In Windows Server 2012 R2 and Windows 8.1, we made SMB1 an optional component that can be completely removed. That optional component is enabled by default, but a system administrator now has the option to completely disable it. For more details, see this blog post on how to completely remove SMB1 in Windows Server 2012 R2.

 

5) SMB1 removal in Windows 10 Technical Preview and Windows Server Technical Preview

SMB1 will continue to be an optional component enabled by default with Windows 10, which is scheduled to be released in 2015. The next version of Windows Server, which is expected in 2016, will also likely continue to have SMB as an optional component enabled by default. In that release we will add an option to audit SMB1 usage, so IT Administrators can assess if they can disable SMB1 on their own.

 

6) What you should be doing about SMB1

If you are a systems administrator and you manage IT infrastructure that relies on SMB1, you should prepare to remove SMB1.  Once Windows Server 2003 is gone, the main concern will be third party software or hardware like printers, scanners, NAS devices and WAN accelerators. You should make sure that any new software and hardware that requires the SMB protocol is able to negotiate newer versions (at least SMB2, preferably SMB3). For existing devices and software that only support SMB1, you should contact the manufacturer for updates to support the newer dialects.

If you are a software or hardware manufacturer that has a dependency on the SMB1 protocol, you should have a clear plan for removing any such dependencies. Your hardware or software should be ready to operate in an environment where Windows clients and servers only support SMB2 or SMB3. While it’s true that today SMB1 still works in most environments, the fact that the feature is deprecated is a warning that it could go away at any time.

 

7) Complete removal of SMB1

Since SMB1 is a deprecated component, we will assess for its complete removal with every new release.

SMB3 Networking Links for Windows Server 2012 R2

$
0
0

Someone asked me for a good collection of links about SMB3 Networking, including SMB Multichannel, SMB Direct (SMB over RDMA) and SMB Performance.
Here’s what I offered:

 

SMB Multichannel

 

SMB Direct

 

SMB Performance

 

Performance White Papers

 

SMB Encryption

 

SMB Signing

What’s new in SMB 3.1.1 in the Windows Server 2016 Technical Preview 2

$
0
0

 

1. Introduction

Every new version of Windows brings updates to our main remote file protocol, known as SMB (Server Message Block).

If you’re not familiar with it, you can find some information in this previous blog post: Windows Server 2012 R2: Which version of the SMB protocol (SMB 1.0, SMB 2.0, SMB 2.1, SMB 3.0 or SMB 3.02) are you using?

In this blog post, you’ll see what changed with the new version of SMB that comes with the Windows 10 Insider Preview released in late April 2015 and the Windows Server 2016 Technical Preview 2 released in early May 2015.

 

2. Protocols Changes in SMB 3.1.1

This section covers changes in SMB 3.1.1 related to the protocol itself.

The Protocol Preview document fully describes these changes: [MS-SMB2-Diff]- Server Message Block (SMB) Protocol Versions 2 and 3, but you can see the highlights below.

 

2.1. Pre-Authentication Integrity

Pre-authentication integrity provides improved protection from a man-in-the-middle attacker tampering with SMB’s connection establishment and authentication messages.

Pre-Auth integrity verifies all the “negotiate” and “session setup” exchanges used by SMB with a strong cryptographic hash (SHA-512).

If your client and your server establish an SMB 3.1.1 session, you can be sure that no one has tampered with the connection and session properties.

Using SMB signing on top of an SMB 3.1.1 session protects you from an attacker tampering with any packets.

Using SMB encryption on top of an SMB 3.1.1 session protects you from an attacker tampering with or eavesdropping on any packets.

Although there is a cost to enable SMB signing or SMB encryption, we highly recommend enabling one of them.

Note: While these changes improve overall security, they might interfere with some solutions that rely on modifying SMB network traffic, like certain kinds of WAN accelerators.

 

2.2. SMB Encryption Improvements

SMB Encryption, introduced with SMB 3.0, used a fixed cryptographic algorithm: AES-128-CCM.

Since then, we have learned that AES-128-GCM performs better in most modern processors.

To take advantage of that, SMB 3.1.1 offers a mechanism to negotiate the crypto algorithm per connection, with options for AES-128-CCM and AES-128-GCM.

We made AES-128-GCM the default for new Windows versions, while older versions will continue to use AES-128-CCM.

With this flexible infrastructure for negotiation in place, we could add more algorithms in the future.

We observed that moving from AES-128-CCM to AES-128-GCM showed a 2x improvement in certain scenarios, like copying large files over an encrypted SMB connection.

 

2.3. Cluster Dialect Fencing

Provides support for cluster rolling upgrade for Scale-Out File Servers. For details, see http://technet.microsoft.com/en-us/library/dn765474.aspx#BKMK_RollingUpgrade

In this new scenario, a single SMB server appears to support different maximum dialects of SMB, depending on whether the SMB client is accessing clustered or non-clustered file shares.

For local, non-clustered file shares, the server offers up to 3.1.1 during dialect negotiation.

For clustered shares, if the cluster is in mixed mode (before upgrading the cluster functional level), it will offer up to 3.0.2 during dialect negotiation.

After you upgrade the cluster functional level, it then offers all clients the new 3.1.1 dialect.

 

3. Other SMB changes that are not protocol-related

There are other changes in Windows that change the SMB Client or SMB Server implementation, but not the protocol itself.

Here are a few important changes in that category:

 

3.1. Removing RequireSecureNegotiate setting

In previous versions of SMB, we introduced “Secure Negotiate”, where the SMB client and server verify integrity of the SMB negotiate request and response messages.

Because some third-party implementations of SMB did not correctly perform this negotiation, we introduced a switch to disable “Secure Negotiate”. We explain this in more detail in this blog post.

Since we have learned via our SMB PlugFests that third parties have fixed their implementations, we are removing the option to bypass “Secure Negotiate” and SMB always performs negotiate validation if the connection’s dialect is 2.x.x or 3.0.x.

Note 1: For SMB 3.1.1 clients and servers, the new Pre-Authentication Integrity feature (described in item 2.1 above) supersedes “Secure Negotiate” with many advantages.

Note 2: With the new release, any third party SMB 2.x.x or SMB 3.0.x implementations that do not implement “Secure Negotiate” will be unable to connect to Windows.

Note 3: While this change improves overall security, it might interfere with some solutions that rely on modifying SMB network traffic, like certain kinds of WAN accelerators.

 

3.2. Dialects with non-zero revision number now reported with the x.y.z notation

As you probably noticed throughout this blog post, we’re using 3 separate digits to notate the version of SMB.

In the past, you might have seen us talk about SMB 3.02. Now we call that SMB 3.0.2.

Note that there is no change when the revision number is 0, like SMB 2.1 or SMB 3.0 (we don’t call them SMB 2.1.0 or SMB 3.0.0).

This new format avoids confusion when comparing SMB dialects and better represents the actual version information used by SMB.

You can use the Get-SmbConnection cmdlet on the Windows SMB client to report the currently used SMB dialects.

 

4. Which protocol is negotiated?

Please note that SMB clients and SMB servers negotiate the SMB dialect that they will use based on each side’s offer.

Here’s a table to help you understand what version you will end up using, depending on what Windows version is running as the SMB client and what version of Windows is running as the SMB server:

 

OS

Windows 10
WS* 2016 TP2

Windows 8.1
WS* 2012 R2

Windows 8
WS* 2012

Windows 7
WS* 2008 R2

Windows Vista
WS* 2008

Previous
versions

Windows 10
WS* 2016 TP2

SMB 3.1.1

SMB 3.0.2

SMB 3.0

SMB 2.1

SMB 2.0.2

SMB 1.x

Windows 8.1
WS* 2012 R2

SMB 3.0.2

SMB 3.0.2

SMB 3.0

SMB 2.1

SMB 2.0.2

SMB 1.x

Windows 8
WS* 2012

SMB 3.0

SMB 3.0

SMB 3.0

SMB 2.1

SMB 2.0.2

SMB 1.x

Windows 7
WS* 2008 R2

SMB 2.1

SMB 2.1

SMB 2.1

SMB 2.1

SMB 2.0.2

SMB 1.x

Windows Vista
WS* 2008

SMB 2.0.2

SMB 2.0.2

SMB 2.0.2

SMB 2.0.2

SMB 2.0.2

SMB 1.x

Previous
versions

SMB 1.x

SMB 1.x

SMB 1.x

SMB 1.x

SMB 1.x

SMB 1.x

* WS = Windows Server

 

Note: Earlier Windows 10 and Windows Server 2016 previews used SMB dialect version 3.1.

 

5. Considering your options for removing the older SMB1 protocol

When Windows Server 2003 hits the end of its extended support later this year, the last supported version of Windows that only works with SMB1 will be gone.

SMB1 is already a separate component in Windows that you can completely remove. However, up to this point, Windows still enables it by default for compatibility reasons.

The next logical step (which we are planning for a future release of Windows) will be to ship SMB1 disabled by default, but still available if necessary.

To help with this transition, you can now enable auditing of SMB1 traffic in your SMB server using PowerShell. This will alert you via events if any clients are still using SMB1.

To enable auditing of SMB1 traffic, use the cmdlet: Set-SmbServerConfiguration –AuditSmb1Access $true

To view the SMB1 events, use the cmdlet: Get-WinEvent -LogName Microsoft-Windows-SMBServer/Audit

If you feel confident that there are no SMB1 clients in your network, you can uninstall SMB1 from your server using the cmdlet: Remove-WindowsFeature FS-SMB1

 

6. Conclusion

I hope this blog post helps you prepare for the upcoming changes in SMB.

We also recommend that you download the SNIA Tutorial on SMB 3, which we recently updated to include details of the 3.1.1 dialect. You can find a copy of that tutorial at http://www.snia.org/sites/default/files2/DSI2015/presentations/FileSystems/JoseBarreto_SMB3_remote%20file%20protocol.pdf


Windows Server 2016 Technical Preview 2 (TP2) and Storage Quality of Service (QoS)

$
0
0

 

Storage Quality of Service (Storage QoS) is a new feature in the upcoming Windows Server 2016 that provides a way to centrally monitor and manage storage performance for virtual machines. The feature automatically improves storage resource fairness between multiple virtual machines using the same file server cluster and allows specific minimum and maximum performance goals (Storage QoS policies) to be configured in units of normalized 8KB IOPs.

 

image

 

With the release of Windows Server 2016 Technical Preview 2 (TP2), the Storage QoS feature is available for testing. Virtual Machine Manager 2016 TP2 also includes the ability to manage Storage QoS. Find below some important links to review so you can experiment with Storage QoS in Windows Server 2016 TP2:

 

Download link for Windows Server 2016 TP2 and System Center 2016 TP2:

 

TechNet Guide for Windows Server 2016 TP2 and Storage QoS:

 

Microsoft Ignite 2015 session on Windows Server 2016 TP2 and Storage QoS:

 

Storage QoS demos (videos):

 

Storage QoS script samples

PowerShell script used in the Windows Server 2016 TP2 Storage QoS demo at MSIgnite

$
0
0

 

This is the Windows PowerShell script I used in the Microsoft Ignite 2015 session on Storage QoS: Hyper-V Storage Performance with Storage Quality of Service. You can also find that demo video by itself at Windows Server 2016 TP2 Storage QoS: Managing with a PowerShell script.

 

The script is fairly straightforward, but there a few interesting points to make about what it does:

  • The script is designed to run from a separate management computer, so all Storage QoS cmdlets use the –CimSession option to point to the Scale-Out File Server.
  • There is a handy function called “Get-PolicyName” that translates a PolicyID into a PolicyName by using Get-StorageQosPolicy.
  • It shows how to filter output of Get-StorageQosFlow. It remove the default policy entries from the list and and sorts the list by VM name (InitiatorName).
  • It offers a good example of how to use Expressions to format the output of Get-StorageQosFlow.
  • It uses the $LastChange variable to later show the time passed since the QoS policies were changed with Set-StorageQosPolicy, since the Get-StorageQosFlow is an average for the last 5 minutes.

 

Here’s the script:

# QosDemo.ps1 – Storage QoS demo script for Microsoft Ignite 2015

Function Get-PolicyName([string] $PolicyID)
{
     $Policy = Get-StorageQosPolicy -CimSession JOSEBDA-FS -PolicyID $PolicyID  
     Return $Policy.Name
}

$LastChange = Get-Date
$Option = "R"

While ($Option -ne "Q") {

    Clear-Host

    Write-Host -ForegroundColor Yellow "C:\> Get-StorageQosFlow |”
    Write-Host -ForegroundColor Yellow “FT InitiatorNodeName, InitiatorName, FilePath, MinimumIOPS, MaximumIOPS, InitiatorIOPS, InitiatorLatency, PolicyID"

    Get-StorageQosFlow -CimSession JOSEBDA-FS | ? InitiatorName -ne "" | Sort InitiatorName |
    FT @{Expression={$_.InitiatorNodeName.Split(".")[0]}; Label="Node"},
       @{Expression={$_.InitiatorName}; Label="VM"},
       @{Expression={$_.FilePath.Split("\")[4]}; Label="File"},
       @{Expression={$_.MinimumIOPS}; Label="MinIOPS"},
       @{Expression={$_.MaximumIOPS}; Label="MaxIOPS"},
       @{Expression={$_.InitiatorIOPS}; Label="VM IOPS"},
       @{Expression={[int] $_.InitiatorLatency}; Label="Latency"},
       @{Expression={(Get-PolicyName $_.PolicyID)}; Label="Policy"},
       PolicyID  -AutoSize

    $TimeDiff = (Get-Date) - $LastChange
    $Minutes = [System.Math]::Round($TimeDiff.TotalMinutes, 2)
    $Seconds = [System.Math]::Round($TimeDiff.TotalSeconds, 2)

    Write-Host "IOPS and Latency are 5-minute averages. Last policy change $Seconds seconds ago ($Minutes minutes ago)."
    Write-Host

    Write-Host -ForegroundColor Yellow "C:\> Get-StorageQosPolicy | FT Name, PolicyType, MinimumIOPS, MaximumIOPS, PolicyID"
    Get-StorageQosPolicy -CimSession JOSEBDA-FS | FT Name, PolicyType, MinimumIOPS, MaximumIOPS, PolicyID -AutoSize

    Write-Host -NoNewline "Command: Quit, Refresh, Zero, Low, High: "
    $Option = Read-Host
    $Option = $Option.ToUpper()[0]

    Switch ($Option) {

    "Z" {
            Set-StorageQosPolicy -CimSession JOSEBDA-FS -Name SilverVM -MinimumIops 0 -MaximumIops 0
            Set-StorageQosPolicy -CimSession JOSEBDA-FS -Name GoldVM -MinimumIops 0 -MaximumIops 0
            $LastChange = Get-Date
        }

    "L" {
            Set-StorageQosPolicy -CimSession JOSEBDA-FS -Name SilverVM -MinimumIops 200 -MaximumIops 500
            Set-StorageQosPolicy -CimSession JOSEBDA-FS -Name GoldVM -MinimumIops 500 -MaximumIops 1000
            $LastChange = Get-Date
        }

    "H" {
            Set-StorageQosPolicy -CimSession JOSEBDA-FS -Name SilverVM -MinimumIops 500 -MaximumIops 1000
            Set-StorageQosPolicy -CimSession JOSEBDA-FS -Name GoldVM -MinimumIops 1000 -MaximumIops 5000
            $LastChange = Get-Date
        }

    } #end switch

} #end while

Microsoft Ignite 2015 sessions related to OneDrive and SharePoint

$
0
0

 

Microsoft Ignite 2015 was a great event, but there were so many sessions that it was truly impossible to watch everything that looked interesting. However, all sessions were recorded and by now they should be available on Channel 9 for streaming.

Even with all the recordings available, it's hard to find the links to all sessions for a given topic. For those interested in SharePoint Server and OneDrive, I compiled here is a list of Microsoft Ignite 2015 sessions related those topics, with links to the Channel9 page with the video recording and slides.

 

CodeTitle
FND2203The Evolution of SharePoint: Overview and Roadmap
BRK2192A File's Future with OneDrive for Business
BRK4110I Sync, Therefore I Am: A Deep Dive on OneDrive Sync Capabilities and Roadmap
BRK2188What's New for IT Professionals in SharePoint Server 2016
BRK4111Future-Proofing Your On-Premises SharePoint Development
BRK3176Effective Search Deployment and Operations in SharePoint Server 2013
BRK3164Deep Dive into Safe SharePoint Branding in Office 365 Using Repeatable Patterns and Practices
BRK2102How to Decide When to Use SharePoint and Yammer and Office 365 Groups and Outlook and Skype
BRK2142Microsoft SharePoint Data Security and Compliance
BRK2163MVP Panel: SharePoint On-Premises, Online and Everything in Between
BRK4124Search Extensibility in SharePoint 2013
BRK3134Implementing Next Generation SharePoint Hybrid Search with the Cloud Search Service Application
BRK3190Designing and Applying Information Architecture for Microsoft SharePoint and Office 365
BRK3115Upgrade to Microsoft SharePoint 2013 and Ready for Cloud Potential
BRK4125Transforming Your SharePoint Full Trust Code to the Office App Model
BRK2130Ernst & Young: Microsoft SharePoint Server 2013 Search Adoption
BRK3201The Social Intranet: Integrate Yammer into Your Microsoft SharePoint Experience
BRK2169Best Practices for Design and Performance in SharePoint Online
BRK2150Proven Ways to Build Robust, No-Code Solutions in Microsoft SharePoint
BRK4116Authentication Patterns for SharePoint Server 2013, Office 365 and OneDrive for Business
BRK4113Hybrid Business Connectivity Services with SharePoint Online
BRK2129Driving User Adoption from a Technical Standpoint for SharePoint, Exchange and Office 365
BRK4106Troubleshooting Hybrid Configurations and Workloads with SharePoint 2013 and Office 365
BRK4131Microsoft SharePoint Server with SQL Server: Now Better Than Ever
BRK3182Microsoft OneDrive for Business: Most Secure for Your Data in the Cloud
BRK2206SharePoint UNPLUGGED! Questions Answered on Anything You Heard This Week
BRK3135OneDrive for Business for B2B External Sharing, IT-Lead Cross-Org Collaboration
BRK3123SharePoint Online and OneDrive for Business Management and Control
BRK2140DSM: Their Journey to Microsoft SharePoint Online and Office365
BRK3122Building Solutions and Apps That Leverage OneDrive for Business
BRK2196Re-Thinking Attachments: Collaborating in Outlook with OneDrive
BRK3153Migration to SharePoint Online Best Practices and New API Investments
BRK4101Elastic SharePoint Storage with StorSimple and Microsoft Azure
BRK3210Advanced Enterprise Content Management and Classification in SharePoint On-Premises and Office 365
BRK3124SharePoint 2013 and Azure IaaS: Better Together
BRK4130Developing Microsoft Office and SharePoint Solutions in a Hybrid World
BRK3183Configuring OneDrive for Business Deployment: Options and Best Practices
BRK2156Virtusa: Establishing SharePoint Hybrid for Social, Search and Personal Storage
BRK4104Setting Up Your On-Premises SharePoint Environment for Custom App Development

 

If my calculations are correct, it would take you over 48 hours to watch all 39 sessions. So get started right now :-). Also, let me know if I missed any sessions on the SharePoint and OneDrive topics.

Do you have a suggestion? Check the User Voice sites for several Microsoft products and services

$
0
0

Several teams at Microsoft are now using a User Voice site as your place to suggest ideas on how to improve specific products and services. These sites make it easier for the teams to receive your suggestions and listen to what you say. Here are the User Voice sites that I could find:

Before you post, be sure to check out the ideas others have suggested and vote on your favorites. If you have a suggestion that’s not listed yet, you can submit your own. But keep suggestions focused on a single topic so the team can track and respond accordingly.

Standard Disclaimer

Please note that these suggestion boxes are moderated and they are voluntary participation-based projects. If your submission is not a product feature suggestion, it may be removed. Please do not send any novel or patentable ideas, copyrighted materials, samples or demos which you do not want to grant a license to Microsoft. Your submission is subject to licensing terms like the Microsoft Windows License Terms.

Microsoft SharePoint Server 2013 PowerShell cmdlet popularity

$
0
0

 

If you follow the blog, you probably saw a little PowerShell script I published a while back to measure the popularity of the cmdlets in a certain module using a Bing search. As an example, that blog showed the popularity of the cmdlets in the SmbShare module.

 

Now I got curious about how the cmdlets in other modules would rank, so I spun up some Azure virtual machines to try some other modules. I decided to try the Microsoft SharePoint Server 2013 module (named Microsoft.SharePoint.PowerShell).

image

 

The results are listed below:

PS C:\> .\PopularCmdlet.ps1 | Sort BingCount -Descending | FT -AutoSize

CmdletName                                                        BingCount
----------                                                        ---------
New-SPSite                                                           480000
Get-SPWeb                                                            187000
Get-SPSite                                                           114000
Get-SPWebApplication                                                  70700
New-SPTrustedSecurityTokenIssuer                                      48300
Add-SPShellAdmin                                                      45200
Mount-SPContentDatabase                                               44500
Enable-SPFeature                                                      44300
Import-SPWeb                                                          43300
Restore-SPSite                                                        42900
Add-SPSolution                                                        40700
Install-SPSolution                                                    38000
Get-SPUser                                                            37800
Export-SPWeb                                                          37200
New-SPConfigurationDatabase                                           36700
Get-SPServiceInstance                                                 35800
Get-SPServiceApplication                                              35600
Backup-SPSite                                                         35200
Get-SPFarm                                                            33000
Register-SPWorkflowService                                            31100
New-SPWeb                                                             30200
Get-SPDatabase                                                        30100
Upgrade-SPContentDatabase                                             29900
Test-SPContentDatabase                                                29800
New-SPWebApplication                                                  29700
Move-SPUser                                                           28500
Get-SPFeature                                                         27700
Enable-SPUserLicensing                                                26600
Get-SPEnterpriseSearchServiceApplication                              25200
Get-SPUserLicensing                                                   24700
Backup-SPFarm                                                         24600
Disable-SPUserLicensing                                               24400
Set-SPUser                                                            24300
Disable-SPFeature                                                     24200
Copy-SPSite                                                           24200
Get-SPContentDatabase                                                 23300
Uninstall-SPSolution                                                  23100
Update-SPSolution                                                     21600
Remove-SPSite                                                         21300
Set-SPSite                                                            20800
Get-SPTimerJob                                                        20600
Get-SPManagedAccount                                                  20400
New-SPWOPIBinding                                                     19500
Update-SPProfilePhotoStore                                            19400
Start-SPServiceInstance                                               19300
New-SPUser                                                            18800
Stop-SPAssignment                                                     18700
New-SPTrustedIdentityTokenIssuer                                      18400
Set-SPEnterpriseSearchServiceApplication                              18100
New-SPManagedAccount                                                  17400
New-SPEnterpriseSearchServiceApplication                              17300
Get-SPLogEvent                                                        17000
New-SPCentralAdministration                                           16900
Remove-SPUser                                                         16300
Get-SPEnterpriseSearchServiceInstance                                 16300
New-SPProfileServiceApplication                                       16100
Get-SPSecurityTokenServiceConfig                                      16000
New-SPContentDatabase                                                 15700
Install-SPFeature                                                     15300
Restore-SPFarm                                                        15200
Get-SPSolution                                                        14800
Get-SPShellAdmin                                                      14600
Start-SPEnterpriseSearchServiceInstance                               14500
Get-SPServer                                                          14200
Connect-SPConfigurationDatabase                                       14200
Set-SPWeb                                                             14100
Stop-SPServiceInstance                                                13800
Dismount-SPContentDatabase                                            13700
Get-SPServiceApplicationPool                                          13700
New-SPServiceApplicationPool                                          13500
Convert-SPWebApplication                                              13500
Get-SPServiceContext                                                  13500
Merge-SPLogFile                                                       13500
Set-SPManagedAccount                                                  13400
Start-SPAssignment                                                    13100
Add-SPDistributedCacheServiceInstance                                 12900
Upgrade-SPSite                                                        12800
Get-SPServiceApplicationProxy                                         12800
Remove-SPWeb                                                          12600
Remove-SPSolution                                                     12400
Get-SPWOPIBinding                                                     12000
Get-SPWebTemplate                                                     11700
Start-SPAdminJob                                                      11400
Get-SPEnterpriseSearchService                                         11000
New-SPClaimsPrincipal                                                 10900
Remove-SPWOPIBinding                                                  10800
Restore-SPDeletedSite                                                 10800
Get-SPProduct                                                         10600
Install-SPService                                                     10600
Uninstall-SPFeature                                                   10600
Enable-SPSessionStateService                                          10500
Set-SPWebApplication                                                  10400
New-SPUsageApplication                                                10400
Remove-SPDeletedSite                                                  10400
Import-SPMetadataWebServicePartitionData                              10100
New-SPMetadataServiceApplication                                      10100
Set-SPEnterpriseSearchTopology                                        10100
New-SPSecureStoreServiceApplication                                   10000
Get-SPDeletedSite                                                      9880
Get-SPTrustedSecurityTokenIssuer                                       9730
New-SPSubscriptionSettingsServiceApplication                           9730
Get-SPTrustedIdentityTokenIssuer                                       9660
Get-SPUsageDefinition                                                  9580
Set-SPEnterpriseSearchCrawlDatabase                                    9360
Get-SPEnterpriseSearchStatus                                           9330
New-SPStateServiceDatabase                                             9320
Set-SPSiteURL                                                          9060
Set-SPUsageDefinition                                                  8870
Test-SPSite                                                            8840
Set-SPEnterpriseSearchService                                          8520
New-SPClaimTypeMapping                                                 8430
New-SPEnterpriseSearchCrawlComponent                                   8370
Set-SPAppSiteSubscriptionName                                          8320
Rename-SPServer                                                        8280
Remove-SPClaimProvider                                                 8230
Repair-SPManagedAccountDeployment                                      8070
Set-SPProfileServiceApplication                                        8060
New-SPEnterpriseSearchServiceApplicationProxy                          8040
Get-SPEnterpriseSearchCrawlContentSource                               7980
Remove-SPWebApplication                                                7960
Remove-SPServiceApplication                                            7940
New-SPAuthenticationProvider                                           7930
Restore-SPEnterpriseSearchServiceApplication                           7930
Get-SPSiteAdministration                                               7670
Get-SPAppPrincipal                                                     7610
Disconnect-SPConfigurationDatabase                                     7600
Remove-SPDistributedCacheServiceInstance                               7480
Set-SPBusinessDataCatalogThrottleConfig                                7460
Set-SPPassPhrase                                                       7410
Register-SPAppPrincipal                                                7360
New-SPEnterpriseSearchAdminComponent                                   7320
Remove-SPContentDatabase                                               7280
Install-SPUserSolution                                                 7170
Get-SPEnterpriseSearchMetadataManagedProperty                          7120
Set-SPUsageService                                                     7080
Move-SPProfileManagedMetadataProperty                                  7070
Export-SPMetadataWebServicePartitionData                               7060
Set-SPCustomLayoutsPage                                                7010
Set-SPMetadataServiceApplication                                       7010
Upgrade-SPSingleSignOnDatabase                                         6950
Set-SPLogLevel                                                         6910
New-SPClaimTypeEncoding                                                6910
New-SPEnterpriseSearchIndexComponent                                   6750
Set-SPFarmConfig                                                       6690
Add-SPUserSolution                                                     6610
Add-SPProfileSyncConnection                                            6500
Set-SPAppDomain                                                        6360
Install-SPDataConnectionFile                                           6360
Set-SPDiagnosticsProvider                                              6340
Set-SPEnterpriseSearchAdministrationComponent                          6320
Get-SPEnterpriseSearchTopology                                         6280
Install-SPApplicationContent                                           6270
Update-SPDistributedCacheSize                                          6250
Set-SPWOPIBinding                                                      6250
Get-SPClaimProvider                                                    6250
Get-SPCertificateAuthority                                             6200
Import-SPAppPackage                                                    6190
Get-SPBusinessDataCatalogMetadataObject                                6080
New-SPSecureStoreApplication                                           6030
Install-SPApp                                                          6020
New-SPEnterpriseSearchQueryScope                                       6020
Remove-SPManagedAccount                                                6000
Remove-SPTrustedIdentityTokenIssuer                                    5900
Install-SPHelpCollection                                               5870
Remove-SPEnterpriseSearchComponent                                     5840
New-SPSubscriptionSettingsServiceApplicationProxy                      5790
Add-SPProfileLeader                                                    5790
Suspend-SPEnterpriseSearchServiceApplication                           5780
Get-SPLogLevel                                                         5770
Get-SPAlternateURL                                                     5700
Initialize-SPResourceSecurity                                          5700
Set-SPEnterpriseSearchCrawlContentSource                               5700
Get-SPPendingUpgradeActions                                            5650
New-SPWOPISuppressionSetting                                           5640
Remove-SPServiceApplicationPool                                        5640
Get-SPTaxonomySession                                                  5630
Remove-SPEnterpriseSearchServiceApplication                            5590
Update-SPSecureStoreApplicationServerKey                               5550
Set-SPServiceApplicationSecurity                                       5530
Set-SPTrustedIdentityTokenIssuer                                       5530
Stop-SPDistributedCacheServiceInstance                                 5520
Get-SPEnterpriseSearchMetadataCrawledProperty                          5510
New-SPEnterpriseSearchTopology                                         5490
Start-SPEnterpriseSearchQueryAndSiteSettingsServiceInstance            5420
Get-SPWOPIZone                                                         5390
New-SPEnterpriseSearchMetadataManagedProperty                          5390
Revoke-SPObjectSecurity                                                5320
Start-SPTimerJob                                                       5310
New-SPEnterpriseSearchCrawlContentSource                               5300
Get-SPAuthenticationRealm                                              5300
Grant-SPObjectSecurity                                                 5270
Set-SPUsageApplication                                                 5260
Set-SPTimerJob                                                         5240
Set-SPSecurityTokenServiceConfig                                       5230
New-SPEnterpriseSearchMetadataCrawledProperty                          5220
Receive-SPServiceApplicationConnectionInfo                             5210
Set-SPSiteSubscriptionConfig                                           5160
New-SPWebApplicationAppDomain                                          5160
Set-SPCentralAdministration                                            5150
Get-SPBusinessDataCatalogThrottleConfig                                5110
New-SPManagedPath                                                      5110
Resume-SPEnterpriseSearchServiceApplication                            5090
Uninstall-SPAppInstance                                                5080
Get-SPAppInstance                                                      5060
Get-SPEnterpriseSearchQueryScope                                       5060
Add-SPSiteSubscriptionProfileConfig                                    5040
New-SPEnterpriseSearchLanguageResourcePhrase                           5020
Set-SPWOPIZone                                                         5010
New-SPAppManagementServiceApplication                                  5000
New-SPEnterpriseSearchCrawlDatabase                                    4980
Remove-SPServiceApplicationProxy                                       4930
New-SPEnterpriseSearchCrawlRule                                        4920
Remove-SPSiteURL                                                       4890
Stop-SPEnterpriseSearchServiceInstance                                 4840
New-SPProfileServiceApplicationProxy                                   4840
New-SPEnterpriseSearchQueryProcessingComponent                         4830
Upgrade-SPEnterpriseSearchServiceApplication                           4800
New-SPEnterpriseSearchQueryScopeRule                                   4750
New-SPUserLicenseMapping                                               4750
Install-SPWebPartPack                                                  4690
Get-SPServiceHostConfig                                                4690
Get-SPEnterpriseSearchServiceApplicationProxy                          4650
New-SPBusinessDataCatalogServiceApplication                            4650
New-SPSiteSubscription                                                 4640
Get-SPEnterpriseSearchMetadataCategory                                 4620
Remove-SPShellAdmin                                                    4580
New-SPMetadataServiceApplicationProxy                                  4560
Set-SPAppPrincipalPermission                                           4560
New-SPEnterpriseSearchAnalyticsProcessingComponent                     4540
Set-SPContentDatabase                                                  4520
Get-SPSiteSubscription                                                 4500
Get-SPEnterpriseSearchQueryKeyword                                     4480
Remove-SPEnterpriseSearchCrawlContentSource                            4480
Import-SPEnterpriseSearchThesaurus                                     4420
Get-SPEnterpriseSearchQueryAndSiteSettingsServiceInstance              4410
Get-SPSessionStateService                                              4410
Repair-SPSite                                                          4410
Get-SPServiceApplicationSecurity                                       4410
Remove-SPBusinessDataCatalogModel                                      4390
Set-SPMetadataServiceApplicationProxy                                  4390
New-SPSiteSubscriptionFeaturePack                                      4380
New-SPEnterpriseSearchSecurityTrimmer                                  4380
Remove-SPTrustedSecurityTokenIssuer                                    4340
Set-SPEnterpriseSearchRankingModel                                     4310
Get-SPSecureStoreApplication                                           4300
Get-SPSiteSubscriptionFeaturePack                                      4300
Get-SPEnterpriseSearchCrawlDatabase                                    4270
Set-SPIRMSettings                                                      4260
Remove-SPTrustedRootAuthority                                          4240
Set-SPServiceApplicationPool                                           4240
Get-SPEnterpriseSearchCrawlCustomConnector                             4230
Remove-SPEnterpriseSearchQueryScope                                    4210
Get-SPEnterpriseSearchComponent                                        4200
Get-SPEnterpriseSearchQuerySuggestionCandidates                        4200
Get-SPEnterpriseSearchOwner                                            4190
Get-SPEnterpriseSearchRankingModel                                     4190
Set-SPProfileServiceApplicationSecurity                                4190
Get-SPSiteURL                                                          4180
Remove-SPEnterpriseSearchLanguageResourcePhrase                        4170
Set-SPEnterpriseSearchServiceInstance                                  4160
Update-SPInfoPathUserFileUrl                                           4160
Update-SPUserSolution                                                  4150
New-SPAlternateURL                                                     4150
Import-SPEnterpriseSearchTopology                                      4130
Get-SPTopologyServiceApplication                                       4120
Grant-SPBusinessDataCatalogMetadataObject                              4090
Get-SPEnterpriseSearchFileFormat                                       4090
Set-SPEnterpriseSearchContentEnrichmentConfiguration                   4080
Remove-SPEnterpriseSearchCrawlMapping                                  4070
Get-SPEnterpriseSearchQueryScopeRule                                   4070
New-SPEnterpriseSearchCrawlExtension                                   4070
Get-SPUserLicense                                                      4040
Set-SPEnterpriseSearchMetadataManagedProperty                          4030
Get-SPStateServiceApplication                                          4020
Publish-SPServiceApplication                                           3990
New-SPODataConnectionSetting                                           3980
Get-SPUserSolution                                                     3980
Uninstall-SPUserSolution                                               3980
Remove-SPEnterpriseSearchMetadataManagedProperty                       3970
Set-SPEnterpriseSearchMetadataMapping                                  3970
Remove-SPEnterpriseSearchTopology                                      3960
New-SPServiceApplicationProxyGroup                                     3940
Get-SPSiteSubscriptionConfig                                           3930
Backup-SPConfigurationDatabase                                         3920
Set-SPSiteAdministration                                               3910
Set-SPAuthenticationRealm                                              3900
Add-SPSiteSubscriptionFeaturePackMember                                3890
Remove-SPEnterpriseSearchCrawlCustomConnector                          3890
Remove-SPSiteSubscription                                              3890
Get-SPEnterpriseSearchMetadataMapping                                  3880
Get-SPTrustedRootAuthority                                             3880
New-SPEnterpriseSearchSiteHitRule                                      3870
Get-SPEnterpriseSearchLanguageResourcePhrase                           3870
Get-SPFarmConfig                                                       3870
Remove-SPEnterpriseSearchCrawlDatabase                                 3850
Set-SPEnterpriseSearchQueryScope                                       3840
New-SPAppManagementServiceApplicationProxy                             3840
Remove-SPEnterpriseSearchCrawlRule                                     3840
Add-SPClaimTypeMapping                                                 3830
Remove-SPEnterpriseSearchQueryAuthority                                3820
Get-SPEnterpriseSearchCrawlExtension                                   3820
Set-SPEnterpriseSearchMetadataCrawledProperty                          3810
Remove-SPEnterpriseSearchServiceApplicationProxy                       3810
Remove-SPSiteSubscriptionSettings                                      3800
Get-SPEnterpriseSearchCrawlMapping                                     3800
Remove-SPEnterpriseSearchMetadataCategory                              3790
Import-SPEnterpriseSearchCustomExtractionDictionary                    3790
Set-SPDiagnosticConfig                                                 3780
Update-SPSecureStoreMasterKey                                          3780
Remove-SPEnterpriseSearchQueryScopeRule                                3760
Update-SPAppInstance                                                   3760
Set-SPEnterpriseSearchQueryScopeRule                                   3760
Import-SPBusinessDataCatalogModel                                      3760
Remove-SPEnterpriseSearchMetadataMapping                               3740
Get-SPUsageService                                                     3740
Get-SPEnterpriseSearchSecurityTrimmer                                  3740
Stop-SPEnterpriseSearchQueryAndSiteSettingsServiceInstance             3730
Remove-SPEnterpriseSearchQueryDemoted                                  3720
New-SPEnterpriseSearchFileFormat                                       3720
Remove-SPEnterpriseSearchSiteHitRule                                   3720
New-SPSecureStoreServiceApplicationProxy                               3710
Remove-SPEnterpriseSearchCrawlExtension                                3700
Disable-SPTimerJob                                                     3700
Get-SPEnterpriseSearchQueryAndSiteSettingsService                      3700
Export-SPEnterpriseSearchTopology                                      3700
Remove-SPEnterpriseSearchQueryKeyword                                  3700
New-SPEnterpriseSearchQueryAuthority                                   3700
Get-SPAuthenticationProvider                                           3700
Get-SPEnterpriseSearchQueryAuthority                                   3690
Get-SPDistributedCacheClientSetting                                    3690
Get-SPEnterpriseSearchSiteHitRule                                      3690
Set-SPEnterpriseSearchQueryAuthority                                   3680
Remove-SPEnterpriseSearchRankingModel                                  3680
New-SPEnterpriseSearchCrawlCustomConnector                             3680
Get-SPEnterpriseSearchQueryAndSiteSettingsServiceProxy                 3670
Get-SPEnterpriseSearchQueryDemoted                                     3670
New-SPEnterpriseSearchQueryDemoted                                     3660
Get-SPEnterpriseSearchCrawlRule                                        3650
Set-SPEnterpriseSearchQueryKeyword                                     3640
Set-SPEnterpriseSearchServiceApplicationProxy                          3640
Set-SPEnterpriseSearchMetadataCategory                                 3640
Set-SPBusinessDataCatalogServiceApplication                            3630
Set-SPEnterpriseSearchCrawlRule                                        3630
New-SPEnterpriseSearchQueryKeyword                                     3630
Remove-SPEnterpriseSearchSecurityTrimmer                               3610
New-SPClaimProvider                                                    3600
Get-SPWorkflowServiceApplicationProxy                                  3590
New-SPEnterpriseSearchMetadataCategory                                 3580
Get-SPUsageApplication                                                 3570
Get-SPProfileLeader                                                    3570
Remove-SPClaimTypeMapping                                              3560
Get-SPServiceApplicationProxyGroup                                     3560
Set-SPTrustedRootAuthority                                             3540
Remove-SPUserSolution                                                  3530
New-SPStateServiceApplicationProxy                                     3520
New-SPEnterpriseSearchCrawlMapping                                     3510
Set-SPServiceApplication                                               3480
Get-SPManagedPath                                                      3460
Add-SPUserLicenseMapping                                               3440
Set-SPSiteSubscriptionProfileConfig                                    3430
Set-SPServiceHostConfig                                                3430
Get-SPMetadataServiceApplication                                       3400
New-SPLogFile                                                          3390
Get-SPEnterpriseSearchAdministrationComponent                          3380
Set-SPSubscriptionSettingsServiceApplication                           3370
Remove-SPEnterpriseSearchLinksDatabase                                 3370
Import-SPSiteSubscriptionSettings                                      3340
Request-SPUpgradeEvaluationSite                                        3330
Get-SPCustomLayoutsPage                                                3310
Update-SPWOPIProofKey                                                  3310
Get-SPEnterpriseSearchQuerySpellingCorrection                          3300
Set-SPEnterpriseSearchQuerySpellingCorrection                          3300
Add-SPServiceApplicationProxyGroupMember                               3290
Upgrade-SPFarm                                                         3290
Remove-SPSiteSubscriptionBusinessDataCatalogConfig                     3280
Get-SPProfileServiceApplicationSecurity                                3260
Remove-SPConfigurationDatabase                                         3260
Export-SPSiteSubscriptionSettings                                      3250
Update-SPInfoPathAdminFileUrl                                          3240
Backup-SPEnterpriseSearchServiceApplicationIndex                       3190
New-SPAzureAccessControlServiceApplicationProxy                        3170
Remove-SPTrustedServiceTokenIssuer                                     3140
Update-SPRepopulateMicroblogFeedCache                                  3130
Set-SPClaimProvider                                                    3110
Remove-SPServiceApplicationProxyGroup                                  3100
Initialize-SPStateServiceDatabase                                      3100
Set-SPPerformancePointSecureDataValues                                 3080
Get-SPServiceApplicationEndpoint                                       3070
Get-SPClaimProviderManager                                             3070
Set-SPDistributedCacheClientSetting                                    3030
Get-SPRequestManagementSettings                                        3020
Get-SPTrustedServiceTokenIssuer                                        2990
Add-SPDiagnosticsPerformanceCounter                                    2990
Test-SPInfoPathFormTemplate                                            2970
Start-SPContentDeploymentJob                                           2960
Set-SPBusinessDataCatalogMetadataObject                                2930
Get-SPDiagnosticConfig                                                 2920
Get-SPInfoPathFormTemplate                                             2920
Update-SPFarmEncryptionKey                                             2920
Get-SPInfoPathFormsService                                             2900
Get-SPClaimTypeEncoding                                                2890
Remove-SPProfileSyncConnection                                         2870
Remove-SPSolutionDeploymentLock                                        2870
Get-SPWebPartPack                                                      2870
Get-SPBackupHistory                                                    2870
Remove-SPUsageApplication                                              2860
Remove-SPServiceApplicationProxyGroupMember                            2850
Get-SPWebApplicationHttpThrottlingMonitor                              2810
Get-SPWorkflowConfig                                                   2800
Get-SPSiteUpgradeSessionInfo                                           2800
Get-SPWOPISuppressionSetting                                           2800
Remove-SPSiteSubscriptionFeaturePack                                   2790
Mount-SPStateServiceDatabase                                           2780
Clear-SPLogLevel                                                       2780
Set-SPTrustedServiceTokenIssuer                                        2730
Remove-SPManagedPath                                                   2700
Get-SPDiagnosticsProvider                                              2700
Get-SPHealthAnalysisRule                                               2680
Get-SPMetadataServiceApplicationProxy                                  2620
Update-SPRepopulateMicroblogLMTCache                                   2610
Import-SPInfoPathAdministrationFiles                                   2610
Set-SPWebApplicationHttpThrottlingMonitor                              2600
Enable-SPTimerJob                                                      2590
Export-SPInfoPathAdministrationFiles                                   2590
Set-SPInfoPathFormsService                                             2580
Set-SPTopologyServiceApplication                                       2580
Get-SPHelpCollection                                                   2570
Get-SPInfoPathUserAgent                                                2570
Remove-SPSiteSubscriptionFeaturePackMember                             2570
Set-SPServiceApplicationEndpoint                                       2560
Get-SPStateServiceDatabase                                             2550
Set-SPAlternateURL                                                     2550
Remove-SPSiteSubscriptionProfileConfig                                 2540
Set-SPDataConnectionFile                                               2540
Update-SPInfoPathFormTemplate                                          2530
Unpublish-SPServiceApplication                                         2530
Import-SPBusinessDataCatalogDotNetAssembly                             2510
Add-SPRoutingMachinePool                                               2500
Get-SPDataConnectionFileDependent                                      2500
Get-SPDataConnectionFile                                               2490
Uninstall-SPDataConnectionFile                                         2450
Get-SPPerformancePointServiceApplication                               2420
Remove-SPStateServiceDatabase                                          2400
Set-SPSiteSubscriptionMetadataConfig                                   2400
Get-SPInfoPathWebServiceProxy                                          2400
Set-SPInfoPathWebServiceProxy                                          2380
Add-SPInfoPathUserAgent                                                2360
Remove-SPDiagnosticsPerformanceCounter                                 2360
Get-SPTopologyServiceApplicationProxy                                  2360
Start-SPInfoPathFormTemplate                                           2340
Remove-SPInfoPathUserAgent                                             2330
New-SPPerformancePointServiceApplication                               2300
Set-SPContentDeploymentJob                                             2290
Get-SPContentDeploymentJob                                             2270
Stop-SPInfoPathFormTemplate                                            2260
Move-SPBlobStorageLocation                                             2260
Set-SPInfoPathFormTemplate                                             2260
Export-SPBusinessDataCatalogModel                                      2260
Get-SPDiagnosticsPerformanceCounter                                    2260
Set-SPDesignerSettings                                                 2260
Set-SPProfileServiceApplicationProxy                                   2250
Remove-SPProfileLeader                                                 2250
Get-SPContentDeploymentPath                                            2250
New-SPContentDeploymentPath                                            2240
Update-SPSecureStoreCredentialMapping                                  2230
Get-SPUserLicenseMapping                                               2230
Set-SPStateServiceApplication                                          2220
Get-SPStateServiceApplicationProxy                                     2210
Enable-SPBusinessDataCatalogEntity                                     2210
Revoke-SPBusinessDataCatalogMetadataObject                             2200
Clear-SPDistributedCacheItem                                           2190
Get-SPProcessAccount                                                   2190
Get-SPDesignerSettings                                                 2170
Get-SPRoutingMachinePool                                               2150
Copy-SPBusinessDataCatalogAclToChildren                                2150
Set-SPWorkflowConfig                                                   2120
Export-SPAppPackage                                                    2110
Get-SPAppDomain                                                        2100
Set-SPSessionStateService                                              2090
New-SPWorkManagementServiceApplication                                 2090
Dismount-SPStateServiceDatabase                                        2090
Set-SPPerformancePointServiceApplication                               2070
New-SPSecureStoreApplicationField                                      2070
Get-SPRoutingMachineInfo                                               2070
Add-SPThrottlingRule                                                   2070
Add-SPRoutingRule                                                      2060
Update-SPSecureStoreGroupCredentialMapping                             2040
Remove-SPUserLicenseMapping                                            2040
Uninstall-SPHelpCollection                                             2030
Get-SPRoutingRule                                                      2020
Resume-SPStateServiceDatabase                                          2000
Get-SPMobileMessagingAccount                                           2000
Set-SPSiteSubscriptionEdiscoveryHub                                    2000
Set-SPBrowserCustomerExperienceImprovementProgram                      2000
Set-SPStateServiceDatabase                                             1980
Disable-SPBusinessDataCatalogEntity                                    1980
Suspend-SPStateServiceDatabase                                         1980
Remove-SPContentDeploymentPath                                         1970
Set-SPSecureStoreApplication                                           1960
Set-SPStateServiceApplicationProxy                                     1960
Enable-SPWebApplicationHttpThrottling                                  1960
Set-SPContentDeploymentPath                                            1950
Remove-SPSocialItemByDate                                              1950
Remove-SPSiteUpgradeSessionInfo                                        1940
Disable-SPWebApplicationHttpThrottling                                 1940
Set-SPMobileMessagingAccount                                           1930
Remove-SPContentDeploymentJob                                          1930
Get-SPBrowserCustomerExperienceImprovementProgram                      1920
Clear-SPSiteSubscriptionBusinessDataCatalogConfig                      1910
Disable-SPSessionStateService                                          1900
Remove-SPSiteSubscriptionMetadataConfig                                1890
Clear-SPMetadataWebServicePartitionData                                1890
Uninstall-SPWebPartPack                                                1870
New-SPPerformancePointServiceApplicationProxy                          1840
Get-SPSiteSubscriptionMetadataConfig                                   1840
Export-SPSiteSubscriptionBusinessDataCatalogConfig                     1830
New-SPContentDeploymentJob                                             1790
Get-SPODataConnectionSettingMetadata                                   1730
Set-SPWorkManagementServiceApplication                                 1710
Set-SPRoutingMachineInfo                                               1700
Get-SPSiteSubscriptionEdiscoveryHub                                    1690
Get-SPSiteSubscriptionEdiscoverySearchScope                            1680
Set-SPRequestManagementSettings                                        1670
Get-SPUpgradeActions                                                   1660
New-SPUsageLogFile                                                     1660
Enable-SPAppAutoProvision                                              1650
Disable-SPHealthAnalysisRule                                           1650
Set-SPBingMapsKey                                                      1590
New-SPTranslationServiceApplication                                    1570
Disable-SPSingleSignOn                                                 1540
Get-SPPerformancePointServiceApplicationTrustedLocation                1480
Clear-SPSecureStoreCredentialMapping                                   1470
Set-SPSecureStoreServiceApplication                                    1420
Set-SPWordConversionServiceApplication                                 1370
Clear-SPSecureStoreDefaultProvider                                     1330
Set-SPSecureStoreDefaultProvider                                       1290
Remove-SPWordConversionServiceJobHistory                               1240
New-SPPerformancePointServiceApplicationTrustedLocation                1210
Remove-SPPerformancePointServiceApplicationTrustedLocation             1210
Remove-SPPerformancePointServiceApplication                            1200
Clear-SPPerformancePointServiceApplicationTrustedLocation              1190
Remove-SPPerformancePointServiceApplicationProxy                       1160
New-SPWorkManagementServiceApplicationProxy                            1080
New-SPPowerPointConversionServiceApplication                           1070
Set-SPTranslationServiceApplication                                    1020
New-SPPowerPointConversionServiceApplicationProxy                       994
Enable-SPHealthAnalysisRule                                             881
New-SPTranslationServiceApplicationProxy                                 50
Get-SPEnterpriseSearchResultItemType                                     50
Remove-SPODataConnectionSetting                                          49
Add-SPPluggableSecurityTrimmer                                           49
Get-SPBingMapsKey                                                        49
Get-SPAppSiteSubscriptionName                                            47
Get-SPEnterpriseSearchPropertyRule                                       47
Get-SPEnterpriseSearchHostController                                     47
Add-SPRoutingMachineInfo                                                 46
Get-SPEnterpriseSearchPropertyRuleCollection                             46
Get-SPPluggableSecurityTrimmer                                           46
Get-SPAppStateUpdateInterval                                             45
Set-SPSiteSubscriptionIRMConfig                                          45
Set-SPRoutingMachinePool                                                 45
Get-SPEnterpriseSearchLinksDatabase                                      45
Add-SPAppDeniedEndpoint                                                  45
Update-SPAppCatalogConfiguration                                         45
Clear-SPBusinessDataCatalogEntityNotificationWeb                         45
Disable-SPAppAutoProvision                                               45
Get-SPThrottlingRule                                                     44
Remove-SPRoutingMachineInfo                                              44
Get-SPIRMSettings                                                        43
Get-SPEnterpriseSearchContentEnrichmentConfiguration                     43
Set-SPThrottlingRule                                                     43
Get-SPEnterpriseSearchLinguisticComponentsStatus                         42
Set-SPODataConnectionSetting                                             42
Set-SPEnterpriseSearchLinksDatabase                                      42
Get-SPSiteSubscriptionIRMConfig                                          41
Get-SPScaleOutDatabaseLogEntry                                           41
Get-SPAppStateSyncLastRunTime                                            41
Add-SPScaleOutDatabase                                                   41
Install-SPEduSites                                                       41
Import-SPEnterpriseSearchPopularQueries                                  41
Get-SPAppScaleProfile                                                    41
Add-SPServerScaleOutDatabase                                             41
Export-SPScaleOutDatabaseTenantData                                      40
New-SPEnterpriseSearchResultItemType                                     40
Get-SPServerScaleOutDatabase                                             40
Get-SPBusinessDataCatalogEntityNotificationWeb                           40
Remove-SPPluggableSecurityTrimmer                                        40
Set-SPEnterpriseSearchCrawlLogReadPermission                             40
Get-SPScaleOutDatabase                                                   40
Get-SPEnterpriseSearchVssDataPath                                        40
Get-SPAppAutoProvisionConnection                                         39
Clear-SPScaleOutDatabaseTenantData                                       39
Remove-SPEnterpriseSearchContentEnrichmentConfiguration                  39
Set-SPInternalAppStateUpdateInterval                                     39
Get-SPScaleOutDatabaseInconsistency                                      39
Get-SPInternalAppStateSyncLastRunTime                                    38
Set-SPODataConnectionSettingMetadata                                     38
Get-SPInternalAppStateUpdateInterval                                     38
Get-SPScaleOutDatabaseDataState                                          38
Get-SPODataConnectionSetting                                             38
Move-SPDeletedSite                                                       38
New-SPBECWebServiceApplicationProxy                                      37
Set-SPAppStoreConfiguration                                              37
Clear-SPScaleOutDatabaseLog                                              37
Get-SPEnterpriseSearchCrawlLogReadPermission                             37
Get-SPOfficeStoreAppsDefaultActivation                                   37
Clear-SPServerScaleOutDatabaseDeletedDataSubRange                        35
Set-SPBusinessDataCatalogEntityNotificationWeb                           35
Clear-SPServerScaleOutDatabaseTenantData                                 35
Get-SPEnterpriseSearchServiceApplicationBackupStore                      35
Clear-SPScaleOutDatabaseDeletedDataSubRange                              35
Get-SPServerScaleOutDatabaseInconsistency                                35
Export-SPServerScaleOutDatabaseTenantData                                34
Remove-SPRoutingRule                                                     34
Get-SPServerScaleOutDatabaseLogEntry                                     34
Remove-SPEnterpriseSearchFileFormat                                      34
New-SPMarketplaceWebServiceApplicationProxy                              34
Set-SPAppManagementDeploymentId                                          34
Start-SPDiagnosticsSession                                               33
Set-SPAppStateUpdateInterval                                             33
Remove-SPThrottlingRule                                                  33
Set-SPWorkManagementServiceApplicationProxy                              33
Split-SPScaleOutDatabase                                                 33
Set-SPPowerPointConversionServiceApplication                             33
Get-SPServerScaleOutDatabaseDataState                                    33
Import-SPScaleOutDatabaseTenantData                                      33
Set-SPTranslationServiceApplicationProxy                                 32
Set-SPAppAutoProvisionConnection                                         32
Set-SPTrustedSecurityTokenIssuer                                         32
Copy-SPActivitiesToWorkflowService                                       32
Set-SPOfficeStoreAppsDefaultActivation                                   32
Remove-SPRoutingMachinePool                                              32
Stop-SPDiagnosticsSession                                                32
Set-SPRoutingRule                                                        31
Clear-SPServerScaleOutDatabaseLog                                        31
Remove-SPServerScaleOutDatabase                                          31
Remove-SPAppDeniedEndpoint                                               31
Set-SPScaleOutDatabaseDataSubRange                                       31
Remove-SPAppPrincipalPermission                                          30
Remove-SPTranslationServiceJobHistory                                    30
Remove-SPScaleOutDatabase                                                30
Get-SPAppStoreConfiguration                                              30
Upgrade-SPEnterpriseSearchServiceApplicationSiteSettings                 30
Set-SPAppScaleProfile                                                    30
Set-SPScaleOutDatabaseDataRange                                          29
Import-SPServerScaleOutDatabaseTenantData                                29
Split-SPServerScaleOutDatabase                                           29
Remove-SPEnterpriseSearchResultItemType                                  29
Restore-SPEnterpriseSearchServiceApplicationIndex                        28
Set-SPEnterpriseSearchLinguisticComponentsStatus                         28
Set-SPEnterpriseSearchPrimaryHostController                              28
Set-SPServerScaleOutDatabaseDataRange                                    27
New-SPEduClass                                                           27
Set-SPServerScaleOutDatabaseDataSubRange                                 26
Get-SPAppAcquisitionConfiguration                                        26
Move-SPEnterpriseSearchLinksDatabases                                    26
Get-SPWebApplicationAppDomain                                            26
Get-SPAppHostingQuotaConfiguration                                       25
Remove-SPEnterpriseSearchServiceApplicationSiteSettings                  25
Get-SPSecureStoreSystemAccount                                           25
Clear-SPAppDeniedEndpointList                                            25
Get-SPAppDeniedEndpointList                                              25
Remove-SPEnterpriseSearchCrawlLogReadPermission                          25
Set-SPEnterpriseSearchResultItemType                                     25
Add-SPSecureStoreSystemAccount                                           24
Remove-SPEnterpriseSearchTenantSchema                                    24
Get-SPAppDisablingConfiguration                                          24
Import-SPPerformancePointContent                                         24
Set-SPAppHostingQuotaConfiguration                                       23
Remove-SPSecureStoreSystemAccount                                        23
Get-SPBingMapsBlock                                                      23
Export-SPPerformancePointContent                                         23
New-SPUserSettingsProvider                                               22
Remove-SPEnterpriseSearchTenantConfiguration                             22
Remove-SPUserSettingsProvider                                            22
Remove-SPActivityFeedItems                                               22
Get-SPUserSettingsProvider                                               21
Get-SPUserSettingsProviderManager                                        21
Set-SPAppAcquisitionConfiguration                                        21
Set-SPAppDisablingConfiguration                                          21
Remove-SPWebApplicationAppDomain                                         21
Add-SPEduUser                                                            20
New-SPStateServiceApplication                                            20
Set-SPBingMapsBlock                                                      19
Add-SPEduClassMember                                                     19
Get-SPEduServiceSetting                                                  18
Move-SPSite                                                              17
Remove-SPEduClassMember                                                  16
New-SPWorkflowServiceApplicationProxy                                    16
Set-SPEduServiceSetting                                                  16
Get-SPEnterpriseSearchResultSource                                       14
New-SPBusinessDataCatalogServiceApplicationProxy                         13
New-SPEnterpriseSearchRankingModel                                       12
Set-SPAppSiteDomain                                                      12
New-SPOnlineApplicationPrincipalManagementServiceApplicationProxy        12
Restart-SPAppInstanceJob                                                 11
Remove-SPAlternateURL                                                    10
New-SPEnterpriseSearchResultSource                                       10
New-SPEnterpriseSearchContentEnrichmentConfiguration                      9
Update-SPHelp                                                             8
Get-SPTranslationThrottlingSetting                                        7
Set-SPTranslationThrottlingSetting                                        7
New-SPTrustedRootAuthority                                                7
Remove-SPWOPISuppressionSetting                                           7
Remove-SPSecureStoreApplication                                           7
Move-SPSocialComment                                                      7
New-SPWebApplicationExtension                                             6
Set-SPTopologyServiceApplicationProxy                                     5
Set-SPEnterpriseSearchResultSource                                        5
Remove-SPEnterpriseSearchResultSource                                     5
New-SPEnterpriseSearchMetadataMapping                                     4
New-SPEnterpriseSearchContentProcessingComponent                          4
New-SPTrustedServiceTokenIssuer                                           3
Set-SPEnterpriseSearchFileFormatState                                     3
Import-SPSiteSubscriptionBusinessDataCatalogConfig                        3
New-SPSecureStoreTargetApplication                                        3
New-SPEnterpriseSearchLinksDatabase                                       3
New-SPWordConversionServiceApplication                                    3
New-SPRequestManagementRuleCriteria                                       3

PS C:\>

Microsoft SQL Server 2014 PowerShell cmdlet popularity

$
0
0

If you follow the blog, you probably saw a little PowerShell script I published a while back to measure the popularity of the cmdlets in a certain module using a Bing search. As an example, that blog showed the popularity of the cmdlets in the SmbShare module.

Now I got curious about how the cmdlets in other modules would rank, so I spun up some Azure virtual machines to try some other modules. I decided to try the Microsoft SQL Server 2014 main module (named SQLPS).

image

The results are listed below:

PS C:\> .\PopularCmdlet.ps1 | Sort BingCount -Descending | FT -AutoSize

CmdletName                               BingCount
----------                               ---------
Invoke-Sqlcmd                                70700
Backup-SqlDatabase                           34100
Restore-SqlDatabase                          20300
Get-SqlDatabase                               7250
Invoke-PolicyEvaluation                       7170
Enable-SqlAlwaysOn                            5460
Add-SqlAvailabilityDatabase                   4230
Test-SqlAvailabilityGroup                     4050
Get-SqlInstance                               3850
Encode-SqlName                                3040
Set-SqlAvailabilityReplica                    2970
Test-SqlAvailabilityReplica                   2680
Join-SqlAvailabilityGroup                     2350
Switch-SqlAvailabilityGroup                   2330
Test-SqlDatabaseReplicaState                  2250
Set-SqlHADREndpoint                           2230
Set-SqlAvailabilityGroupListener              1930
Remove-SqlAvailabilityDatabase                1920
Convert-UrnToPath                             1790
Decode-SqlName                                1690
Disable-SqlAlwaysOn                           1370
Remove-SqlAvailabilityReplica                 1290
Set-SqlAvailabilityGroup                      1100
Suspend-SqlAvailabilityDatabase               1070
Resume-SqlAvailabilityDatabase                1050
Remove-SqlAvailabilityGroup                    889
Add-SqlAvailabilityGroupListenerStaticIp        50
New-SqlBackupEncryptionOption                   34
Get-SqlSmartAdmin                               34
Set-SqlSmartAdmin                               31
Get-SqlCredential                               28
Set-SqlCredential                               25
Remove-SqlCredential                            23
Set-SqlAuthenticationMode                       20
Test-SqlSmartAdmin                              18
Start-SqlInstance                               15
Stop-SqlInstance                                15
Set-SqlNetworkConfiguration                     10
Add-SqlFirewallRule                              9
Remove-SqlFirewallRule                           9
New-SqlAvailabilityGroup                         5
New-SqlAvailabilityGroupListener                 4
New-SqlHADREndpoint                              3
New-SqlCredential                                3
New-SqlAvailabilityReplica                       3

PS C:\>

Windows Server 2012 R2 Storage PowerShell cmdlet popularity

$
0
0

If you follow the blog, you probably saw a little PowerShell script I published a while back to measure the popularity of the cmdlets in a certain module using a Bing search. As an example, that blog showed the popularity of the cmdlets in the SmbShare module.

Now I got curious about how the cmdlets in other modules would rank, so I spun up some Azure virtual machines to try some other modules. I decided to try the Storage module in Windows Server 2012 R2 (named simply Storage).

image

The results are listed below:

PS C:\> .\PopularCmdlet.ps1 | Sort BingCount -Descending | FT -AutoSize

CmdletName                             BingCount
----------                             ---------
New-Volume                                688000
New-Partition                             403000
Set-Volume                                268000
Get-Volume                                156000
Format-Volume                             111000
Set-Partition                              82300
Set-Disk                                   73700
Get-Disk                                   72500
Flush-Volume                               71300
Resize-Partition                           66000
Clear-Disk                                 65900
Repair-Volume                              63400
Get-Partition                              62100
Initialize-Disk                            56000
Update-Disk                                52000
Remove-Partition                           49000
Optimize-Volume                            38700
New-VirtualDisk                            29500
Mount-DiskImage                            25700
Get-VirtualDisk                            17900
Get-PhysicalDisk                           17400
Repair-VirtualDisk                         14600
Remove-PhysicalDisk                        11200
Set-VirtualDisk                            10600
Remove-VirtualDisk                          9340
Set-PhysicalDisk                            8250
Get-DiskImage                               7220
New-StoragePool                             7130
Dismount-DiskImage                          7080
Initialize-Volume                           7040
Get-StoragePool                             5860
Set-FileStorageTier                         4810
Resize-VirtualDisk                          4620
Get-StorageEnclosure                        4380
Get-StorageReliabilityCounter               4340
Connect-VirtualDisk                         4340
Add-PartitionAccessPath                     4330
Set-StoragePool                             4250
Get-StorageSubSystem                        4170
Get-StorageTier                             4080
Get-InitiatorPort                           4060
Set-FileIntegrity                           4030
Get-FileStorageTier                         3930
Add-PhysicalDisk                            3930
Reset-PhysicalDisk                          3790
Get-FileIntegrity                           3780
Clear-FileStorageTier                       3620
Get-StorageProvider                         3500
Set-ResiliencySetting                       3490
Get-PartitionSupportedSize                  3440
Get-MaskingSet                              3360
Unregister-StorageSubsystem                 3270
Repair-FileIntegrity                        3250
Add-InitiatorIdToMaskingSet                 3200
Set-StorageSubSystem                        3180
Remove-StoragePool                          3160
Get-StorageJob                              2910
Get-InitiatorId                               48
Get-ResiliencySetting                         47
Remove-PartitionAccessPath                    47
Get-TargetPort                                46
Add-TargetPortToMaskingSet                    45
Get-StorageTierSupportedSize                  44
Get-StorageSetting                            39
Resize-StorageTier                            39
Disconnect-VirtualDisk                        36
Get-StorageNode                               36
Set-StorageSetting                            35
Get-TargetPortal                              35
Get-OffloadDataTransferSetting                35
Set-StorageProvider                           32
Enable-PhysicalDiskIndication                 30
Add-VirtualDiskToMaskingSet                   29
Set-InitiatorPort                             29
Set-StorageTier                               27
Remove-StorageTier                            26
Remove-MaskingSet                             26
Remove-InitiatorIdFromMaskingSet              25
Get-VirtualDiskSupportedSize                  25
Remove-InitiatorId                            25
Remove-VirtualDiskFromMaskingSet              25
Get-SupportedFileSystems                      24
Hide-VirtualDisk                              24
Get-VolumeCorruptionCount                     23
Rename-MaskingSet                             23
Get-VolumeScrubPolicy                         22
Write-VolumeCache                             22
Get-SupportedClusterSizes                     21
Enable-StorageEnclosureIdentification         21
Remove-TargetPortFromMaskingSet               20
Get-StorageEnclosureVendorData                17
Set-VolumeScrubPolicy                         16
Disable-PhysicalDiskIndication                 9
Disable-StorageEnclosureIdentification         9
Register-StorageSubsystem                      8
Reset-StorageReliabilityCounter                7
Write-FileSystemCache                          7
Update-StoragePool                             4
New-VirtualDiskSnapshot                        3
New-VirtualDiskClone                           3
Update-StorageProviderCache                    3
Update-HostStorageCache                        3
New-MaskingSet                                 3
New-StorageSubsystemVirtualDisk                3
Show-VirtualDisk                               3
New-StorageTier                                3
Get-PhysicalDiskStorageNodeView                2
Get-PhysicalDiskSNV                            1

PS C:\>


New PowerShell cmdlets in Windows Server 2016 TP2 (compared to Windows Server 2012 R2)

$
0
0

 

1. State the problem

 

With the release of Windows Server 2016 TP2 a few weeks ago, I was wondering what new PowerShell cmdlets are now included (when you compare to Windows Server 2012 R2). However, the list of cmdlets is so long now that it is hard to spot the differences by hand.

However, there a cmdlet in PowerShell to show all the cmdlets available (Get-Command) and a little bit of programming would make it easy to find out what are the main differences. So I set out to collect the data and compare the list.

 

DISCLAIMER: As you probably know already, the Technical Preview is subject to change so all the information about Windows Server 2016 TP2 is preliminary and may not make it into the final product. Use with care, your mileage may vary, not available in all areas, some restrictions apply, professional PowerShell operator on a closed Azure VM course, do not attempt.

 

2. Gather the data

 

First, I needed the list of cmdlets from both versions of the operating system. That was actually pretty easy to gather, with a little help from Azure. I basically provisioned two Azure VM, one running Windows Server 2012 R2 and one running Windows Server 2016 Technical Preview 2 (yes, TP2 is now available in the regular Azure VM image gallery).

Second, I installed all of the Remote Server Administration Tools (RSAT) on both versions. That loads the PowerShell modules used for managing features that are not installed by default, like Failover Cluster or Storage Replica.

Finally, I ran a simple cmdlet to gather the list from Get-Command and save it to an XML file. This made it easier to put all the data I needed in a single place (my desktop machine running Windows 10 Insider Preview). Here's a summary of what it took:

  • Create WS 2012 R2 Azure VM
  • Install RSAT in the WS 2012 R2 VM
    • Get-WindowsFeature RSAT* | Install-WindowsFeature
  • Capture XML file with all the WS 2012 R2 cmdlet information
    • Get-Command | Select * | Export-CliXml C:\WS2012R2Cmdlets.XML
  • Create WS 2016 TP2 Azure VM
  • Install RSAT in the WS 2016 TP2 VM
    • Get-WindowsFeature RSAT* | Install-WindowsFeature

  • Capture XML file with all the WS 2016 TP2 cmdlet information
    • Get-Command | Select * | Export-CliXml C:\WS2016TP2Cmdlets.XML

 

3. Process the data

 

With the two XML files at hand, all I had left to do was to compare them to produce a good list of what's new. The first attempt resulted in a long list that was hard to understand, so I decided to do it module by module.

The code starts by creating a combined list of modules from both operating systems. Then it builds a dictionary of all cmdlets for a given module, assigning the value 1 if it's in WS 2012 R2, 2 if it's in WS 2016 TP2 and 3 if it's in both.

Then I would show the total number of cmdlets per module per OS, then number of new cmdlets and the actual list of new cmdlets. Since the goal was to publish this blog, I actually wrote the script to format the output as an HTML table. Quite handy :-).

 

4. Show the results

 

Finally, here is resulting table with all the new PowerShell cmdlets (by module) in Windows Server 2016 TP2, compared to Windows Server 2012. Enjoy!

 

ModuleNew CmdletsWS 2016 TP2
Cmdlets
WS 2012 R2
Cmdlets
03838
ActiveDirectory0147147
ADRMSAdmin02121
AppLocker055
Appx8146
+ Add-AppxVolume
+ Dismount-AppxVolume
+ Get-AppxDefaultVolume
+ Get-AppxVolume
+ Mount-AppxVolume
+ Move-AppxPackage
+ Remove-AppxVolume
+ Set-AppxDefaultVolume
BestPractices044
BitLocker01313
BitsTransfer088
BranchCache03232
CimCmdlets01414
CIPolicy110
+ ConvertFrom-CIPolicy
ClusterAwareUpdating01717
ConfigCI10100
+ Edit-CIPolicyRule
+ Get-CIPolicy
+ Get-CIPolicyInfo
+ Get-SystemDriver
+ Merge-CIPolicy
+ New-CIPolicy
+ New-CIPolicyRule
+ Remove-CIPolicyRule
+ Set-HVCIOptions
+ Set-RuleOption
Defender11110
+ Add-MpPreference
+ Get-MpComputerStatus
+ Get-MpPreference
+ Get-MpThreat
+ Get-MpThreatCatalog
+ Get-MpThreatDetection
+ Remove-MpPreference
+ Remove-MpThreat
+ Set-MpPreference
+ Start-MpScan
+ Update-MpSignature
DFSN02323
DFSR34542
+ Get-DfsrDelegation
+ Grant-DfsrDelegation
+ Revoke-DfsrDelegation
DhcpServer0121121
DirectAccessClientComponents01111
Dism44339
+ Add-WindowsCapability
+ Expand-WindowsCustomDataImage
+ Get-WindowsCapability
+ Remove-WindowsCapability
DnsClient01717
DnsServer21122101
+ Add-DnsServerClientSubnet
+ Add-DnsServerQueryResolutionPolicy
+ Add-DnsServerRecursionScope
+ Add-DnsServerZoneScope
+ Add-DnsServerZoneTransferPolicy
+ Disable-DnsServerPolicy
+ Enable-DnsServerPolicy
+ Get-DnsServerClientSubnet
+ Get-DnsServerQueryResolutionPolicy
+ Get-DnsServerRecursionScope
+ Get-DnsServerZoneScope
+ Get-DnsServerZoneTransferPolicy
+ Remove-DnsServerClientSubnet
+ Remove-DnsServerQueryResolutionPolicy
+ Remove-DnsServerRecursionScope
+ Remove-DnsServerZoneScope
+ Remove-DnsServerZoneTransferPolicy
+ Set-DnsServerClientSubnet
+ Set-DnsServerQueryResolutionPolicy
+ Set-DnsServerRecursionScope
+ Set-DnsServerZoneTransferPolicy
EventTracingManagement14140
+ Add-EtwTraceProvider
+ Get-AutologgerConfig
+ Get-EtwTraceProvider
+ Get-EtwTraceSession
+ New-AutologgerConfig
+ New-EtwTraceSession
+ Remove-AutologgerConfig
+ Remove-EtwTraceProvider
+ Remove-EtwTraceSession
+ Send-EtwTraceSession
+ Set-AutologgerConfig
+ Set-EtwTraceProvider
+ Set-EtwTraceSession
+ Start-AutologgerConfig
FailoverClusters28482
+ New-ClusterNameAccount
+ Update-ClusterFunctionalLevel
GroupPolicy02929
HgsClient11110
+ Export-HgsGuardian
+ Get-HgsAttestationBaselinePolicy
+ Get-HgsClientConfiguration
+ Get-HgsGuardian
+ Grant-HgsKeyProtectorAccess
+ Import-HgsGuardian
+ New-HgsGuardian
+ New-HgsKeyProtector
+ Remove-HgsGuardian
+ Revoke-HgsKeyProtectorAccess
+ Set-HgsClientConfiguration
Hyper-V26204178
+ Add-VMGroupMember
+ Add-VMSwitchTeamMember
+ Add-VMTPM
+ Disable-VMConsoleSupport
+ Enable-VMConsoleSupport
+ Get-VHDSet
+ Get-VHDSnapshot
+ Get-VMGroup
+ Get-VMHostCluster
+ Get-VMSwitchTeam
+ Get-VMTPM
+ Get-VMVideo
+ New-VMGroup
+ Optimize-VHDSet
+ Remove-VHDSnapshot
+ Remove-VMGroup
+ Remove-VMGroupMember
+ Remove-VMSwitchTeamMember
+ Rename-VMGroup
+ Set-VMHostCluster
+ Set-VMSwitchTeam
+ Set-VMTPM
+ Set-VMVideo
+ Start-VMTrace
+ Stop-VMTrace
+ Update-VMVersion
IISAdministration17170
+ Get-IISAppPool
+ Get-IISConfigCollectionItem
+ Get-IISConfigElement
+ Get-IISConfigSection
+ Get-IISConfigValue
+ Get-IISServerManager
+ Get-IISSite
+ New-IISConfigCollectionItem
+ New-IISSite
+ Remove-IISConfigCollectionItem
+ Remove-IISSite
+ Reset-IISServerManager
+ Set-IISConfigValue
+ Start-IISCommitDelay
+ Start-IISSite
+ Stop-IISCommitDelay
+ Stop-IISSite
International01818
iSCSI01313
IscsiTarget02828
ISE033
Kds066
Microsoft.PowerShell.Archive220
+ Compress-Archive
+ Expand-Archive
Microsoft.PowerShell.Core56055
+ Debug-Job
+ Enter-PSHostProcess
+ Exit-PSHostProcess
+ Get-PSHostProcessInfo
+ Register-ArgumentCompleter
Microsoft.PowerShell.Diagnostics055
Microsoft.PowerShell.Host022
Microsoft.PowerShell.Management48682
+ Clear-RecycleBin
+ Get-Clipboard
+ Get-ItemPropertyValue
+ Set-Clipboard
Microsoft.PowerShell.ODataUtils110
+ Export-ODataEndpointProxy
Microsoft.PowerShell.Security01313
Microsoft.PowerShell.Utility1110594
+ ConvertFrom-String
+ Convert-String
+ Debug-Runspace
+ Disable-RunspaceDebug
+ Enable-RunspaceDebug
+ Format-Hex
+ Get-Runspace
+ Get-RunspaceDebug
- GetStreamHash
+ New-Guid
+ New-TemporaryFile
+ Wait-Debugger
+ Write-Information
Microsoft.WSMan.Management01313
MMAgent055
MsDtc04141
NetAdapter46864
+ Disable-NetAdapterPacketDirect
+ Enable-NetAdapterPacketDirect
+ Get-NetAdapterPacketDirect
+ Set-NetAdapterPacketDirect
NetConnection022
NetEventPacketCapture02323
NetLbfo01313
NetNat01313
NetQos044
NetSecurity08585
NetSwitchTeam077
NetTCPIP03434
NetWNV01919
NetworkConnectivityStatus044
NetworkController1411410
+ Add-NetworkControllerNode
+ Clear-NetworkControllerNodeContent
+ Disable-NetworkControllerNode
+ Enable-NetworkControllerNode
+ Export-NetworkController
+ Get-NetworkController
+ Get-NetworkControllerCanaryConfiguration
+ Get-NetworkControllerCluster
+ Get-NetworkControllerCredential
+ Get-NetworkControllerDevice
+ Get-NetworkControllerDeviceGroupingTestConfiguration
+ Get-NetworkControllerDeviceGroups
+ Get-NetworkControllerDeviceGroupUsage
+ Get-NetworkControllerDeviceUsage
+ Get-NetworkControllerDiagnostic
+ Get-NetworkControllerDiscoveredTopology
+ Get-NetworkControllerExternalTestRule
+ Get-NetworkControllerFabricRoute
+ Get-NetworkControllerGoalTopology
+ Get-NetworkControllerInterface
+ Get-NetworkControllerInterfaceUsage
+ Get-NetworkControllerIpPool
+ Get-NetworkControllerIpPoolStatistics
+ Get-NetworkControllerIpSubnetStatistics
+ Get-NetworkControllerLogicalNetwork
+ Get-NetworkControllerLogicalSubnet
+ Get-NetworkControllerMonitoringService
+ Get-NetworkControllerNode
+ Get-NetworkControllerPhysicalHostInterfaceParameter
+ Get-NetworkControllerPhysicalHostParameter
+ Get-NetworkControllerPhysicalSwitchCpuUtilizationParameter
+ Get-NetworkControllerPhysicalSwitchInterfaceParameter
+ Get-NetworkControllerPhysicalSwitchMemoryUtilizationParameter
+ Get-NetworkControllerPhysicalSwitchParameter
+ Get-NetworkControllerPSwitch
+ Get-NetworkControllerPublicIpAddress
+ Get-NetworkControllerServer
+ Get-NetworkControllerServerInterface
+ Get-NetworkControllerSwitchBgpPeer
+ Get-NetworkControllerSwitchBgpRouter
+ Get-NetworkControllerSwitchConfig
+ Get-NetworkControllerSwitchNetworkRoute
+ Get-NetworkControllerSwitchPort
+ Get-NetworkControllerSwitchPortChannel
+ Get-NetworkControllerSwitchVlan
+ Get-NetworkControllerTopologyConfiguration
+ Get-NetworkControllerTopologyDiscoveryStatistics
+ Get-NetworkControllerTopologyLink
+ Get-NetworkControllerTopologyNode
+ Get-NetworkControllerTopologyTerminationPoint
+ Get-NetworkControllerTopologyValidationReport
+ Get-NetworkControllerVirtualInterface
+ Get-NetworkControllerVirtualNetworkUsage
+ Get-NetworkControllerVirtualPort
+ Get-NetworkControllerVirtualServer
+ Get-NetworkControllerVirtualServerInterface
+ Get-NetworkControllerVirtualSwitch
+ Get-NetworkControllerVirtualSwitchPortParameter
+ Import-NetworkController
+ Install-NetworkController
+ Install-NetworkControllerCluster
+ New-NetworkControllerCanaryConfiguration
+ New-NetworkControllerCredential
+ New-NetworkControllerDevice
+ New-NetworkControllerDeviceGroupingTestConfiguration
+ New-NetworkControllerDeviceGroups
+ New-NetworkControllerExternalTestRule
+ New-NetworkControllerInterface
+ New-NetworkControllerIpPool
+ New-NetworkControllerLogicalNetwork
+ New-NetworkControllerMonitoringService
+ New-NetworkControllerNodeObject
+ New-NetworkControllerPhysicalHostInterfaceParameter
+ New-NetworkControllerPhysicalHostParameter
+ New-NetworkControllerPhysicalSwitchCpuUtilizationParameter
+ New-NetworkControllerPhysicalSwitchInterfaceParameter
+ New-NetworkControllerPhysicalSwitchMemoryUtilizationParameter
+ New-NetworkControllerPhysicalSwitchParameter
+ New-NetworkControllerPSwitch
+ New-NetworkControllerPublicIpAddress
+ New-NetworkControllerServer
+ New-NetworkControllerServerInterface
+ New-NetworkControllerSwitchBgpPeer
+ New-NetworkControllerSwitchBgpRouter
+ New-NetworkControllerSwitchNetworkRoute
+ New-NetworkControllerSwitchPortChannel
+ New-NetworkControllerSwitchVlan
+ New-NetworkControllerTopologyLink
+ New-NetworkControllerTopologyNode
+ New-NetworkControllerTopologyTerminationPoint
+ New-NetworkControllerVirtualInterface
+ New-NetworkControllerVirtualPort
+ New-NetworkControllerVirtualServer
+ New-NetworkControllerVirtualServerInterface
+ New-NetworkControllerVirtualSwitch
+ New-NetworkControllerVirtualSwitchPortParameter
+ Remove-NetworkControllerCanaryConfiguration
+ Remove-NetworkControllerCredential
+ Remove-NetworkControllerDevice
+ Remove-NetworkControllerDeviceGroupingTestConfiguration
+ Remove-NetworkControllerDeviceGroups
+ Remove-NetworkControllerExternalTestRule
+ Remove-NetworkControllerFabricRoute
+ Remove-NetworkControllerInterface
+ Remove-NetworkControllerIpPool
+ Remove-NetworkControllerLogicalNetwork
+ Remove-NetworkControllerLogicalSubnet
+ Remove-NetworkControllerNode
+ Remove-NetworkControllerPhysicalSwitchCpuUtilizationParameter
+ Remove-NetworkControllerPhysicalSwitchMemoryUtilizationParameter
+ Remove-NetworkControllerPSwitch
+ Remove-NetworkControllerPublicIpAddress
+ Remove-NetworkControllerServer
+ Remove-NetworkControllerServerInterface
+ Remove-NetworkControllerSwitchBgpPeer
+ Remove-NetworkControllerSwitchBgpRouter
+ Remove-NetworkControllerSwitchNetworkRoute
+ Remove-NetworkControllerSwitchPortChannel
+ Remove-NetworkControllerSwitchVlan
+ Remove-NetworkControllerTopologyLink
+ Remove-NetworkControllerTopologyNode
+ Remove-NetworkControllerTopologyTerminationPoint
+ Remove-NetworkControllerVirtualInterface
+ Remove-NetworkControllerVirtualPort
+ Remove-NetworkControllerVirtualServer
+ Remove-NetworkControllerVirtualServerInterface
+ Remove-NetworkControllerVirtualSwitch
+ Repair-NetworkControllerCluster
+ Set-NetworkController
+ Set-NetworkControllerCluster
+ Set-NetworkControllerDiagnostic
+ Set-NetworkControllerFabricRoute
+ Set-NetworkControllerGoalTopology
+ Set-NetworkControllerLogicalSubnet
+ Set-NetworkControllerNode
+ Set-NetworkControllerSwitchConfig
+ Set-NetworkControllerSwitchPort
+ Set-NetworkControllerTopologyConfiguration
+ Start-NetworkControllerTopologyDiscovery
+ Uninstall-NetworkController
+ Uninstall-NetworkControllerCluster
NetworkLoadBalancingClusters03535
NetworkSwitchManager19190
+ Disable-NetworkSwitchEthernetPort
+ Disable-NetworkSwitchFeature
+ Disable-NetworkSwitchVlan
+ Enable-NetworkSwitchEthernetPort
+ Enable-NetworkSwitchFeature
+ Enable-NetworkSwitchVlan
+ Get-NetworkSwitchEthernetPort
+ Get-NetworkSwitchFeature
+ Get-NetworkSwitchGlobalData
+ Get-NetworkSwitchVlan
+ New-NetworkSwitchVlan
+ Remove-NetworkSwitchEthernetPortIPAddress
+ Remove-NetworkSwitchVlan
+ Restore-NetworkSwitchConfiguration
+ Save-NetworkSwitchConfiguration
+ Set-NetworkSwitchEthernetPortIPAddress
+ Set-NetworkSwitchPortMode
+ Set-NetworkSwitchPortProperty
+ Set-NetworkSwitchVlanProperty
NetworkTransition03434
NFS04242
Nps-6713
- Get-NpsRemediationServer
- Get-NpsRemediationServerGroup
- New-NpsRemediationServer
- New-NpsRemediationServerGroup
- Remove-NpsRemediationServer
- Remove-NpsRemediationServerGroup
PackageManagement10100
+ Find-Package
+ Get-Package
+ Get-PackageProvider
+ Get-PackageSource
+ Install-Package
+ Register-PackageSource
+ Save-Package
+ Set-PackageSource
+ Uninstall-Package
+ Unregister-PackageSource
PcsvDevice495
+ Clear-PcsvDeviceLog
+ Get-PcsvDeviceLog
+ Set-PcsvDeviceNetworkConfiguration
+ Set-PcsvDeviceUserPassword
Pester20200
+ AfterAll
+ AfterEach
+ Assert-MockCalled
+ Assert-VerifiableMocks
+ BeforeAll
+ BeforeEach
+ Context
+ Describe
+ Get-MockDynamicParameters
+ Get-TestDriveItem
+ In
+ InModuleScope
+ Invoke-Mock
+ Invoke-Pester
+ It
+ Mock
+ New-Fixture
+ Set-DynamicParameterVariables
+ Setup
+ Should
PKI01717
PnpDevice440
+ Disable-PnpDevice
+ Enable-PnpDevice
+ Get-PnpDevice
+ Get-PnpDeviceProperty
PowerShellGet11110
+ Find-Module
+ Get-InstalledModule
+ Get-PSRepository
+ Install-Module
+ Publish-Module
+ Register-PSRepository
+ Save-Module
+ Set-PSRepository
+ Uninstall-Module
+ Unregister-PSRepository
+ Update-Module
PrintManagement02222
PSDesiredStateConfiguration51712
+ Connect-DscConfiguration
+ Find-DscResource
+ Get-DscConfigurationStatus
+ Invoke-DscResource
+ Publish-DscConfiguration
PSDiagnostics01010
PSReadline550
+ Get-PSReadlineKeyHandler
+ Get-PSReadlineOption
+ Remove-PSReadlineKeyHandler
+ Set-PSReadlineKeyHandler
+ Set-PSReadlineOption
PSScheduledJob01616
PSWorkflow022
PSWorkflowUtility011
RemoteAccess14121107
+ Add-BgpRouteAggregate
+ Add-VpnSstpProxyRule
+ Clear-BgpRouteFlapDampening
+ Disable-BgpRouteFlapDampening
+ Enable-BgpRouteFlapDampening
+ Get-BgpRouteAggregate
+ Get-BgpRouteFlapDampening
+ Get-VpnSstpProxyRule
+ New-VpnSstpProxyRule
+ Remove-BgpRouteAggregate
+ Remove-VpnSstpProxyRule
+ Set-BgpRouteAggregate
+ Set-BgpRouteFlapDampening
+ Set-VpnSstpProxyRule
RemoteDesktop57873
+ Export-RDPersonalSessionDesktopAssignment
+ Get-RDPersonalSessionDesktopAssignment
+ Import-RDPersonalSessionDesktopAssignment
+ Remove-RDPersonalSessionDesktopAssignment
+ Set-RDPersonalSessionDesktopAssignment
ScheduledTasks01919
SecureBoot055
ServerCore022
ServerManager077
ServerManagerTasks01111
ShieldedVMDataFile330
+ Import-ShieldingDataFile
+ New-VolumeIDQualifier
+ Protect-ShieldingDataFile
ShieldedVMTemplate110
+ Protect-ServerVHDX
SmbShare03535
SmbWitness033
SoftwareInventoryLogging01111
StartScreen033
Storage32140108
+ Block-FileShareAccess
+ Clear-StorageDiagnosticInfo
+ Debug-FileShare
+ Debug-StorageSubSystem
+ Disable-PhysicalDiskIdentification
+ Disable-StorageDiagnosticLog
+ Enable-PhysicalDiskIdentification
+ Enable-StorageDiagnosticLog
+ Get-DedupProperties
+ Get-DiskSNV
+ Get-DiskStorageNodeView
+ Get-FileShare
+ Get-FileShareAccessControlEntry
+ Get-StorageAdvancedProperty
+ Get-StorageDiagnosticInfo
+ Get-StorageEnclosureSNV
+ Get-StorageEnclosureStorageNodeView
+ Get-StorageFaultDomain
+ Get-StorageFileServer
+ Grant-FileShareAccess
+ New-FileShare
+ New-StorageFileServer
+ Optimize-StoragePool
+ Remove-FileShare
+ Remove-StorageFileServer
+ Revoke-FileShareAccess
+ Set-FileShare
+ Set-StorageFileServer
+ Start-StorageDiagnosticLog
+ Stop-StorageDiagnosticLog
+ Stop-StorageJob
+ Unblock-FileShareAccess
StorageQoS660
+ Get-StorageQoSFlow
+ Get-StorageQoSPolicy
+ Get-StorageQoSVolume
+ New-StorageQoSPolicy
+ Remove-StorageQoSPolicy
+ Set-StorageQoSPolicy
StorageReplica11110
+ Get-SRGroup
+ Get-SRPartnership
+ New-SRGroup
+ New-SRPartnership
+ Remove-SRGroup
+ Remove-SRPartnership
+ Set-SRGroup
+ Set-SRPartnership
+ Suspend-SRGroup
+ Sync-SRGroup
+ Test-SRTopology
TLS374
+ Disable-TlsCipherSuite
+ Enable-TlsCipherSuite
+ Get-TlsCipherSuite
TroubleshootingPack022
TrustedPlatformModule01111
UpdateServices41612
+ Add-WsusDynamicCategory
+ Get-WsusDynamicCategory
+ Remove-WsusDynamicCategory
+ Set-WsusDynamicCategory
UserAccessLogging01414
VpnClient01919
Wdac01212
WebAdministration08080
Whea022
WindowsDeveloperLicense033
WindowsErrorReporting033
WindowsSearch022

 

5. Share the code

 

For those wondering about the script I used to compile the results, here it goes.

#
# Enumerating all the modules from both OS versions
#

# Load XML files into memory: $Files[0] and $Files[1]

$Files
= ( (Import-Clixml"C:\WS2012R2Cmdlets.XML"),
           (Import-Clixml"C:\WS2016TP2Cmdlets.XML") )

# Create empty dictionary for modules


$ModuleDict
= @{}

# Loop through the two files to gather module info

$Files
|% {
  $_|GroupModuleName|SortName|% {
    $Module=$_.Name

    # If found, increase count. If not, add to dictionary


    If
($ModuleDict.ContainsKey($Module)) {
      $ModuleDict.$Module++
    } Else {
      $ModuleDict.Add($Module,1)
    } # End If

  }
# End Import

}
# End $Files

#
# Enumerate the cmdlets in every module
#

# Add the HTML table header

Write-Host
"<table border=1><tr><td><b>Module</b></td><td>New Cmdlets</td><td>WS 2016 TP2</td><td>WS 2012 R2</td></tr>"

# Loop through the modules in the dictionary

$ModuleDict
.GetEnumerator() |SortName|% {

  # Initialize variables for a new module


  $Module
=$_.Name
  $VersionCount= (0,0)
  $CmdletDict= @{}

  # Loop through the two files, filtering by module

  0
..1|% {

    $WSVersion
=$_
    $Files[$_]|?ModuleName-eq$Module|% {

      $Cmdlet
=$_.Name

      # Count cmdlets by module for each OS version

      $VersionCount
[$WSVersion]++

      # Increase per-cmdlet value by 1 (WS2012R2) or by 2 (WS2016TP2)
      # If cmdlet exists in both OSes, value will be 3

      If
($CmdletDict.ContainsKey($Cmdlet)) {
        $CmdletDict.$Cmdlet+= ($WSVersion+1)
      } Else {
        $CmdletDict.Add($Cmdlet, ($WSVersion+1))
      } # End If

    }
# End %

  }
# End 0..1

  #
  # Output the list of cmdlets that changed in every module
  #

  # Copy data to single variables for easy use with Write-Host
  
  $WS0
=$VersionCount[0]
  $WS1=$VersionCount[1]
  $Dif=$WS1-$WS0
  $CrLf="<BR>"+[char]10+[char]13

  # Write HTML table row with module summary information

  Write-Host
"<tr><td><b>$Module</b></td><td align=`"right`">$Dif</td><td align=`"right`">$WS1</td><td align=`"right`">$WS0</td></tr>" 

  # If there are cmdlets in the module

  If
($CmdletDict.Count -gt0) {

    # Gather all new and removed cmdlets in a variable

    $CmdletList
=""
    $CmdletDict.GetEnumerator() |? {$_.Value -eq2-or$_.Value -eq1} |SortName|% {

      # 1 means removed cmdlet. 2 means new cmdlet

      $Name
=$_.Name
      If ($_.Value -eq1) {
        $CmdletList+="- $Name"+$CrLf
      } else {
        $CmdletList+="+ $Name"+$CrLf
      } # End If

    }
# End Enumerator

    # If new or removed exist, write another HTML table row

    If
($CmdletList-ne"") {
      Write-Host"<tr><td colspan=4>$CmdletList</td></tr>"
    } # End If

  }
# End if

} # End Module

# Write HTML table end. All done.

Write-Host
"</table>"

 

Using PowerShell and Excel PivotTables to understand the files on your disk

$
0
0

 

Introduction

I am a big fan of two specific technologies that usually don’t get mentioned together: PowerShell and Excel PivotTables. It started when I was explaining PivotTables to someone and the main issue I had was finding a good set of example data that is familiar to everyone. That’s when it hit me. People using a computer have tons of files stored in their local disks and most don’t have a clue about those files. That’s the perfect example!

So I set out to gather the steps to gather information about your local files and extract the most information about it.

 

Step 1 – List the questions you need to answer

To start, here are a few questions you might have about the files in your local hard drive:

  • Which folder is storing the most files or using up the most space in your local disk?
  • What kind of data (pictures, music, video) is using the most space in your local disk?
  • What is the average size of the pictures you took this year?
  • How much of the files in your local disk was created in the last 30 days? Or this year?
  • Which day of the week do you create the most new pictures? Or PowerPoint presentations?

Now you could write a PowerShell script to answer any of those questions. It would in itself be a great programming exercise, but some would be quite tricky to code. However, those questions are just the tip of the iceberg. Given that dataset, you could come up with many, many more. So the point is that you would use Excel PivotTables to explore the data set and come up with the answers while interacting with it.

 

Step 2 – Gather the required raw data

In any work with PivotTables and BI (Business Inteligence) in general, you need to identify the raw data that you can use to produce the answers to your questions. As you problably already figured out, we’ll use PowerShell to query the file system and extract that data. Using the Get-ChildItem (more commonly known by its alias: DIR), you can get information about each folder and file on the disk.

Now with possibly hundreds of thousands of files, you want to make sure you gather only the necessary data. That will make it faster to obtain and will give Excel less data to chew on, which is always a good thing.Here’s what you could use (running as an administrator), to get information:

Dir C:\ -Recurse | Select FullName, Extension, Length, CreationTime, Attributes

Next, you want to make sure you transform into into a format that Excel can consume. Luckly, PowerShell has a cmdlet to transform data into Comma-Separated Values, also known as CSV. You need to also include something to avoid any permission errors while accessing the data and output the results to a file, so we can load it into Excel. Here’s the final command line:

Dir \ -Recurse -ErrorAction SilentlyContinue | Select FullName, Extension, Length, CreationTime, Attributes | ConvertTo-Csv  -NoTypeInformation | Out-File C:\AllFiles.csv

That command will take several minutes to run, depending on the number of files on your disk, the speed of the disk and the speed of your computer. The resulting file can get quite big as well. In my case, it took a few minutes and the resulting file size was 135,359,136 bytes (or around 130MB).

 

Step 3 – Load into Excel and build the right table

With the AllFiles.csv file available, we can now load the raw data in Excel and start working with it. Just open Excel (I’m using Excel 2016 Preview) and load the CSV file. When importing, make sure to select “Delimited” in the first page of the wizard and check the “comma” checkbox in the second page.

clip_image001

clip_image002

Excel loaded the data and I ended up with 412,012 rows (including one row for the header). However the formating was a little lacking…

clip_image003

clip_image004

Next, I applied a format each column for best results. You want to format the Length to a number with comma separators and no decimals. To do that, select the third column and click to format as a number.

clip_image005

You can also use the same process to format the fourth column with a more interesting date format.

clip_image006

Here’s what it looks like at this point.

clip_image007

Last but not least, you want to freeze the top row of the spreadsheet and format the whole think as a table.

clip_image008

clip_image009

clip_image010

Here’s the final look for this phase:

clip_image011

 

Step 4 – Add fields that will help with your questions

While you have most of the data you need readily acessible, it helps to add to your table some additional fields. You could add those to your original PowerShell query, but Excel is probably better equipped to generate those extra columns on the fly.

Also, you might notice the need to add those only after you have played with the data a bit with Excel. That will also give you a chance to brush up on your Excel formula skills. In this example, we will add the following fields to the table:

  • CreatedYear – Year the file was created. Formula =YEAR([@CreationTime])
  • CreatedDays – Days since the file was created. Formula =TODAY()-[@CreationTime]
  • CreatedDow – Day of the week the file was created. Formula = =WEEKDAY([@CreationTime])
  • IsFolder – True if the item is folder, not a file. Formula =NOT(ISERROR(FIND("Directory",[@Attributes])))
  • TopFolder – First folder in the file name. Formula = =IF(ISERROR(FIND("\",[@FullName],4)),"C:\",LEFT([@FullName],FIND("\",[@FullName],4)))

Just insert the extra columns (right click column, click insert) and fill in the title and the formula. Excel will apply the formula to all cells in that column automatically. You will need to reformat the columns for CreatedYear, CreatedDays, CreatedDow to show as regular numbers, without any decimals.

clip_image012

 

Step 5 – Create a Pivot Table

With all the columns in place, you should proceed and create the Pivot Table. Just click on a cell at the table and choose Pivot Table under the “Insert” tab.

clip_image013

That will create an empty PivotTable with all the fields on the table available to you.

clip_image014

Now you just have to drag the fields to one of the four white boxes below the field list: Filters, Columns, Rows or Values. You will have options on how things are summarized (count, sum, average), how to format the data, how fields are sorted, etc.

To start, you can drag TopFolder to the Rows and Length to the Values. You should make adjustments to the “Count of Length” under Values to format as a number with no decimals.

clip_image015

You will also need to change the “More sort options” of the “TopFolder” field to sort on descending order by “Sum of Length”.

clip_image016

To avoid counting folders, you could add the IsFolder field to the filter box and then click on cell B1 to change the filter to false. Here’s what you should get: A sorted list of top-level folders with the number of files in each.

clip_image017

Simply by changing the settings in “Count of Length” to make it a sum, you get the list of top folders with the total size in bytes for each one:

clip_image018

Those two will answer the first question on our list: Which folder is storing the most files or using up the most space in your local disk?

 

Step 6 – Tweak the PivotTable to your heart’s content

Now you have everything you need to slice and dice the data, answering any of the questions posed at the beginning of this blog. Here are a few examples, with specific comments for each one. Top 20 extensions for all the disk. Start with dragging extension to the rows, then filter by TOP 10 and adjust:

clip_image019

So I have a lot used by programs (DLL, EXE), but also a fair amount of bytes used by music (WMA), videos (MP4) and pictures (JPG).

clip_image020

Next I could filter only to files under the C:\Users\ folder, which would exclude the operating system. After that, PowerPoint files jump up to number 4, right after music, videos and pictures.

clip_image021

If I want to look at the size of a particular kind of file, I would filter by that extension and add a few things to the values. To look at statistics of pictures I took this year, I dragged length to the values a few times and adjusted to do count, sum and max. I also moved the “∑ Values” from Columns to Rows. I finished by adding Created Year to the filters and selecting 2015.

clip_image022

Lastly, I looked at the the breakdown of files by the day of the week they were created. I was looking at the total number of files created in a given day of the week, broken down by the top 20 file extension. I had filters for user files only and restricted it also to files created in 2015. I also removed the Grand totals for this one. Apparently I did a lot of my file creation this year on this computer on Thursdays and Fridays.

clip_image023

Finally, here’s a more complex scenario showing a total of files, capacity, oldest year and largest size. I played with changing the default name of the values, which makes the labels a bit more readable. There’s also multiple items in the rows, creating a hieararchy. I’ll let you figure out how to get to this particular view.

clip_image024

 

Conclusion

I hope this post was a good example of all the things you can do with Excel PivotTables. In my view, this gets really powerful if you have an interesting data set to play with, which PowerShell and the file system were glad to provide for us. Let me know if you found this useful and share your experience with file/folder statistics, gathering data sets with PowerShell and PivotTables.

For my next data set, I was thinking about gathering some data about e-mails. There’s another thing that everyone has in large quantities…

Drive Performance Report Generator - PowerShell script using DiskSpd by Arnaud Torres

$
0
0

Arnaud Torres is a Senior Premier Field Engineer at Microsoft in France who sent me the PowerShell script below called "Drive Performance Report Generator".

He created the script to test a wide range of profiles in one run to allow people to build a baseline of their storage using DiskSpd.EXE.

The script is written in PowerShell v1 and was tested on a Windows Server 2008 SP2 (really!), Windows Server 2012 R2 and Windows 10.

It displays results in real time, is highly documented and creates a text report which can be imported as CSV in Excel.

 

Thanks to Arnaud for sharing!

 

----------------------

 

# Drive performance Report Generator

# by Arnaud TORRES

# Microsoft provides script, macro, and other code examples for illustration only, without warranty either expressed or implied, including but not

# limited to the implied warranties of merchantability and/or fitness for a particular purpose. This script is provided 'as is' and Microsoft does not

# guarantee that the following script, macro, or code can be used in all situations.

# Script will stress your computer CPU and storage, be sure that no critical workload is running

 

# Clear screen

Clear

 

write-host "DRIVE PERFORMANCE REPORT GENERATOR" -foregroundcolor green

write-host "Script will stress your computer CPU and storage layer (including network if applciable !), be sure that no critical workload is running" -foregroundcolor yellow

write-host "Microsoft provides script, macro, and other code examples for illustration only, without warranty either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose. This script is provided 'as is' and Microsoft does not guarantee that the following script, macro, or code can be used in all situations." -foregroundcolor darkred

"   "

"Test will use all free space on drive minus 2 GB !"

"If there are less than 4 GB free test will stop"

 

# Disk to test

$Disk = Read-Host 'Which disk would you like to test ? (example : D:)'

# $Disk = "D:"

if ($disk.length -ne 2){"Wrong drive letter format used, please specify the drive as D:"

                         Exit}

if ($disk.substring(1,1) -ne ":"){"Wrong drive letter format used, please specify the drive as D:"

                         Exit}

$disk = $disk.ToUpper()

 

# Reset test counter

$counter = 0

 

# Use 1 thread / core

$Thread = "-t"+(Get-WmiObject win32_processor).NumberofCores

 

# Set time in seconds for each run

# 10-120s is fine

$Time = "-d1"

 

# Outstanding IOs

# Should be 2 times the number of disks in the RAID

# Between  8 and 16 is generally fine

$OutstandingIO = "-o16"

 

# Disk preparation

# Delete testfile.dat if it exists

# The test will use all free space -2GB

 

$IsDir = test-path -path "$Disk\TestDiskSpd"

$isdir

if ($IsDir -like "False"){new-item -itemtype directory -path "$Disk\TestDiskSpd\"}

# Just a little security, in case we are working on a compressed drive ...

compact /u /s $Disk\TestDiskSpd\

 

$Cleaning = test-path -path "$Disk\TestDiskSpd\testfile.dat"

if ($Cleaning -eq "True")

{"Removing current testfile.dat from drive"

  remove-item $Disk\TestDiskSpd\testfile.dat}

 

$Disks = Get-WmiObject win32_logicaldisk

$LogicalDisk = $Disks | where {$_.DeviceID -eq $Disk}

$Freespace = $LogicalDisk.freespace

$FreespaceGB = [int]($Freespace / 1073741824)

$Capacity = $freespaceGB - 2

$CapacityParameter = "-c"+$Capacity+"G"

$CapacityO = $Capacity * 1073741824

 

if ($FreespaceGB -lt "4")

{

       "Not enough space on the Disk ! More than 4GB needed"

       Exit

}

 

write-host " "

$Continue = Read-Host "You are about to test $Disk which has $FreespaceGB GB free, do you wan't to continue ? (Y/N) "

if ($continue -ne "y" -or $continue -ne "Y"){"Test Cancelled !!"

                                        Exit}

 

"   "

"Initialization can take some time, we are generating a $Capacity GB file..."

"  "

 

 

# Initialize outpout file

$date = get-date

 

# Add the tested disk and the date in the output file

"Disque $disk, $date" >> ./output.txt

 

# Add the headers to the output file

“Test N#, Drive, Operation, Access, Blocks, Run N#, IOPS, MB/sec, Latency ms, CPU %" >> ./output.txt

 

# Number of tests

# Multiply the number of loops to change this value

# By default there are : (4 blocks sizes) X (2 for read 100% and write 100%) X (2 for Sequential and Random) X (4 Runs of each)

$NumberOfTests = 64

 

"  "

write-host "TEST RESULTS (also logged in .\output.txt)" -foregroundcolor yellow

 

# Begin Tests loops

 

# We will run the tests with 4K, 8K, 64K and 512K blocks

(4,8,64,512) | % { 

$BlockParameter = ("-b"+$_+"K")

$Blocks = ("Blocks "+$_+"K")

 

# We will do Read tests and Write tests

  (0,100) | % {

      if ($_ -eq 0){$IO = "Read"}

      if ($_ -eq 100){$IO = "Write"}

      $WriteParameter = "-w"+$_

 

# We will do random and sequential IO tests

  ("r","si") | % {

      if ($_ -eq "r"){$type = "Random"}

      if ($_ -eq "si"){$type = "Sequential"}

      $AccessParameter = "-"+$_

 

# Each run will be done 4 times

  (1..4) | % {

     

      # The test itself (finally !!)

         $result = .\diskspd.exe $CapacityPArameter $Time $AccessParameter $WriteParameter $Thread $OutstandingIO $BlockParameter -h -L $Disk\TestDiskSpd\testfile.dat

     

      # Now we will break the very verbose output of DiskSpd in a single line with the most important values

      foreach ($line in $result) {if ($line -like "total:*") { $total=$line; break } }

      foreach ($line in $result) {if ($line -like "avg.*") { $avg=$line; break } }

      $mbps = $total.Split("|")[2].Trim()

      $iops = $total.Split("|")[3].Trim()

      $latency = $total.Split("|")[4].Trim()

      $cpu = $avg.Split("|")[1].Trim()

      $counter = $counter + 1

 

      # A progress bar, for the fun

      Write-Progress -Activity ".\diskspd.exe $CapacityPArameter $Time $AccessParameter $WriteParameter $Thread $OutstandingIO $BlockParameter -h -L $Disk\TestDiskSpd\testfile.dat" -status "Test in progress" -percentComplete ($counter / $NumberofTests * 100)

     

      # Remove comment to check command line ".\diskspd.exe $CapacityPArameter $Time $AccessParameter $WriteParameter $Thread -$OutstandingIO $BlockParameter -h -L $Disk\TestDiskSpd\testfile.dat"

     

      # We output the values to the text file

      “Test $Counter,$Disk,$IO,$type,$Blocks,Run $_,$iops,$mbps,$latency,$cpu"  >> ./output.txt

 

      # We output a verbose format on screen

      “Test $Counter, $Disk, $IO, $type, $Blocks, Run $_, $iops iops, $mbps MB/sec, $latency ms, $cpu CPU"

}

}

}

}

 

Twenty years as a Microsoft Certified Professional – time flies when you’re having fun

$
0
0

 

I just noticed that last week was the 20th anniversary of my first Microsoft certification. I had to travel nearly 500 miles (from Fortaleza to Recife) to reach the closest official testing center available in Brazil in August 1995.

You’re probably thinking that I started by taking the Windows 95 exam, but it was actually the Windows 3.1 exam (which included a lot of MS-DOS 6.x stuff). The Windows 95 exam was my next one, but that only happened over a year later in December 1996.

I went on to take absolutely all of the Windows NT 4.0 and Windows 2000 exams (many of them in their beta version). At that point we had multiple Microsoft Certified Partners in Fortaleza and I worked for one of them.

I continued to take lots of exams even after moved to the US in October 2000 and after I joined Microsoft in October 2002. I only slowed down a bit after joining the Windows Server engineering team in October 2007.

In 2009 I achieved my last certification as a Microsoft Certified Master on SQL Server 2008. That took a few weeks of training, a series of written exams and a final, multi-hour lab exam. Exciting stuff! That also later granted me a charter certification as  Microsoft Certified Solutions Master (Data Platform), Microsoft Certified Solutions Expert (Data Platform) and Microsoft Certified Solutions Associate (SQL Server 2012).

My full list is shown below. In case you’re wondering, the Windows 10 exam (Configuring Windows Devices) is already in development and you can find the details at https://www.microsoft.com/learning/en-us/exam-70-697.aspx.

 

image

image

image

Raw notes from the Storage Developer Conference 2015 (SNIA SDC 2015)

$
0
0

Notes and disclaimers:

  • This blog post contains raw notes for some of the SNIA’s SDC 2015 presentations (SNIA’s Storage Developers Conference 2015)
  • These notes were typed during the talks and they may include typos and my own misinterpretations.
  • Text in the bullets under each talk are quotes from the speaker or text from the speaker slides, not my personal opinion.
  • If you feel that I misquoted you or badly represented the content of a talk, please add a comment to the post.
  • I spent limited time fixing typos or correcting the text after the event. There are only so many hours in a day...
  • I have not attended all sessions (since there are many being delivered at a time, that would actually not be possible :-)…
  • SNIA usually posts the actual PDF decks a few weeks after the event. Attendees have access immediately.
  • You can find the event agenda at http://www.snia.org/events/storage-developer/agenda

 

Understanding the Intel/Micron 3D XPoint Memory
Jim Handy, General Director, Objective Analysis

  • Memory analyst, SSD analyst, blogs: http://thememoryguy.com, http://thessdguy.com
  • Not much information available since the announcement in July: http://newsroom.intel.com/docs/DOC-6713
  • Agenda: What? Why? Who? Is the world ready for it? Should I care? When?
  • What: Picture of the 3D XPoint concept (pronounced 3d-cross-point). Micron’s photograph of “the real thing”.
  • Intel has researched PCM for 45 years. Mentioned in an Intel article at “Electronics” in Sep 28, 1970.
  • The many elements that have been tried shown in the periodic table of elements.
  • NAND laid the path to the increased hierarchy levels. Showed prices of DRAM/NAND from 2001 to 2015. Gap is now 20x.
  • Comparing bandwidth to price per gigabytes for different storage technologies: Tape, HDD, SSD, 3D XPoint, DRAM, L3, L2, L1
  • Intel diagram mentions PCM-based DIMMs (far memory) and DDR DIMMs (near memory).
  • Chart with latency for HDD SAS/SATA, SSD SAS/SATA, SSD NVMe, 3D XPoint NVMe – how much of it is the media, how much is the software stack?
  • 3D Xpoint’s place in the memory/storage hierarchy. IOPS x Access time. DRAM, 3D XPoint (Optane), NVMe SSD, SATA SSD
  • Great gains at low queue depth. 800GB SSD read IOPS using 16GB die. IOPS x queue depth of NAND vs. 3D XPoint.
  • Economic benefits: measuring $/write IOPS for SAS HDD, SATA SSD, PCIe SSD, 3D XPoint
  • Timing is good because: DRAM is running out of speed, NVDIMMs are catching on, some sysadmins understand how to use flash to reduce DRAM needs
  • Timing is bad because: Nobody can make it economically, no software supports SCM (storage class memory), new layers take time to establish Why should I care: better cost/perf ratio, lower power consumption (less DRAM, more perf/server, lower OpEx), in-memory DB starts to make sense
  • When? Micron slide projects 3D XPoint at end of FY17 (two months ahead of CY). Same slide shows NAND production surpassing DRAM production in FY17.
  • Comparing average price per GB compared to the number of GB shipped over time. It takes a lot of shipments to lower price.
  • Looking at the impact in the DRAM industry if this actually happens. DRAM slows down dramatically starting in FY17, as 3D XPoint revenues increase (optimistic).

 

Next Generation Data Centers: Hyperconverged Architectures Impact On Storage
Mark OConnell, Distinguished Engineer, EMC

  • History: Client/Server –> shared SANs –> Scale-Out systems
  • >> Scale-Out systems: architecture, expansion, balancing
  • >> Evolution of the application platform: physical servers à virtualization à Virtualized application farm
  • >> Virtualized application farms and Storage: local storage à Shared Storage (SAN) à Scale-Out Storage à Hyper-converged
  • >> Early hyper-converged systems: HDFS (Hadoop) à JVM/Tasks/HDFS in every node
  • Effects of hyper-converged systems
  • >> Elasticity (compute/storage density varies)
  • >> App management, containers, app frameworks
  • >> Storage provisioning: frameworks (openstack swift/cinder/manila), pure service architectures
  • >> Hybrid cloud enablement. Apps as self-describing bundles. Storage as a dynamically bound service. Enables movement off-prem.

 

Implications of Emerging Storage Technologies on Massive Scale Simulation Based Visual Effects
Yahya H. Mirza, CEO/CTO, Aclectic Systems Inc

  • Steve Jobs quote: "You‘ve got to start with the customer experience and work back toward the technology".
  • Problem 1: Improve customer experience. Higher resolution, frame rate, throughput, etc.
  • Problem 2: Production cost continues to rise.
  • Problem 3: Time to render single frame remains constant.
  • Problem 4: Render farm power and cooling increasing. Coherent shared memory model.
  • How do you reduce customer CapEx/OpEx. Low efficiency: 30% CPU. Prooblem is memory access latency and I/O.
  • Production workflow: modeling, animation/simulation/shading, lighting, rendering, compositing. More and more simulation.
  • Concrete production experiment: 2005. Story boards. Attempt to create a short film. Putting himself in the customer’s shoes. Shot decomposition.
  • Real 3-minute short costs $2 million. Animatic to pitch the project.
  • Character modeling and development. Includes flesh and muscle simulation. A lot of it done procedurally.
  • Looking at Disney’s “Big Hero 6”, DreamWorks’ “Puss in Boots” and Weta’s “The Hobbit”, including simulation costs, frame rate, resolution, size of files, etc.
  • Physically based rendering: global illumination effects, reflection, shadows. Comes down to light transport simulation, physically based materials description.
  • Exemplary VFX shot pipeline. VFX Tool (Houdini/Maya), Voxelized Geometry (OpenVDB), Scene description (Alembic), Simulation Engine (PhysBam), Simulation Farm (RenderFarm), Simulation Output (OpenVDB), Rendering Engine (Mantra), Render Farm (RenderFarm), Output format (OpenEXR), Compositor (Flame), Long-term storage.
  • One example: smoke simulation – reference model smoke/fire VFX. Complicated physical model. Hotspot algorithms: monte-carlo integration, ray-intersection test, linear algebra solver (multigrid).
  • Storage implications. Compute storage (scene data, simulation data), Long term storage.
  • Is public cloud computing viable for high-end VFX?
  • Disney’s data center. 55K cores across 4 geos.
  • Vertically integrated systems are going to be more and more important. FPGAs, ARM-based servers.
  • Aclectic Colossus smoke demo. Showing 256x256x256.
  • We don’t want coherency; we don’t want sharing. Excited about Intel OmniPath.
  • http://www.intel.com/content/www/us/en/high-performance-computing-fabrics/omni-path-architecture-fabric-overview.html

 

How Did Human Cells Build a Storage Engine?
Sanjay Joshi, CTO Life Sciences, EMC

  • Human cell, Nuclear DNA, Transcription and Translation, DNA Structure
  • The data structure: [char(3*10^9) human_genome] strand
  • 3 gigabases [(3*10^9)*2]/8 = ~750MB. With overlaps, ~1GB per cell. 15-70 trillion cells.
  • Actual files used to store genome are bigger, between 10GB and 4TB (includes lots of redundancy).
  • Genome sequencing will surpass all other data types by 2040
  • Protein coding portion is just a small portion of it. There’s a lot we don’t understand.
  • Nuclear DNA: Is it a file? Flat file system, distributed, asynchronous. Search header, interpret, compile, execute.
  • Nuclear DNA properties: Large:~20K genes/cell, Dynamic: append/overwrite/truncate, Semantics: strict, Consistent: No, Metadata: fixed, View: one-to-many
  • Mitochondrial DNA: Object? Distributed hash table, a ring with 32 partitions. Constant across generations.
  • Mitochondrial DNA: Small: ~40 genes/cell, Static: constancy, energy functions, Semantics: single origin, Consistent: Yes, Metadata: system based, View: one-to-one
  • File versus object. Comparing Nuclear DNA and Mitochondrial DNA characteristics.
  • The human body: 7500 names parts, 206 regularly occurring bones (newborns close to 300), ~640 skeletal muscles (320 pairs), 60+ organs, 37 trillion cells. Distributed cluster.
  • Mapping the ISO 7 layers to this system. Picture.
  • Finite state machine: max 10^45 states at 4*10^53 state-changes/sec. 10^24 NOPS (nucleotide ops per second) across biosphere.
  • Consensus in cell biology: Safety: under all conditions: apoptosis. Availability: billions of replicate copies. Not timing dependent: asynchronous. Command completion: 10 base errors in every 10,000 protein translation (10 AA/sec).
  • Object vs. file. Object: Maternal, Static, Haploid. Small, Simple, Energy, Early. File: Maternal and paternal, Diploid. Scalable, Dynamic, Complex. All cells are female first.

 

Move Objects to LTFS Tape Using HTTP Web Service Interface
Matt Starr, Chief Technical Officer, Spectra Logic
Jeff Braunstein, Developer Evangelist, Spectra Logic

  • Worldwide data growth: 2009 = 800 EB, 2015 = 6.5ZB, 2020 = 35ZB
  • Genomics. 6 cows = 1TB of data. They keep it forever.
  • Video data. SD to Full HD to 4K UHD (4.2TB per hours) to 8K UHD. Also kept forever.
  • Intel slide on the Internet minute. 90% of the people of the world never took a picture with anything but a camera phone.
  • IOT - Total digital info create or replicated.
  • $1000 genome scan take 780MB fully compressed. 2011 HiSeq-2000 scanner generates 20TB per month. Typical camera generates 105GB/day.
  • More and more examples.
  • Tape storage is the lowest cost. But it’s also complex to deploy. Comparing to Public and Private cloud…
  • Pitfalls of public cloud – chart of $/PB/day. OpEx per PB/day reaches very high for public cloud.
  • Risk of public cloud: Amazon has 1 trillion objects. If they lose 1% it would 10 billion objects.
  • Risk of public cloud: Nirvanix. VC pulled the plug in September 2013.
  • Cloud: Good: toolkits, naturally WAN friendly, user expectation: put it away.
  • What if: Combine S3/Object with tape. Spectra S3 – Front end is REST, backend is LTFS tape.
  • Cost: $.09/GB. 7.2PB. Potentially a $0.20 two-copy archive.
  • Automated: App or user-built. Semi-Automated: NFI or scripting.
  • Information available at https://developer.spectralogic.com
  • All the tools you need to get started. Including simulator of the front end (BlackPearl) in a VM.
  • S3 commands, plus data to write sequentially in bulk fashion.
  • Configure user for access, buckets.
  • Deep storage browser (source code on GitHub) allows you to browse the simulated storage.
  • SDK available in Java, C#, many others. Includes integration with Visual Studio (demonstrated).
  • Showing sample application. 4 lines of code from the SDK to move a folder to tape storage.
  • Q: Access times when not cached? Hours or minutes. Depends on if the tape is already in the drive. You can ask to pull those to cache, set priorities. By default GET has higher priority than PUT. 28TB or 56TB of cache.
  • Q: Can we use CIFS/NFS? Yes, there is an NFI (Network File Interface) using CIFS/NFS, which talks to the cache machine. Manages time-outs.
  • Q: Any protection against this being used as disk? System monitors health of the tape. Using an object-based interface helps.
  • Q: Can you stage a file for some time, like 24h? There is a large cache. But there are no guarantees on the latency. Keeping it on cache is more like Glacier. What’s the trigger to bring the data?
  • Q: Glacier? Considering support for it. Data policy to move to lower cost, move it back (takes time). Not a lot of product or customers demanding it. S3 has become the standard, not sure if Glacier will be that for archive.
  • Q: Drives are a precious resource. How do you handle overload? By default, reads have precedence over writes. Writes usually can wait.

 

Taxonomy of Differential Compression
Liwei Ren, Scientific Adviser, Trend Micro

  • Mathematical model for describing file differences
  • Lossless data compression categories: data compression (one file), differential compression (two files), data deduplication (multiple files)
  • Purposes: network data transfer acceleration and storage space reduction
  • Areas for DC – mobile phones’ firmware over the air, incremental update of files for security software, file synchronization and transfer over WAN, executable files
  • Math model – Diff procedure: Delta = T – R, Merge procedure: T = R + Delta. Model for reduced network bandwidth, reduced storage cost.
  • Applications: backup, revision control system, patch management, firmware over the air, malware signature update, file sync and transfer, distributed file system, cloud data migration
  • Diff model. Two operations: COPY (source address, size [, destination address] ), ADD (data block, size [, destination address] )
  • How to create the delta? How to encode the delta into a file? How to create the right sequence of COPY/ADD operations?
  • Top task is an effective algorithm to identify common blocks. Not covering it here, since it would take more than half an hour…
  • Modeling a diff package. Example.
  • How do you measure the efficiency of an algorithm? You need a cost model.
  • Categorizing: Local DC - LDC (xdelta, zdelta, bsdiff), Remote DC - RDC (rsync, RDC protocol, tsync), Iterative – IDC (proposed)
  • Categorizing: Not-in-place merging: general files (xdelta, zdelta, bsdiff), executable files (bsdiff, courgette)
  • Categorizing: In place merging: firmware as general files (FOTA), firmware as executable files (FOTA)
  • Topics in depth: LDC vs RDC vs IDC for general files
  • Topics in depth: LDC for executable files
  • Topics in depth: LDC for in-place merging

 

New Consistent Hashing Algorithms for Data Storage
Jason Resch, Software Architect, Cleversafe

  • Introducing a new algorithm for hashing.
  • Hashing is useful. Used commonly is distributed storage, distributed caching.
  • Independent users can coordinate (readers know where writers would write without talking to them).
  • Typically, resizing a Hash Table is inefficient. Showing example.
  • That’s why we need “Stable Hashing”. Showing example. Only a small portion of the keys need to be re-mapped.
  • Stable hashing becomes a necessity when system is stateful and/or transferring state is expensive,
  • Used in Caching/Routing (CARP), DHT/Storage (Gluster, DynamoDB, Cassandra, ceph, openstack)
  • Stable Hashing with Global Namespaces. If you have a file name, you know what node has the data.
  • Eliminates points of contention, no metadata systems. Namespace is fixed, but the system is dynamic.
  • Balances read/write load across nodes, as well as storage utilization across nodes.
  • Perfectly Stable Hashing (Rendezvous Hashing, Consistent Hashing). Precisely weighted (CARP, RUSH, CRUSH).
  • It would be nice to have something that would offer the characteristics of both.
  • Consistent: buckets inserted in random positions. Keys maps to the next node greater than that key. With a new node, only neighbors as disrupted. But neighbor has to send data to new node, might not distribute keys evenly.
  • Rendezvous: Score = Hash (Bucket ID || Key). Bucket with the highest score wins. When adding a new node, some of the keys will move to it. Every node is disrupted evenly.
  • CARP is rendezvous hashing with a twist. It multiples the scores by a “Load Factor” for each node. Allows for some nodes being more capable than others. Not perfectly stable: if node’s weighting changes or node is added, then all load factor must be recomputed.
  • RUSH/CRUSH: Hierarchical tree, with each node assigned a probability to go left/right. CRUSH makes the tree match the fault domains of the system. Efficient to add nodes, but not to remove or re-weight nodes.
  • New algorithm: Weighted Rendezvous Hashing (WRH). Both perfectly stable and precisely weighted.
  • WRH adjusts scores before weighting them. Unlike CARP, scores aren’t relatively scaled.
  • No unnecessary transfer of keys when adding/removing nodes. If adding node or increasing weight on node, other nodes will move keys to it, but nothing else. Transfers are equalized and perfectly efficient.
  • WRH is simple to implement. Whole python code showed in one slide.
  • All the magic is in one line: “Score = 1.0 / -math.log(hash_f)” - Proof of correctness provided for the math inclined.
  • How Cleversafe uses WRH. System is grown by set of devices. Devices have a lifecycle: added, possibly expanded, then retired.
  • Detailed explanation of the lifecycle and how keys move as nodes are added, expanded, retired.
  • Storage Resource Map. Includes weight, hash_seed. Hash seed enables a clever trick to retire device sets more efficiently.
  • Q: How to find data when things are being moved? If clients talk to the old node while keys are being moved. Old node will proxy the request to the new node.

 

Storage Class Memory Support in the Windows Operating System
Neal Christiansen, Principal Development Lead, Microsoft

  • Windows support for non-volatile storage medium with RAM-like performance is a big change.
  • Storage Class Memory (SCM): NVDIMM, 3D XPoint, others
  • Microsoft involved with the standardization efforts in this space.
  • New driver model necessary: SCM Bus Driver, SCM Disk Driver.
  • Windows Goals for SCM: Support zero-copy access, run most user-mode apps unmodified, option for 100% backward compatibility (new types of failure modes), sector granular failure modes for app compat.
  • Applications make lots of assumptions on the underlying storage
  • SCM Storage Drivers will support BTT – Block Translation Table. Provides sector-level atomicity for writes.
  • SCM is disruptive. Fastest performance and application compatibility can be conflicting goals.
  • SCM-aware File Systems for Windows. Volume modes: block mode or DAS mode (chosen at format time).
  • Block Mode Volumes – maintain existing semantics, full application compatibility
  • DAS Mode Volumes – introduce new concepts (memory mapped files, maximizes performance). Some existing functionality is lost. Supported by NTFS and ReFS.
  • Memory Mapped IO in DAS mode. Application can create a memory mapped section. Allowed when volumes resides on SCM hardware and the volume has been formatted for DAS mode.
  • Memory Mapped IO: True zero copy access. BTT is not used. No paging reads or paging writes.
  • Cached IO in DAS Mode: Cache manager creates a DAS-enabled cache map. Cache manager will copy directly between user’s buffer and SCM. Coherent with memory-mapped IO. App will see new failure patterns on power loss or system crash. No paging reads or paging writes.
  • Non-cached IO in DAS Mode. Will send IO down the storage stack to the SCM driver. Will use BTT. Maintains existing storage semantics.
  • If you really want the performance, you will need to change your code.
  • DAS mode eliminates traditional hook points used by the file system to implement features.
  • Features not in DAS Mode: NTFS encryption, NTS compression, NTFS TxF, ReFS integrity streams, ReFS cluster band, ReFS block cloning, Bitlocker volume encryption, snapshot via VolSnap, mirrored or parity via storage spaces or dynamic disks
  • Sparse files won’t be there initially but will come in the future.
  • Updated at the time the file is memory mapped: file modification time, mark file as modified in the USN journal, directory change notification
  • File System Filters in DAS mode: no notification that a DAS volume is mounted, filter will indicate via a flag if they understand DAS mode semantics.
  • Application compatibility with filters in DAS mode: No opportunity for data transformation filters (encryption, compression). Anti-virus are minimally impacted, but will need to watch for creation of writeable mapped sections (no paging writes anymore).
  • Intel NVLM library. Open source library implemented by Intel. Defines set of application APIs for directly manipulating files on SCM hardware.
  • NVLM library available for Linux today via GitHub. Microsoft working with Intel on a Windows port.
  • Q: XIP (Execute in place)? It’s important, but the plans have not solidified yet.
  • Q: NUMA? Can be in NUMA nodes. Typically, the file system and cache are agnostic to NUMA.
  • Q: Hyper-V? Not ready to talk about what we are doing in that area.
  • Q: Roll-out plan? We have one, but not ready to talk about it yet.
  • Q: Data forensics? We’ve yet to discuss this with that group. But we will.
  • Q: How far are you to completion? It’s running and working today. But it is not complete.
  • Q: Windows client? To begin, we’re targeting the server. Because it’s available there first.
  • Q: Effect on performance? When we’re ready to announce the schedule, we will announce the performance. The data about SCM is out there. It’s fast!
  • Q: Will you backport? Probably not. We generally move forward only. Not many systems with this kind of hardware will run a down level OS.
  • Q: What languages for the Windows port of NVML? Andy will cover that in his talk tomorrow.
  • Q: How fast will memory mapped be? Potentially as fast as DRAM, but depends on the underlying technology.

 

The Bw-Tree Key-Value Store and Its Applications to Server/Cloud Data Management in Production
Sudipta Sengupta, Principal Research Scientist, Microsoft Research

  • The B-Tree: key-ordered access to records. Balanced tree via page split and merge mechanisms.
  • Design tenets: Lock free operation (high concurrency), log-structure storage (exploit flash devices with fast random reads and inefficient random writes), delta updates to pages (reduce cache invalidation, garbage creation)
  • Bw-Tree Architecture: 3 layers: B-Tree (expose API, B-tree search/update, in-memory pages), Cache (logical page abstraction, move between memory and flash), Flash (reads/writes from/to storage, storage management).
  • Mapping table: Expose logical pages to access method layer. Isolates updates to single page. Structure for lock-free multi-threaded concurrency control.
  • Highly concurrent page updates with Bw-Tree. Explaining the process using a diagram.
  • Bw-Tree Page Split: No hard threshold for splitting unlike in classical B-Tree. B-link structure allows “half-split” without locking.
  • Flash SSDs: Log-Structured storage. Use log structure to exploit the benefits of flash and work around its quirks: random reads are fast, random in-place writes are expensive.
  • LLAMA Log-Structured Store: Amortize cost of writes over many page updates. Random reads to fetch a “logical page”.
  • Depart from tradition: logical page formed by linking together records on multiple physical pages on flash. Adapted from SkimpyStash.
  • Detailed diagram comparing traditional page writing with the writing optimized storage organization with Bw-Tree.
  • LLAMA: Optimized Logical Page Reads. Multiple delta records are packed when flushed together. Pages consolidated periodically in memory also get consolidated on flash when flushed.
  • LLAMA: Garbage collection on flash. Two types of record units in the log: Valid or Orphaned. Garbage collection starts from the oldest portion of the log. Earliest written record on a logical page is encountered first.
  • LLAMA: cache layer. Responsible for moving pages back and forth from storage.
  • Bw-Tree Checkpointing: Need to flush to buffer and to storage. LLAMA checkpoint for fast recovery.
  • Bw-Tree Fast Recovery. Restore mapping table from latest checkpoint region. Warm-up using sequential I/O.
  • Bw-Tree: Support for transactions. Part of the Deuteronomy Architecture.
  • End-to-end crash recovery. Data component (DC) and transactional component (TC) recovery. DC happens before TC.
  • Bw-Tree in production: Key-sequential index in SQL Server in-memory database
  • Bw-Tree in production: Indexing engine in Azure DocumentDB. Resource governance is important (CPU, Memory, IOPS, Storage)
  • Bw-Tree in production: Sorted key-value store in Bing ObjectStore.
  • Summary: Classic B-Tree redesigned for modern hardware and cloud. Lock-free, delta updating of pages, log-structure, flexible resource governor, transactional. Shipping in production.
  • Going forward: Layer transactional component (Deuteronomy Architecture, CIDR 2015), open-source the codebase

 

ReFS v2: Cloning, Projecting, and Moving Data
J.R. Tipton, Development Lead, Microsoft

  • Agenda: ReFS v1 primer, ReFS v2 at a glance, motivations for ReFS v2, cloning, translation, transformation
  • ReFS v1 primer: Windows allocate-on-write file system, Merkel trees verify metadata integrity, online data correction from alternate copies, online chkdsk
  • ReFS v2: Available in Windows Server 2016 TP4. Efficient, reliable storage for VMs, efficient parity, write tiering, read caching, block cloning, optimizations
  • Motivations for ReFS v2: cheap storage does not mean slow, VM density, VM provisioning, more hardware flavors (SLC, MLC, TLC flash, SMR)
  • Write performance. Magic does not work in a few environments (super fast hardware, small random writes, durable writes/FUA/sync/write-through)
  • ReFS Block Cloning: Clone any block of one file into any other block in another file. Full file clone, reorder some or all data, project data from one area into another without copy
  • ReFS Block Cloning: Metadata only operation. Copy-on-write used when needed (ReFS knows when).
  • Cloning examples: deleting a Hyper-V VM checkpoint, VM provisioning from image.
  • Cloning observations: app directed, avoids data copies, metadata operations, Hyper-V is the first but not the only one using this
  • Cloning is no free lunch: multiple valid copies will copy-on-write upon changes. metadata overhead to track state, slam dunk in most cases, but not all
  • ReFS cluster bands. Volume internally divvied up into bands that contain regular FS clusters (4KB, 64KB). Mostly invisible outside file system. Bands and clusters track independently (per-band metadata). Bands can come and go.
  • ReFS can move bands around (read/write/update band pointer). Efficient write caching and parity. Writes to bands in fast tier. Tracks heat per band. Moves bands between tiers. More efficient allocation. You can move from 100% triple mirroring to 95% parity.
  • ReFS cluster bands: small writes accumulate where writing is cheap (mirror, flash, log-structured arena), bands are later shuffled to tier where random writes are expensive (band transfers are fully sequential).
  • ReFS cluster bands: transformation. ReFS can do stuff to the data in a band (can happen in the background). Examples: band compaction (put cold bands together, squeeze out free space), band compression (decompress on read).
  • ReFS v2 summary: data cloning, data movement, data transformation. Smart when smart makes sense, switches to dumb when dumb is better. Takes advantages of hardware combinations. And lots of other stuff…

 

Innovator, Disruptor or Laggard, Where Will Your Storage Applications Live? Next Generation Storage
Bev Crair, Vice President and General Manager, Storage Group, Intel

  • The world is changing: information growth,  complexity, cloud, technology.
  • Growth: 44ZB of data in all systems. 15% of the data is stored, since perceived cost is low.
  • Every minute of every day: 2013 : 8h of of video uploaded to YouTube, 47,000 apps downloaded, 200 million e-mails
  • Every minute of every day: 2015 : 300h of of video uploaded to YouTube, 51,000 apps downloaded, 204 million e-mails
  • Data never sleeps: the internet in real time. tiles showing activities all around the internet.
  • Data use pattern changes: sense and generate, collect and communicate, analyze and optimize. Example: HADRON collider
  • Data use pattern changes: from collection to analyzing data, valuable data now reside outside the organization, analyzing and optimizing unstructured data
  • Cloud impact on storage solutions: business impact, technology impact. Everyone wants an easy button
  • Intelligent storage: Deduplication, real-time compression, intelligent tiering, thin provisioning. All of this is a software problem.
  • Scale-out storage: From single system with internal network to nodes working together with an external network
  • Non-Volatile Memory (NVM) accelerates the enterprise: Examples in Virtualization, Private Cloud, Database, Big Data and HPC
  • Pyramid: CPU, DRAM, Intel DIMM (3D XPoint), Intel SSD (3D XPoint), NAND SSD, HDD,  …
  • Storage Media latency going down dramatically. With NVM, the bottleneck is now mostly in the software stack.
  • Future storage architecture: complex chart with workloads for 2020 and beyond. New protocols, new ways to attach.
  • Intel Storage Technologies. Not only hardware, but a fair amount of software. SPDK, NVMe driver, Acceleration Library, Lustre, others.
  • Why does faster storage matter? Genome testing for cancer takes weeks, and the cancer mutates. Genome is 10TB. If we can speed up the time it takes to test it to one day, it makes a huge difference and you can create a medicine that saves a person’s life. That’s why it matters.

  

The Long-Term Future of Solid State Storage
Jim Handy, General Director, Objective Analysis

  • How we got here? Why are we in the trouble we’re at right now? How do we get ahead of it? Where is it going tomorrow?
  • Establishing a schism: Memory is in bytes (DRAM, Cache, Flash?), Storage is in blocks (Disk Tape, DVD, SAN, NAS, Cloud, Flash)
  • Is it really about block? Block, NAND page, DRAM pages, CPU cache lines. It’s all in pages anyway…
  • Is there another differentiator? Volatile vs. Persistent. It’s confusing…
  • What is an SSD? SSDs are nothing new. Going back to DEC Bulk Core.
  • Disk interfaces create delays. SSD vs HDD latency chart. Time scale in milliseconds.
  • Zooming in to tens of microseconds. Different components of the SSD delay. Read time, Transfer time, Link transfer, platform and adapter, software
  • Now looking at delays for MLC NAND ONFi2, ONFi3, PCIe x4 Gen3, future NVM on PCIe x4 Gen3
  • Changing the scale to tens of microseconds on future NVM. Link Transfer, Platform & adapter and Software now accounts for most of the latency.
  • How to move ahead? Get rid of the disk interfaces (PCIe, NVMe, new technologies). Work on the software: SNIA.
  • Why now? DRAM Transfer rates. Chart transfer rates for SDRAM, DDR, DDR2, DDR3, DDR4. Designing the bus takes most of the time.
  • DRAM running out of speed? We probably won’t see a DDR5. HMC or HBM a likely next step. Everything points to fixed memory sizes.
  • NVM to the rescue. DRAM is not the only upgrade path. It became cheaper to use NAND flash than DRAM to upgrade a PC.
  • NVM to be a new memory layer between DRAM & NAND: Intel/Micron 3D XPoint – “Optane”
  • One won’t kill the other. Future systems will have DRAM, NVM, NAND, HDD. None of them will go away…
  • New memories are faster than NAND. Chart with read bandwidth vs write bandwidth. Emerging NVRAM: FeRAM, eMRAM, RRAM, PRAM.
  • Complex chart with emerging research memories. Clock frequency vs. Cell Area (cost).
  • The computer of tomorrow. Memory or storage? In the beginning (core memory), there was no distinction between the two.
  • We’re moving to an era where you can turn off the computer, turn it back on and there’s something in memory. Do you trust it?
  • SCM – Storage Class Memory: high performance with archival properties. There are many other terms for it: Persistent Memory, Non-Volatile Memory.
  • New NVM has disruptively low latency: Log chart with latency budgets for HDD, SATA SSD, NVMe, Persistent. When you go below 10 microseconds (as Persistent does), context switching does not make sense.
  • Non-blocking I/O. NUMA latencies up to 200ns have been tolerated. Latencies below these cause disruption.
  • Memory mapped files eliminate file system latency.
  • The computer of tomorrow. Fixed DRAM size, upgradeable NVM (tomorrow’s DIMM), both flash and disk (flash on PCIe or own bus), much work needed on SCM software
  • Q: Will all these layers survive? I believe so. There are potential improvements in all of them (cited a few on NAND, HDD).
  • Q: Shouldn’t we drop one of the layers? Usually, adding layers (not removing them) is more interesting from a cost perspective.
  • Q: Do we need a new protocol for SCM? NAND did well without much of that. Alternative memories could be put on a memory bus.

 

Concepts on Moving From SAS connected JBOD to an Ethernet Connected JBOD
Jim Pinkerton, Partner Architect Lead, Microsoft

  • What if we took a JBOD, a simple device, and just put it on Ethernet?
  • Re-Thinking the Software-defined Storage conceptual model definition: compute nodes, storage nodes, flakey storage devices
  • Front-end fabric (Ethernet, IB or FC), Back-end fabric (directly attached or shared storage)
  • Yesterday’s Storage Architecture: Still highly profitable. Compute nodes, traditional SAN/NAS box (shipped as an appliance)
  • Today: Software Defined Storage (SDS) – “Converged”. Separate the storage service from the JBOD.
  • Today: Software Defined Storage (SDS) – “Hyper-Converged” (H-C). Everything ships in a single box. Scale-out architecture.
  • H-C appliances are a dream for the customer to install/use, but the $/GB storage is high.
  • Microsoft Cloud Platform System (CPS). Shipped as a packaged deal. Microsoft tested and guaranteed.
  • SDS with DAS – Storage layer divided into storage front-end (FE) and storage back-end (BE). The two communicate over Ethernet.
  • SDS Topologies. Going from Converged and Hyper-Converged to a future EBOD topology. From file/block access to device access.
  • Expose the raw device over Ethernet. The raw device is flaky, but we love it. The storage FE will abstract that, add reliability.
  • I would like to have an EBOD box that could provide the storage BE.
  • EBOD works for a variety of access protocols and topologies. Examples: SMB3 “block”, Lustre object store, Ceph object store, NVMe fabric, T10 objects.
  • Shared SAS Interop. Nightmare experience (disk multi-path interop, expander multi-path interop, HBA distributed failure).  This is why customers prefers appliances.
  • To share or not to share. We want to share, but we do not want shared SAS. Customer deployment is more straightforward, but you have more traffic on Ethernet.
  • Hyper-Scale cloud tension – fault domain rebuild time. Depends on number of disks behind a node and how much network you have.
  • Fault domain for storage is too big. Required network speed offsets cost benefits of greater density. Many large disks behind a single node becomes a problem.
  • Private cloud tension – not enough disks. Entry points at 4 nodes, small number of disks. Again, fault domain is too large.
  • Goals in refactoring SDS – Storage back-end is a “data mover” (EBOD). Storage front-end is “general purpose CPU”.
  • EBOD goals – Can you hit a cost point that’s interesting? Reduce storage costs, reduce size of fault domain, build a more robust ecosystem of DAS. Keep topology simple, so customer can build it themselves.
  • EBOD: High end box, volume box, capacity box.
  • EBOD volume box should be close to what a JBOD costs. Basically like exposing raw disks.
  • Comparing current Hyper-Scale to EBOD. EBOD has an NIC and an SOC, in addition to the traditional expander in a JBOD.
  • EBOD volume box – Small CPU and memory, dual 10GbE, SOC with RDMA NIC/SATA/SAS/PCIe, up to 20 devices, SFF-8639 connector, management (IPMI, DMTF Redfish?)
  • Volume EBOD Proof Point – Intel Avaton, PCIe Gen 2, Chelsio 10GbE, SAS HBA, SAS SSD. Looking at random read IOPS (local, RDMA remote and non-RDMA remote). Max 159K IOPS w/RDMA, 122K IOPS w/o RDMA. Latency chart showing just a few msec.
  • EBOD Performance Concept – Big CPU, Dual attach 40GbE, Possibly all NVME attach or SCM. Will show some of the results this afternoon.
  • EBOD is an interesting approach that’s different from what we’re doing. But it’s nicely aligned with software-defined storage.
  • Price point of EBOD must be carefully managed, but the low price point enables a smaller fault domain.

  

Planning for the Next Decade of NVM Programming
Andy Rudoff, SNIA NVM Programming TWG, Intel

  • Looking at what’s coming up in the next decade, but will start with some history.
  • Comparison of data storage technologies. Emerging NV technologies with read times in the same order of magnitude as DRAM.
  • Moving the focus to software latency when using future NVM.
  • Is it memory or storage? It’s persistent (like storage) and byte-addressable (like memory).
  • Storage vs persistent memory. Block IO vs. byte addressable, sync/async (DMA master)  vs. sync (DMA slave). High capacity vs. growing capacity.
  • pmem: The new Tier. Byte addressable, but persistent. Not NAND. Can do small I/O. Can DMA to it.
  • SNIA TWG (lots of companies). Defining the NVM programming model: NVM.PM.FILE mode and NVM.PM.VOLUME mode.
  • All the OSes created in the last 30 years have a memory mapped file.
  • Is this stuff real? Why are we spending so much time on this? Yes – Intel 3D XPoint technology, the Intel DIMM. Showed a wafer on stage. 1000x faster than NAND. 1000X endurance of NAND, 10X denser than conventional memory. As much as 6TB of this stuff…
  • Timeline: Big gap between NAND flash memory (1989) and 3D XPoint (2015).
  • Diagram of of the model with Management, Block, File and Memory access. Link at the end to the diagram.
  • Detecting pmem: Defined in the ACPI 6.0. Linux support upstream (generic DIMM driver, DAX, ext4+DAX, KVM).  Neal talked about Windows support yesterday.
  • Heavy OSV involvement in TWG, we wrote the spec together.
  • We don’t want every application to have to re-architecture itself. That’s why we have block and file there as well.
  • The next decade
  • Transparency  levels: increasing barrier to adoption. increasing leverage. Could do it in layers. For instance, could be file system only, without app modification. For instance, could modify just the JVM to get significant advantages without changing the apps.
  • Comparing to multiple cores in hardware and multi-threaded programming. Took a decade or longer, but it’s commonplace now.
  • One transparent example: pmem Paging. Paging from the OS page cache (diagrams).
  • Attributes of paging : major page faults, memory looks much larger, page in must pick a victim, many enterprise apps opt-out, interesting example: Java GC.
  • What would it look like if you paged to pmem instead of paging to storage. I don’t even care that it’s persistent, just that there’s a lot of it.
  • I could kick a page out synchronously, probably faster than a context switch. But the app could access the data in pmem without swapping it in (that‘s new!). Could have policies for which app lives in which memory. The OS could manage that, with application transparency.
  • Would this really work? It will when pmem costs less, performance is close, capacity is significant and it is reliable. “We’re going to need a bigger byte” to hold error information.
  • Not just for pmem. Other memories technologies are emerging. High bandwidth memory, NUMA localities, different NVM technologies.
  • Extending into user space: NVM Library – pmem.io (64-bit Linux Alpha release). Windows is working on it as well.
  • That is a non-transparent example. It’s hard (like multi-threading). Things can fail in interesting new ways.
  • The library makes it easier and some of it is transactional.
  • No kernel interception point, for things like replication. No chance to hook above or below the file system. You could do it in the library.
  • Non-transparent use cases: volatile caching, in-memory database, storage appliance write cache, large byte-addressable data structures (hash table, dedup), HPC (checkpointing)
  • Sweet spots: middleware, libraries, in-kernel usages.
  • Big challenge: middleware, libraries. Is it worth the complexity.
  • Building a software ecosystem for pmem, cost vs. benefit challenge.
  • Prepare yourself: lean NVM programming model, map use cases to pmem, contribute to the libraries, software ecosystem

 

FS Design Around SMR: Seagate’s Journey and Reference System with EXT4
Adrian Palmer, Drive Development Engineering, Seagate Technologies

  • SNIA Tutorial. I’m talking about the standard, as opposed as the design of our drive.
  • SMR is being embraced by everyone, since this is a major change, a game changes.
  • From random writes to resemble the write profile of sequential-access tape.
  • 1 new condition: forward-write preferred. ZAC/ZBD spec: T10/13. Zones, SCSI ZBC standards, ATA ZAC standards.
  • What is a file system? Essential software on a system, structured and unstructured data, stores metadata and data.
  • Basic FS requirements: Write-in-place (superblock, known location on disk), Sequential write (journal), Unrestricted write type (random or sequential)
  • Drive parameters: Sector (atomic unit of read/write access). Typically 512B size. Independently accessed. Read/write, no state.
  • Drive parameters: Zone (atomic performant rewrite unit). Typically 256 MiB in size. Indirectly addressed via sector. Modified with ZAC/ZBD commands. Each zone has state (WritePointer, Condition, Size, Type).
  • Write Profiles. Conventional (random access), Tape (sequential access), Flash (sequential access, erase blocks), SMR HA/HM (sequential access, zones). SMR write profile is similar to Tape and Flash.
  • Allocation containers. Drive capacities are increasing, location mapping is expensive. 1.56% with 512B blocks or 0.2% with 4KB blocks.
  • Remap the block device as a… block device. Partitions (w*sector size), Block size (x*sector size), Group size (y*Block size), FS (z*group size, expressed as blocks).
  • Zones are a good fit to be matched with Groups. Absorb and mirror the metadata, don’t keep querying drive for metadata.
  • Solving the sequential write problem. Separate the problem spaces with zones.
  • Dedicate zones to each problem space: user data, file records, indexes, superblock, trees, journal, allocation containers.
  • GPT/Superblocks: First and last zone (convention, not guaranteed). Update infrequently, and at dismount. Looks at known location and WritePointer. Copy-on-update. Organized wipe and update algorithm.
  • Journal/soft updates. Update very frequently, 2 or more zones, set up as a circular buffer. Checkpoint at each zone. Wipe and overwrite oldest zone. Can be used as NV cache for metadata. Requires lots of storage space for efficient use and NV.
  • Group descriptors: Infrequently changed. Changes on zone condition change, resize, free block counts. Write cached, butwritten at WritePointer. Organized as a B+Tree, not an indexed array. The B+Tree needs to be stored on-disk.
  • File Records: POSIX information (ctime, mtime, atime, msize, fs specific attributes), updated very frequently. Allows records to be modified in memory, written to journal cache, gather from journal, write to new blocks at WritePointer.
  • Mapping (file records to blocks). File ideally written as a single chunk (single pointer), but could become fragmented (multiple pointers). Can outgrow file record space, needs its own B+Tree. List can be in memory, in the journal, written out to disk at WritePointer.
  • Data: Copy-on-write. Allocator chooses blocks at WritePointer. Writes are broken at zone boundary, creating new command and new mapping fragment.
  • Cleanup: Cannot clean up as you go, need a separate step. Each zone will have holes. Garbage collection: Journal GC, Zones GC, Zone Compaction, Defragmentation.
  • Advanced features: indexes, queries, extended attributes, snapshots, checksums/parity, RAID/JBOD.

 

Azure File Service: ‘Net Use’ the Cloud
David Goebel, Software Engineer, Microsoft

  • Agenda: features and API (what), scenarios enabled (why), design of an SMB server not backed by a conventional FS (how)
  • It’s not the Windows SMB server (srv2.sys). Uses Azure Tables and Azure Blobs for the actual files.
  • Easier because we already have a highly available and distributed architecture.
  • SMB 2.1 in preview since last summer. SMB 3.0 (encryption, persistent handles) in progress.
  • Azure containers mapped as shares. Clients work unmodified out-of-the-box. We implemented the spec.
  • Share namespace is coherently accessible
  • MS-SMB2, not SMB1. Anticipates (but does not require) a traditional file system on the other side.
  • In some ways it’s harder, since what’s there is not a file system. We have multiple tables (for leases, locks, etc). Nice and clean.
  • SMB is a stateful protocol, while REST is all stateless. Some state is immutable (like FileId), some state is transient (like open counts), some is maintained by the client (like CreateGuid), some state is ephemeral (connection).
  • Diagram with the big picture. Includes DNS, load balancer, session setup & traffic, front-end node, azure tables and blobs.
  • Front-end has ephemeral and immutable state. Back-end has solid and fluid durable state.
  • Diagram with two clients accessing the same file and share, using locks, etc. All the state handled by the back-end.
  • Losing a front-end node considered a regular event (happens during updates), the client simple reconnects, transparently.
  • Current state, SMB 2.1 (SMB 3.0 in the works). 5TB per share and 1TB per file. 1,000 8KB IOPS per share, 60MB/sec per share. Some NTFS features not supported, some limitations on characters and path length (due to HTTP/REST restrictions).
  • Demo: I’m actually running my talk using a PPTX file on Azure File. Robocopy to file share. Delete, watch via explorer (notifications working fine). Watching also via wireshark.
  • Currently Linux Support. Lists specific versions Ubuntu Server, Ubuntu Core, CentOS, Open SUSE, SUSE Linux Enterprise Server.
  • Why: They want to move to cloud, but they can’t change their apps. Existing file I/O applications. Most of what was written over the last 30 years “just works”. Minor caveats that will become more minor over time.
  • Discussed specific details about how permissions are currently implemented. ACL support is coming.
  • Example: Encryption enabled scenario over the internet.
  • What about REST? SMB and REST access the same data in the same namespace, so a gradual application transition without disruption is possible. REST for container, directory and file operations.
  • The durability game. Modified state that normally exists only in server memory, which must be durably committed.
  • Examples of state tiering: ephemeral state, immutable state, solid durable state, fluid durable state.
  • Example: Durable Handle Reconnect. Intended for network hiccups, but stretched to also handles front-end reconnects. Limited our ability because of SMB 2.1 protocol compliance.
  • Example: Persistent Handles. Unlike durable handles, SMB 3 is actually intended to support transparent failover when a front-end dies. Seamless transparent failover.
  • Resource Links: Getting started blog (http://blogs.msdn.com/b/windowsazurestorage/archive/2014/05/12/introducing-microsoft-azure-file-service.aspx) , NTFS features currently not supported (https://msdn.microsoft.com/en-us/library/azure/dn744326.aspx), naming restrictions for REST compatibility (https://msdn.microsoft.com/library/azure/dn167011.aspx).

 

Software Defined Storage - What Does it Look Like in 3 Years?
Richard McDougall, Big Data and Storage Chief Scientist, VMware

  • How do you come up with a common, generic storage platform that serves the needs of application?
  • Bringing a definition of SDS. Major trends in hardware, what the apps are doing, cloud platforms
  • Storage workloads map. Many apps on 4 quadrants on 2 axis: capacity (10’s of Terabytes to 10’s of Petabytes) and IOPS (1K to 1M)
  • What are cloud-native applications? Developer access via API, continuous integration and deployment, built for scale, availability architected in the app, microservices instead of monolithic stacks, decoupled from infrastructure
  • What do Linux containers need from storage? Copy/clone root images, isolated namespace, QoS controls
  • Options to deliver storage to containers: copy whole root tree (primitive), fast clone using shared read-only images, clone via “Another Union File System” (aufs), leverage native copy-on-write file system.
  • Shared data: Containers can share file system within host or across hots (new interest in distributed file systems)
  • Docker storage abstractions for containers: non-persistent boot environment, persistent data (backed by block volumes)
  • Container storage use cases: unshared volumes, shared volumes, persist to external storage (API to cloud storage)
  • Eliminate the silos: converged big data platform. Diagram shows Hadoop, HBase, Impala, Pivotal HawQ, Cassandra, Mongo, many others. HDFS, MAPR, GPFS, POSIX, block storage. Storage system common across all these, with the right access mechanism.
  • Back to the quadrants based on capacity and IOPS. Now with hardware solutions instead of software. Many flash appliances in the upper left (low capacity, high IOPS). Isilon in the lower right (high capacity, low IOPS).
  • Storage media technologies in 2016. Pyramid with latency, capacity per device, capacity per host for each layer: DRAM (1TB/device, 4TB/host, ~100ns latency), NVM (1TB, 4TB, ~500ns), NVMe SSD (4TB, 48TB, ~10us), capacity SSD (16TB, 192TB, ~1ms), magnetic storage (32TB, 384TB, ~10ms), object storage (?, ?, ~1s). 
  • Back to the quadrants based on capacity and IOPS. Now with storage media technologies.
  • Details on the types of NVDIMM (NVIMM-N - Type 1, NVDIMM-F – Type 2, Type 4). Standards coming up for all of these. Needs work to virtualize those, so they show up properly inside VMs.
  • Intel 3D XPoint Technology.
  • What are the SDS solutions than can sit on top of all this? Back to quadrants with SDS solutions. Nexenta, Mentions ScaleiO, VSAN, ceph, Scality, MAPR, HDFS. Can you make one solution that works well for everything?
  • What’s really behind a storage array? The value from the customer is that it’s all from one vendor and it all works. Nothing magic, but the vendor spent a ton of time on testing.
  • Types of SDS: Fail-over software on commodity servers (lists many vendors), complexity in hardware, interconnects. Issues with hardware compatibility.
  • Types of SDS: Software replication using servers + local disks. Simpler, but not very scalable.
  • Types of SDS: Caching hot core/cold edge. NVMe flash devices up front, something slower behind it (even cloud). Several solutions, mostly startups.
  • Types of SDS: Scale-out SDS. Scalable, fault-tolerant, rolling updates. More management, separate compute and storage silos. Model used by ceph, ScaleiO. Issues with hardware compatibility. You really need to test the hardware.
  • Types of SDS: Hyper-converged SDS. Easy management, scalable, fault-tolerant, rolling upgrades. Fixed compute to storage ration. Model used by VSAN, Nutanix. Amount of variance in hardware still a problem. Need to invest in HCL verification.
  • Storage interconnects. Lots of discussion on what’s the right direction. Protocols (iSCSI, FC, FCoE, NVMe, NVMe over Fabrics), Hardware transports (FC, Ethernet, IB, SAS), Device connectivity (SATA, SAS, NVMe)
  • Network. iSCSI, iSER, FCoE, RDMA over Ethernet, NVMe Fabrics. Can storage use the network? RDMA debate for years. We’re at a tipping point.
  • Device interconnects: HCA with SATA/SAS. NVMe SSD, NVM over PCIe. Comparing iSCSI, FCoE and NVMe over Ethernet.
  • PCIe rack-level Fabric. Devices become addressable. PCIe rack-scale compute and storage, with host-to-host RDMA.
  • NVMe – The new kid on the block. Support from various vendors. Quickly becoming the all-purpose stack for storage, becoming the universal standard for talking block.
  • Beyond block: SDS Service Platforms. Back to the 4 quadrants, now with service platforms.
  • Too many silos: block, object, database, key-value, big data. Each one is its own silo with its own machines, management stack, HCLs. No sharing of infrastructure.
  • Option 1: Multi-purpose stack. Has everything we talked about, but it’s a compromise.
  • Option 2: Common platform + ecosystem of services. Richest, best-of-breed services, on a single platform, manageable, shared resources.

 

Why the Storage You Have is Not the Storage Your Data Needs
Laz Vekiarides, CTO and Co-founder, ClearSky Data

  • ClearSky Data is a tech company, consumes what we discussed in this conference.
  • The problem we’re trying to solve is the management of the storage silos
  • Enterprise storage today. Chart: Capacity vs. $/TB. Flash, Mid-Range, Scale-Out. Complex, costly silos
  • Describe the lifecycle of the data, the many copies you make over time, the rebuilding and re-buying of infrastructure
  • What enterprises want: buy just enough of the infrastructure, with enough performance, availability, security.
  • Cloud economics – pay only for the stuff that you use, you don’t have to see all the gear behind the storage, someone does the physical management
  • Tiering is a bad answer – Nothing remains static. How fast does hot data cool? How fast does it re-warm? What is the overhead to manage it? It’s a huge overhead. It’s not just a bandwidth problem.
  • It’s the latency, stupid. Data travels at the speed of light. Fast, but finite. Boston to San Francisco: 29.4 milliseconds of round-trip time (best case). Reality (with switches, routers, protocols, virtualization) is more like 70 ms.
  • So, where exactly is the cloud? Amazon East is near Ashburn, VA. Best case is 10ms RTT. Worst case is ~150ms (does not include time to actually access the storage).
  • ClearSky solution: a global storage network. The infrastructure becomes invisible to you, what you see is a service level agreement.
  • Solution: Geo-distributed data caching. Customer SAN, Edge, Metro POP, Cloud. Cache on the edge (all flash), cache on the metro POP.
  • Edge to Metro POP are private lines (sub millisecond latency). Addressable market is the set of customers within a certain distance to the Metro POP.
  • Latency math: Less than 1ms to the Metro POP, cache miss path is between 25ms and 50ms.
  • Space Management: Edge (hot, 10%, 1 copy), POP (warm, <30%, 1-2 copies), Cloud (100%, n copies). All data is deduplicated and encrypted.
  • Modeling cache performance: Miss ratio curve (MRC). Performance as f(size), working set knees, inform allocation policy.
  • Reuse distance (unique intervening blocks between use and reuse). LRU is most of what’s out there. Look at stacking algorithms. Chart on cache size vs. miss ratio. There’s a talk on this tomorrow by CloudPhysics.
  • Worked with customers to create a heat map data collector. Sizing tool for VM environments. Collected 3-9 days of workload.
  • ~1,400 virtual disks, ~800 VMs, 18.9TB (68% full), avg read IOPS 5.2K, write IOPS 5.9K. Read IO 36KB, write IO 110KB. Read Latency 9.7ms, write latency 4.5ms.
  • This is average latency, maximum is interesting, some are off the chart. Some were hundred of ms, even 2 second.
  • Computing the cache miss ratio. How much cache would we need to get about 90% hit ratio? Could do it with less than 12% of the total.
  • What is cache hit for writes? What fits in the write-back cache. You don’t want to be synchronous with the cloud. You’ll go bankrupt that way.
  • Importance of the warm tier. Hot data (Edge, on prem, SSD) = 12%, warm data (Metro PoP, SSD and HDD) = 6%, cold data (Cloud) = 82%. Shown as a “donut”.
  • Yes, this works! We’re having a very successful outcome with the customers currently engaged.
  • Data access is very tiered. Small amounts of flash can yield disproportionate performance benefits. Single tier cache in front of high latency storage can’t work. Network latency is as important as bounding media latency.
  • Make sure your caching is simple. Sometimes you are overthinking it.
  • Identifying application patterns is hard. Try to identify the sets of LBA that are accessed. Identify hot spots, which change over time. The shape of the miss ratio remains similar.

 

Emerging Trends in Software Development
Donnie Berkholz, Research Director, 451 Research

  • How people are building applications. How storage developers are creating and shipping software.
  • Technology adoption is increasingly bottom-up. Open source, cloud. Used to be like building a cathedral, now it’s more like a bazaar.
  • App-dev workloads are quickly moving to the cloud. Chart from all-on-prem at the top to all-cloud at the bottom.
  • All on-prem going from 59% now to 37% in a few years. Moving to different types of clouds (private cloud, Public cloud (IaaS), Public cloud (SaaS).
  • Showing charts for total data at organization, how much in off-premises cloud (TB and %). 64% of people have less than 20% on the cloud.
  • The new stack. There’s a lot of fragmentation. 10 languages in the top 80%. Used to be only 3 languages. Same thing for databases. It’s more composable, right tool for the right job.
  • No single stack. An infinite set of possibilities.
  • Growth in Web APIs charted since 2005 (from ProgrammableWeb). Huge growth.
  • What do enterprises think of storage vendors. Top vendors. People not particularly happy with their storage vendors. Promise index vs. fulfillment index.
  • Development trends that will transform storage.
  • Containers. Docker, docker, docker. Whale logos everywhere. When does it really make sense to use VMs or containers? You need lots of random I/O for these to work well. 10,000 containers in a cluster? Where do the databases go?
  • Developers love Docker. Chart on configuration management GitHub totals (CFEngine, Puppet, Chef, Ansible, Salt, Docker). Shows developer adoption. Docker is off the charts.
  • It’s not just a toy. Survey of 1,000 people on containers. Docker is only 2.5 years old now. 20% no plans, 56% evaluating. Total doing pilot or more add up to 21%. That’s really fast adoption
  • Docker to microservices.
  • Amazon: “Every single data transfer between teams has to happen through an API or you’re fired”. Avoid sending spreadsheets around.
  • Microservices thinking is more business-oriented, as opposed to technology-oriented.
  • Loosely couple teams. Team organization has a great influence in your development.
  • The foundation of microservices. Terraform, MANTL, Apache Mesos, Capgemini Appollo, Amazon EC2 Container Service.
  • It’s a lot about scheduling. Number of schedulers that use available resources. Makes storage even more random.
  • Disruption in data processing. Spark. It’s a competitor to Hadoop, really good at caching in memory, also very fast on disk. 10x faster than map-reduce. People don’t have to be big data experts. Chart: Spark came out of nowhere (mining data from several public forums).
  • The market is coming. Hadoop market as a whole growing 46% (CAGR).
  • Storage-class memory. Picture of 3D XPoint. Do app developer care? Not sure. Not many optimize for cache lines in memory. Thinking about Redis in-memory database for caching. Developers probably will use SCM that way. Caching in the order of TB instead of GB.
  • Network will be incredibly important. Moving bottlenecks around.
  • Concurrency for developers. Chart of years vs. Percentage of Ohlon. Getting near to 1%. That’s a lot single the most popular is around 10%.
  • Development trends
  • DevOps. Taking agile development all the way to production. Agile, truly tip to tail. You want to iterate while involving your customers. Already happening with startups, but how do you scale?
  • DevOps: Culture, Automation (Pets vs. Cattle), Measurement
  • Automation: infrastructure as code. Continuous delivery.
  • Measurement: Nagios, graphite, Graylog2, splunk, Kibana, Sensu, etsy/statsd
  • DevOps is reaching DBAs. #1 stakeholder in recent survey.
  • One of the most popular team structure change. Dispersing the storage team.
  • The changing role of standards
  • The changing role of benchmarks. Torturing databases for fun and profit.
  • I would love for you to join our panel. If you fill our surveys, you get a lot of data for free.

 

Learnings from Nearly a Decade of Building Low-cost Cloud Storage
Gleb Budman, CEO, Backblaze

  • What we learned, specifically the cost equation
  • 150+ PB of customer data. 10B files.
  • In 2007 we wanted to build something that would backup your PC/Mac data to the cloud. $5/month.
  • Originally we wanted to put it all on S3, but we would lose money on every single customer.
  • Next we wanted to buy SANs to put the data on, but that did not make sense either.
  • We tried a whole bunch of things. NAS, USB-connected drives, etc.
  • Cloud storage has a new player, with a shockingly low price: B2. One fourth of the cost of S3.
  • Lower than Glacier, Nearline, S3-Infrequent Access, anything out there. Savings here add up.
  • Datacenter: convert kilowatts-to-kilobits
  • Datacenter Consideration: local cost to power, real state, taxes, climate, building/system efficiency, proximity to good people, connectivity.
  • Hardware: Connect hard drives to the internet, with as little as possible in between.
  • Blackblaze storage box, costs about $3K. As simple as possible, don’t make the hardware itself redundant. Use commodity parts (example: desktop power supply), use consumer hard drives, insource & use math for drive purchases.
  • They told us we could not use consumer hard drives. But reality is that the failure rate was actually lower. They last 6 years on average. Even if the enterprise HDD never fail, they still don’t make sense.
  • Insource & use math for drive purchases. Drives are the bulk of the cost. Chart with time vs. price per gigabyte. Talking about the Thailand Hard Drive Crisis.
  • Software: Put all intelligence here.
  • Blackblaze vault: 20 hard drives create 1 tome that share parts of a file, spread across racks.
  • Avoid choke point. Every single storage pods is a first class citizen. We can parallelize.
  • Algorithmically monitor SMART stats. Know which SMART codes correlate to annual failure rate. All the data is available on the site (all the codes for all the drives). https://www.backblaze.com/SMART
  • Plan for silent corruption. Bad drive looks exactly like a good drive.
  • Put replication above the file system.
  • Run out of resources simultaneous. Hardware and software together. Avoid having CPU pegged and your memory unused. Have your resources in balance, tweak over time.
  • Model and monitor storage burn. It’s important not to have too much or too little storage. Leading indicator is not storage, it’s bandwidth.
  • Business processes. Design for failure, but fix failures quickly. Drives will die, it’s what happens at scale.
  • Create repeatable repairs. Avoid the need for specialized people to do repair. Simple procedures: either swap a drive or swap a pod. Requires 5 minutes of training.
  • Standardize on the pod chassis. Simplifies so many things…
  • Use ROI to drive automation. Sometimes doing things twice is cheaper than automation. Know when it makes sense.
  • Workflow for storage buffer. Treat buffer in days, not TB. Model how many days of space available you need. Break into three different buffer types: live and running vs. in stock but not live vs. parts.
  • Culture: question “conventional wisdom”. No hardware worshippers. We love our red storage boxes, but we are a software team.
  • Agile extends to hardware. Storage Pod Scrum, with product backlog, sprints, etc.
  • Relentless focus on cost: Is it required? Is there a comparable lower cost option? Can business processes work around it? Can software work around it?

 

f4: Facebook’s Warm BLOB Storage System
Satadru Pan, Software Engineer, Facebook

  • White paper “f4: Facebook’s Warm BLOB Storage System” at http://www-bcf.usc.edu/~wyattllo/papers/f4-osdi14.pdf
  • Looking at how data cools over time. 100x drop in reads in 60 days.
  • Handling failure. Replication: 1.2 * 3 = 3.6. To lose data we need to lose 9 disks or 3 hosts. Hosts in different racks and datacenters.
  • Handling load. Load spread across 3 hosts.
  • Background: Data serving. CDN protects storage, router abstracts storage, web tier adds business logic.
  • Background: Haystack [OSDI2010]. Volume is a series of blobs. In-memory index.
  • Introducing f4: Haystack on cells. Cells = disks spread over a set of racks. Some compute resource in each cell. Tolerant to disk, host, rack or cell failures.
  • Data splitting: Split data into smaller blocks. Reed Solomon encoding, Create stripes with 5 data blocks and 2 parity blocks.
  • Blobs laid out sequentially in a block. Blobs do not cross block boundary. Can also rebuild blob, might not need to read all of the block.
  • Each stripe in a different rack. Each block/blob split into racks. Mirror to another cell. 14 racks involved.
  • Read. Router does Index read, Gets physical location (host, filename, offset). Router does data read. If data read fails, router sends request to compute (decoders).
  • Read under datacenter failure. Replica cell in a different data center. Router proxies read to a mirror cell.
  • Cross datacenter XOR. Third cell has a byte-by-byte XOR of the first two. Now mix this across 3 cells (triplet). Each has 67% data and 33% replica. 1.5 * 1.4 = 2.1X.
  • Looking at reads with datacenter XOR. Router sends two read requests to two local routers. Builds the data from the reads from the two cells.
  • Replication factors: Haystack with 3 copies (3.6X), f4 2.8 (2.8X), f4 2.1 (2.1X). Reduced replication factor, increased fault tolerance, increase load split.
  • Evaluation. What and how much data is “warm”?
  • CDN data: 1 day, 0.5 sampling. BLOB storage data: 2 week, 0.1%, Random distribution of blobs assumed, the worst case rates reported.
  • Hot data vs. Warm data. 1 week – 350 reads/sec/disk, 1 month – 150r/d/s, 3 months – 70r/d/s, 1 year 20r/d/s. Wants to keep above 80 reads/sec/disk. So chose 3 months as divider between hot and warm.
  • It is warm, not cold. Chart of blob age vs access. Even old data is read.
  • F4 performance: most loaded disk in cluster: 35 reads/second. Well below the 80r/s threshold.
  • F4 performance: latency. Chart of latency vs. read response. F4 is close to Haystack.
  • Conclusions. Facebook blob storage is big and growing. Blobs cool down with age very rapidly. 100x drop in reads in 60 days. Haystack 3.6 replication over provisioning for old, warm data. F4 encodes data to lower replication to 2.1X, without compromising performance significantly.

 

Pelican: A Building Block for Exascale Cold Data Storage
Austin Donnelly, Principal Research Software Development Engineer, Microsoft

  • White paper “Pelican: A building block for exascale cold data storage” at http://research.microsoft.com/pubs/230697/osdi2014-Pelican.pdf
  • This is research, not a product. No product announcement here. This is a science project that we offer to the product teams.
  • Background: Cold data in the cloud. Latency (ms. To hours) vs. frequency of access. SSD, 15K rpm HDD, 7.2K rpm HDD, Tape.
  • Defining hot, warm, archival tiers. There is a gap between warm and archival. That’s were Pelican (Cold) lives.
  • Pelican: Rack-scale co-design. Hardware and software (power, cooling, mechanical, HDD, software). Trade latency for lower cost. Massive density, low per-drive overhead.
  • Pelican rack: 52U, 1152 3.5” HDD. 2 servers, PCIe bus stretched rack wide. 4 x 10Gb links. Only 8% of disks can spin.
  • Looking at pictures of the rack. Very little there. Not many cables.
  • Interconnect details. Port multiplier, SATA controller, Backplane switch (PCIe), server switches, server, datacenter network. Showing bandwidth between each.
  • Research challenges: Not enough cooling, power, bandwidth.
  • Resource use: Traditional systems can have all disks running at once. In Pelican, a disk is part of a domain: power (2 of 16), cooling (1 of 12), vibration (1 of 2), bandwidth (tree).
  • Data placement: blob erasure-encoded on a set of concurrently active disks. Sets can conflict in resource requirement.
  • Data placement: random is pretty bad for Pelican. Intuition: concentrate conflicts over a few set of disks. 48 groups of 24 disk. 4 classes of 12 fully-conflicting groups. Blob storage over 18 disks (15+3 erasure coding).
  • IO scheduling: “spin up is the new seek”. All our IO is sequential, so we only need to optimize for spin up. Four schedulers, with 12 groups per scheduler, only one active at a time.
  • Naïve scheduler: FIFO. Pelican scheduler: request batching – trade between throughput and fairness.
  • Q: Would this much spin up and down reduce endurance of the disk. We’re studying it, not conclusive yet, but looking promising so far.
  • Q: What kind of drive? Archive drives, not enterprise drives.
  • Demo. Showing system with 36 HBAs in device manager. Showing Pelican visualization tool. Shows trays, drives, requests. Color-coded for status.
  • Demo. Writing one file: drives spin up, request completes, drives spin down. Reading one file: drives spin up, read completes, drives spin down.
  • Performance. Compare Pelican to a mythical beast. Results based on simulation.
  • Simulator cross-validation. Burst workload.
  • Rack throughput. Fully provisioned vs. Pelican vs. Random placement. Pelican works like fully provisioned up to 4 requests/second.
  • Time to first byte. Pelican adds spin-up time (14.2 seconds).
  • Power consumption. Comparing all disks on standby (1.8kW) vs. all disks active (10.8kW) vs. Pelican (3.7kW).
  • Trace replay: European Center for Medium-range Weather Forecast. Every request for 2.4 years. Run through the simulator. Tiering model. Tiered system with Primary storage, cache and pelican.
  • Trace replay: Plotting highest response time for a 2h period. Response time was not bad, simulator close to the rack.
  • Trace replay: Plotting deepest queues for a 2h period. Again, simulator close to the rack.
  • War stories. Booting a system with 1152 disks (BIOS changes needed). Port multiplier – port 0 (firmware change needed). Data model for system (serial numbers for everything). Things to track: slots, volumes, media.

 

Torturing Databases for Fun and Profit
Mai Zheng, Assistant Professor Computer Science Department - College of Arts and Sciences, New Mexico State University

  • White paper “Torturing Databases for Fun and Profit” at https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-zheng_mai.pdf
  • Databases are used to store important data. Should provide ACID properties: atomicity, consistency, isolation, durability – even under failures.
  • List of databases that passed the tests: <none>. Everything is broken under simulated power faults.
  • Power outages are not that uncommon. Several high profile examples shown.
  • Fault model: clean termination of I/O stream. Model does not introduce corruption/dropping/reorder.
  • How to test: Connect database to iSCSI target, then decouple the database from the iSCSI target.
  • Workload example. Key/value table. 2 threads, 2 transactions per thread.
  • Known initial state, each transaction updates N random work rows and 1 meta row. Fully exercise concurrency control.
  • Simulates power fault during our workload. Is there any ACID violation after recovery? Found atomicity violation.
  • Capture I/O trace without kernel modification. Construct a post-fault disk image. Check the post-fault DB.
  • This makes testing different fault points easy. But enhanced it with more context, to figure out what makes some fault points special.
  • With that, five patterns found. Unintended update to the mmap’ed blocks. Pattern-based ranking of where fault injections will lead to pattern.
  • Evaluated 8 databases (open source and commercial). Not a single database could survive.
  • The most common violation was durability. Some violations are difficult to trigger, but the framework helped.
  • Case study: A TokyoCabinet Bug. Looking at the fault and why the database recovery did not work.
  • Pattern-based fault injection greatly reduced test points while achieving similar coverage.
  • Wake up call: Traditional testing methodology may not be enough for today’s complex storage systems.
  • Thorough testing requires purpose-built workloads and intelligent fault injection techniques.
  • Different layers in the OS can help in different ways. For instance, iSCSI is an ideal place for fault injection.
  • We should bridge the gaps in understanding and assumptions. For instance, durability might not be provided by the default DB configuration.

 

Personal Cloud Self-Protecting Self-Encrypting Storage Devices
Robert Thibadeau, Ph.D., Scientist and Entrepreneur, CMU, Bright Plaza
http://www.snia.org/sites/default/files/DSS-Summit-2015/presentations/RobertThibadeau_Personal%20Cloud.pdf

  • This talk is about personal devices, not enterprise storage.
  • The age of uncontrolled data leaks. Long list of major hacks recently. All phishing initiated.
  • Security ~= Access Control.  Security should SERVE UP privacy.
  • Computer security ~= IPAAAA, Integrity, Private, Authentication, Authorization, Audit, Availability. The first 3 are encryption, the other aren’t.
  • A storage device is a computing device. Primary host interface, firmware, special hardware functions, diagnostic parts, probe points.
  • For years, there was a scripting language inside the drives.
  • TCG Core Spec. Core (Data Structures, Basic Operations) + Scripting (Amazing use cases).
  • Security Provider: Admin, Locking, Clock, Forensic Logging, Crypto services, internal controls, others.
  • What is an SED (Self-Encrypting Device)? Drive Trust Alliance definition: Device uses built-in hardware encryption circuits to read/write data in/out of NV storage.
  • At least one Media Encryption Key (MEK) is protected by at least one Key Encryption Key (KEK, usually a “password”).
  • Self-Encrypting Storage. Personal Storage Landscape. People don’t realize how successful it is.
  • All self-encrypting today: 100% of all SSDs, 100% of all enterprise storage (HDD, SSD, etc), all iOS devices, 100% of WD USB HDDs,
  • Much smaller number of personal HDDs are Opal or SED. But Microsoft Bitlocker supports “eDrive” = Opal 2.0 drives of all kinds.
  • You lose 40% of performance of a phone if you’re doing software encryption. You must do it in hardware.
  • Working on NVM right now.
  • Drive Trust Alliance: sole purpose to facilitate adoption of Personal SED. www.drivetrust.org
  • SP-SED Rule 1 – When we talk about cloud things, every personal device is actually in the cloud so… Look in the clouds for what should be in personal storage devices.
  • TCG SED Range. Essentially partitions in the storage devices that have their own key. Bitlocker eDrive – 4 ranges. US Government uses DTA open source for creating resilient PCs using ranges. BYOD and Ransomware protection containers.
  • Personal Data Storage (PDS). All data you want to protect can be permitted to be queried under your control.
  • Example: You can ask if you are over 21, but not what your birthday is or how old you are, although data is in your PDS.
  • MIT Media Lab, OpenPDS open source offered by Kerberos Consortium at MIT.
  • Homomorphic Encryption. How can you do computing operations on encrypted data without ever decrypting the data. PDS: Ask questions without any possibility of getting at the data.
  • It’s so simple, but really hard to get your mind wrapped around it. The requests come encrypted, results are encrypted and you can never see the plaintext over the line.
  • General solution was discovered but it is not computationally infeasible (like Bitcoin). Only in the last few years (2011) it improved.
  • HE Cloud Model and SP-DED Model. Uses OAuth. You can create personal data and you can get access to questions to your personal data. No plain text.
  • Solution for Homomorphic Encryption. Examples – several copies of the data. Multiple encryption schemes. Each operation (Search, Addition, Multiplication) uses a different scheme.
  • There’s a lot of technical work on this now. Your database will grow a lot to accommodate these kinds of operations.
  • SP-SED Rule 2 – Like the internet cloud: if anybody can make money off an SP-SED, then people get really smart really fast… SP-SED should charge $$ for access to the private data they protect.
  • The TCG Core Spec was written with this in mind. PDS and Homomorphic Encryption provide a conceptual path.
  • Challenges to you: The TCG Core was designed to provide service identical to the Apple App Store, but in Self-Protecting Storage devices. Every personal storage device should let the owner of the device make money off his private data on it.

 

Hitachi Data Systems - Security Directions and Trends
Eric Hibbard, Chair SNIA Security Technical Working Group, CTO Security and Privacy HDS

  • Protecting critical infrastructure. No agreement on what is critical.
  • What are the sections of critical infrastructure (CI)? Some commonality, but no agreement. US=16 sectors, CA=10, EU=12, UK=9, JP=10.
  • US Critical Infrastructure. Less than 20% controlled by the government. Significant vulnerabilities. Good news is that cybersecurity is a focus now. Bad news: a lot of interdependencies (lots of things depend on electric power).
  • Threat landscape for CI. Extreme weather, pandemics, terrorism, accidents/technical failures, cyber threats.
  • CI Protection – Catapulted to the forefront. Several incidents, widespread concern, edge of cyber-warfare, state-sponsored actions.
  • President Obama declared a National Emergency on 04/01/2015 due to rising number of cyberattacks.
  • CI protection initiatives. CI Decision-making organizations, CIP decisions. CIP decision-support system. The goal is to learn from attacks, go back and analyze what we could have done better.
  • Where is the US public sector going? Rethinking strategy, know what to protect, understand value of information, beyond perimeter security, cooperation.
  • Disruptive technologies:  Mobile computing, cloud computing, machine-to-machine, big data analytics, industrial internet, Internet of things, Industry 4.0, software defined “anything”. There are security and privacy issues for each. Complexity compounded if used together.
  • M2M maturity. Machine-to-machine communication between devices that are extremely intelligent, maybe AI.
  • M2M analytics building block. Big Data + M2M. This is the heart and soul of smart cities. This must be secured.
  • IoT. 50 billion connected objects expected by 2020. These will stay around for a long time. What if they are vulnerable and inside a wall?
  • IoT will drive big data adoption. Real time and accurate data sensing. They will know where you are at any point in time.
  • CI and emerging technology. IoT helps reduce cost, but it increases risks.
  • Social Infrastructure (Hitachi View). Looking at all kinds of technologies and their interplay. It requires collaborative system.
  • Securing smart sustainable cities. Complex systems, lots of IoT and cloud and big data, highly vulnerable. How to secure them?

 

Enterprise Key Management & KMIP: The Real Story  - Q&A with EKM Vendors
Moderator: Tony Cox, Chair SNIA Storage Security Industry Forum, Chair OASIS KMIP Technical Committee
Panelists: Tim Hudson, CTO, Cryptsoft
Nathan Turajski, Senior Product Manager, HP
Bob Lockhart, Chief Solutions Architect, Thales e-Security, Inc
Liz Townsend, Director of Business Development, Townsend Security
Imam Sheikh, Director of Product Management, Vormetric Inc

  • Goal: Q&A to explore perspective in EKM, KMIP.
  • What are the most critical concerns and barriers to adoption?
  • Some of developers that built the solution are no longer there. Key repository is an Excel spreadsheet. Need to explain that there are better key management solutions.
  • Different teams see this differently (security, storage). Need a set of requirements across teams.
  • Concern with using multiple vendors, interoperability.
  • Getting the right folks educated about basic key management, standards, how to evaluate solutions.
  • Understanding the existing solutions already implemented.
  • Would you say that the OASIS key management has progressed to a point where it can be implemented with multiple venders?
  • Yes, we have demonstrated this many times.
  • Trend to use KMIP to pull keys down from repository.
  • Different vendors excel in different areas and complex system do use multiple vendors.
  • We have seen migrations from one vendor to another. The interoperability is real.
  • KMIP has become a cost of entry. Vendors that do not implement it are being displaced.
  • It’s not just storage. Mobile and Cloud as well.
  • What’s driving customer purchasing? Is it proactive or reactive? With interoperability, where is the differentiation?
  • It’s a mix of proactive and reactive. Each vendor has different background and different strengths (performance, clustering models). There are also existing vendor relationships.
  • Organizations still buy for specific applications.
  • It’s mixed, but some customers are planning two years down the line. One vendor might not be able to solve all the problems.
  • Compliance is driving a lot of the proactive work, although meeting compliance is a low bar.
  • Storage drives a lot of it, storage encryption drives a lot of it.
  • What benefits are customers looking for when moving to KMIP? Bad guy getting to the key, good guy losing the key, reliably forget the key to erase data?
  • There’s quote a mix of priorities. The operational requirements not to disrupt operations. Assurances that a key has been destroyed and are not kept anywhere.
  • Those were all possible before. KMIP is about making those things easier to use and integrate.
  • Motivation is to follow the standard, auditing key transitions across different vendors.
  • When I look at the EU regulation, cloud computing federating key management. Is KMIP going to scale to billions of keys in the future?
  • We have vendors that work today with tens of billions of key and moving beyond that. The underlying technology to handle federation is there, the products will mature over time.
  • It might actually be trillions of keys, when you count all the applications like the smart cities, infrastructure.
  • When LDAP is fully secure and everything is encrypted. How does secure and unsecure merge?
  • Having conversations about different levels of protections for different attributes and objects.
  • What is the different from a local key management to a remote or centralized approaches?
  • There are lots of best practices in the high scale solutions (like separation of duties), and not all of them are there for the local solution.
  • I don’t like to use simple and enterprise to classify. It’s better to call them weak and strong.
  • There are scenarios where the key needs to local for some reason, but need to secure the key, maybe have a hybrid solution with a cloud component.
  • Some enterprises think in terms of individual projects, local key management. If they step back, they will see the many applications and move to centralized.
  • With the number of keys grows will we need a lot more repositories with more interop?
  • Yes. It is more and more a requirement, like in cloud and mobile.
  • Use KMIP layer to communicate between them.
  • We’re familiar with use cases? What about abuse cases? How to protect that infrastructure?
  • It goes back to not doing security by obscurity.
  • You use a standard and audit the accesses. The system will be able to audit, analyze and alert you when it sees these abuses.
  • The repository has to be secure, with two-factor authentication, real time monitoring, allow lists for who can access the system. Multiple people to control your key sets.
  • Key management is part of the security strategy, which needs to be multi-layered.
  • Simple systems and a common language is a vector for attack, but we need to do it.
  • Key management and encryption is not the end all and be all. There must be multiple layers. Firewall, access control, audit, logging, etc. It needs to be comprehensive.

 

Lessons Learned from the 2015 Verizon Data Breach Investigations Report
Suzanne Widup, Senior Analyst, Verizon
http://www.snia.org/sites/default/files/DSS-Summit-2015/presentations/SuzanneWidupLearned_Lessons_Verizon.pdf

  • Fact based research, gleaned from case reports. Second year that we used data visualization. Report at http://www.verizonenterprise.com/DBIR/2015/
  • 2015 DBIR: 70 contributed organizations, 79,790 security incidents, 2,122 confirmed data breaches, 61 countries
  • The VERIS framework (actor – who did it, action - how they did it, asset – what was affected, attribute – how it was affected). Given away for free.
  • We can’t share all the data. But some if it publicly disclosed and it’s in a GitHub repository as JSON files. http://www.vcdb.org.
  • You can be a part of it. Vcdb.org needs volunteers – be a security hero.
  • Looking at incidents vs. breaches. Divided by industry. Some industries have higher vulnerabilities, but a part of it is due to visibility.
  • Which industries exhibit similar threat profiles? There might be other industries that look similar to yours…
  • Zooming into healthcare and other industries with similar threat profiles.
  • Threat actors. Mostly external. Less than 20% internal.
  • Threat actions. Credentials (down), RAM scrapers (up), spyware/keyloggers (down), phishing (up).
  • The detection deficit. Overall trend is still pretty depressing. The bad guys are innovating faster than we are.
  • Discovery time line (from 2015). Mostly discovered in days or less.
  • The impact of breaches. We’re were not equipped to measure impact before. This year we partnered with insurance partners. We only have 50% of what is going on here.
  • Plotting the impact of breaches. If you look at the number of incidents, it was going down. If you look at the records lost, it is growing.
  • Charting number of records (1 to 100M) vs. expected loss (US$). There is a band from optimist to pessimist.
  • The nefarious nine: misc errors, crimeware, privilege misuse, lost/stolen assets, web applications, denial of service, cyber-espionage, point of sale, payment card skimmers.
  • Looks different if you use just breaches instead of all incidents. Point of sale is higher, for instance.
  • All incidents, charted over time (graphics are fun!)
  • More charts. Actors and the nine patterns. Breaches by industry.
  • Detailed look at point of sale (highest in accommodation, entertainment and retail), crimeware, cyber-espionage (lots of phishing), insider and privilege misuse (financial motivation), lost/stolen devices, denial of service.
  • Threat intelligence. Share early so it’s actionable.
  • Phishing for hire companies (23% of recipients open phishing messages, 11% click on attachments)
  • 10 CVEs account for 97% of exploits. Pay attention to the old vulnerabilities.
  • Mobile malware. Android “wins” over iOS.
  • Two-factor authentication and patching web servers mitigates 24% of vulnerabilities each.
Viewing all 182 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>