When  Bad  Things  Happen         To  Good  Disks  aka   Disks  Don’t  Have  File   Descriptors   Erik  Riedel,  EMC   CloudOpen   August  2015   revision  3   right  picture  by  AusHn  Marshall  via  flickr/cc from  flickr/Blude,  floppy  disks  for  breakfast from  flickr/purplemaNish,  Broken  hard  drive? Problem  Overview   •  set  up  a  collecHon  of  10-­‐node  to  500-­‐node   Linux  clusters  at  100s  of  sites  worldwide   •  deployed,  managed,  monitored,  serviced  by  a   diverse  group  of  Ops  +  Service  folks   •  when  something  goes  (really)  wrong,  they  call   your  (cell)  phone   •  approach:  keep  it  simple,  make  it  easy,  be   proacHve,  turn  off  your  (cell)  phone What  Makes  It  Harder   •  each  node  has  60  disks   –  why  doesn't  smartd  report  on  all  my  disks?   –  /dev/sd? != /dev/sd* (actually /dev/sd[a-z]+) •  where  did  /dev/sddh come  from?     –  device  briefly  offline  =>  new  dev!!   •  disks  don't  have  file  descriptors   –  sg,  sd,  md,  dm,  lvm,  fs  (ext3,  ext4,  xfs,  btrfs)   •  SATA  disks  are  big  &  cheap  and  all,  but  can  be  a  bit   "unruly"...  temporary  disconnects   •  hardware  RAID  is  yucky   •  databases  are  ogen  stale Configurations •  high  capacity  drives                   (as  many  as  possible)   •  x86  servers/controllers         (as  few  as  possible)   •  SAS  backplanes/cables   (not  too  many,  not  too   few)   Figure 3 Promo Code 1 Front (tray pulled out) 14.1  drives/U   24 EMC Atmos Hardware Guide Example – Device names Disks(s): ONE NODE SCSI Device Block Device Enclosure Slot Serial Number SMART Status ----------- ------------ ---------- ---- ------------------- ------------ n/a /dev/md126 RAID vol n/a not supported n/a /dev/sg0 /dev/sda intl/sys 0 PWHHBZ7F GOOD /dev/sg1 /dev/sdb intl/sys 1 PWHGVT6F GOOD /dev/sg3 /dev/sdc /dev/sg2 C00 YVHSKHWA GOOD /dev/sg4 /dev/sdd /dev/sg2 A01 YVHRUYEA GOOD /dev/sg5 /dev/sde /dev/sg2 A02 YVHSSHXA GOOD /dev/sg6 /dev/sdf /dev/sg2 B00 YVHRL21A GOOD /dev/sg7 /dev/sdg /dev/sg2 C01 YVHSB98A GOOD /dev/sg8 /dev/sdh /dev/sg2 A03 YVHSJRRA GOOD /dev/sg9 /dev/sdi /dev/sg2 A00 YVHSMK7A GOOD /dev/sg10 /dev/sdj /dev/sg2 B01 YVHLVEND GOOD . . . . . . /dev/sg63 /dev/sdbj /dev/sg2 E07 YVHSB4BA GOOD Disks(s): ANOTHER NODE SCSI Device Block Device Enclosure Slot Serial Number SMART Status ----------- ------------ ---------- ---- ------------------- ------------ n/a /dev/md126 RAID vol n/a not supported n/a /dev/sg0 /dev/sda intl/sys 0 PWJMRV8D GOOD /dev/sg1 /dev/sdb intl/sys 1 PWJLVH2F GOOD /dev/sg4 /dev/sdu /dev/sg3 C00 YVK2EWWA GOOD /dev/sg5 /dev/sdx /dev/sg3 A01 YVJWLP3D GOOD /dev/sg6 /dev/sdbk /dev/sg3 A02 YVK078ED GOOD /dev/sg7 /dev/sdbl /dev/sg3 B00 YVK2V6SA GOOD /dev/sg8 /dev/sde /dev/sg3 C01 YVJWB5KD GOOD /dev/sg9 /dev/sdbm /dev/sg3 A03 YVK2V9BA GOOD /dev/sg10 /dev/sdbn /dev/sg3 A00 YVK1S2RA GOOD /dev/sg11 /dev/sdbo /dev/sg3 B01 YVK2V68A GOOD . . . . . . /dev/sg66 /dev/sddl /dev/sg3 E07 YVK3487A GOOD Example  –  DAE  reconnects   Jul 1 21:37:37 localhost kernel: mptbase ioc0 LogInfo(0x31130000) Code={IO Not Yet Executed}, SubCode(0x0000) Jul 1 23:50:06 localhost kernel: mptbase ioc1 LogInfo(0x31112000) Code={Reset}, SubCode(0x2000) Jul 1 23:50:09 localhost kernel: mptbase ioc1 LogInfo(0x31112000) Code={Reset}, SubCode(0x2000) Jul 1 23:50:12 20xx : WARNING : Disk Event : Disk is moved to DAE: Slot ID: 0 : Serial NO: WCAVY4897042 Jul 1 23:50:12 20xx : WARNING : Disk Event : Disk is moved to DAE: Slot ID: 0 : Serial NO: WCAVY5192630 Jul 1 23:50:13 20xx : WARNING : Disk Event : Disk is moved to DAE: Slot ID: 0 : Serial NO: WCAVY5186052 Jul 1 23:50:14 20xx : WARNING : Disk Event : Disk is moved to DAE: Slot ID: 0 : Serial NO: WCAVY3550485 Jul 1 23:50:14 20xx : WARNING : Disk Event : Disk is moved to DAE: Slot ID: 0 : Serial NO: WCAVY360702 (…all 60 disks…) Jul 1 23:50:15 20xx : ERROR : DAE Event : DAE (device path: /dev/sg66) lost. : Serial NO: , Device path: /dev/sg66, Device ID: 5000097a780747be Jul 1 23:50:15 20xx : WARNING : Disk Event : Disk is moved to DAE: Slot ID: 0 : Serial NO: WCAVY5349410 Jul 1 23:51:14 20xx : INFO : DAE Event : New DAE (device path: /dev/sg66) is added. : Serial NO: , Device path: /dev/sg66, Device ID: 5000097a780747be Jul 1 23:51:14 20xx : WARNING : Disk Event : Disk is moved to DAE: 5f4ad992-724e-48af-8cac-a68b7d859593 Slot ID: 11 : Serial NO: WCAVY5182031 , Device path: /dev/sdaq, Slot ID: Jul 1 23:51:14 20xx : WARNING : Disk Event : Disk is moved to DAE: 5f4ad992-724e-48af-8cac-a68b7d859593 Slot ID: 13 : Serial NO: WCAVY5186052 , Device path: /dev/sdas, Slot ID: (…all 60 disks…) Jul 1 23:51:16 20xx : WARNING : Disk Event : Disk is moved to DAE: e70905ad-5736-48d9-8a1b-a15a2d116825 Slot ID: 4 : Serial NO: WCAVY5349410 , Device path: /dev/sday, Slot ID: (outage ends, log ends) Reset  on  the  SAS/SATA  bus,  enclosure  idenHfiers  re-­‐assigned   “<NULL>”;  enclosure  returns  ager  68  seconds,  disks  are  assigned   back  where  they  belong.  EnHre  episode  lasts  70  seconds.   BUT  system  management  database  remembers  this  for  weeks. Example – Proactive Smarts erik-riedels-macbook-pro:logs er1p$ cat 2014-*/halreport | grep SUSP /dev/sg4 /dev/sdc /dev/sg3 C00 YVJZ8XRK SUSPECT: Reallocated(5)=99 /dev/sg49 /dev/sdav /dev/sg2 D10 YVK6378A SUSPECT: Reallocated(5)=35 /dev/sg45 /dev/sdaq /dev/sg3 B10 YVJZW8EA SUSPECT: Reallocated(5)=19 /dev/sg6 /dev/sde /dev/sg3 A02 YVK4UJ5A SUSPECT: Reallocated(5)=10 /dev/sg21 /dev/sdt /dev/sg3 E02 YVJG6X4D SUSPECT: Reallocated(5)=66 /dev/sg32 /dev/sdae /dev/sg3 C05 YVK25MKA SUSPECT: Reallocated(5)=78 /dev/sg35 /dev/sdag /dev/sg3 A06 YVJYBDSA SUSPECT: Reallocated(5)=43 /dev/sg15 /dev/sdn /dev/sg3 D00 YVJB5TAA SUSPECT: Reallocated(5)=42 /dev/sg58 /dev/sdbd /dev/sg3 C07 YVJYRKYA SUSPECT: Reallocated(5)=59 erik-riedels-macbook-pro:logs er1p$ cat 2014-*/halreport | grep FAIL /dev/sg12 /dev/sdl /dev/sg2 A04 YVJZMN3K FAILED: Reallocated(5)=110 /dev/sg60 /dev/sdbk /dev/sg3 E08 YVK2GNRA FAILED: Reallocated(5)=1577 /dev/sg37 /dev/sdai /dev/sg2 B09 YVJYR8KA FAILED: Reallocated(5)=101 /dev/sg41 /dev/sdam /dev/sg3 B08 YVJEZT7A FAILED: Reallocated(5)=682 erik-riedels-macbook-pro:logs er1p$ cat 2014-*/halreport | grep GOOD | wc -l 12228