Marcus has been about to replace his personal workstation for over 1 year now.
And today, Monday Feb 27 2017 I can say it is at last done.
Athena is alive.
In the background for the last 6 months I've been extensively testing and prototyping the base Operating System that Athena will use. I certainly want to do better than Windows 2012 Server R2.
This article discusses which OS I chose to install and why. I'll report on the hardware, which IMHO was the easier part later.
The idea is this. My current workstation is about 5 years old. Having said that it has multiple 4K screens, 24 processors and over 40GB of memory. It runs well and quietly. So why change?
- It's 5 years old and quite ancient
- I want faster disk with NVMe technology, specifically at OS booting and for Virtualised OS
- Might be nice to have a smaller server, physically
- Water cooling and lower Power CPU's nice.
- Better support for USB 3.1
- 10 GBe Networking a plus
The above arguments are bolstered emotionally by:
- I use this computer for over 4 hours per day
- You can't imagine how grumpy I am if its broken. Basically, life as I know it almost stops
- So I want to preemptively replace my workstation, not wait until it falls apart.
- I deserve it!
The Big Picture
The requirements are:
- New Workstation to last upto 5 years with minimum of upgrading
- Power: a Lot of power, say more than 48 processors :-)
- Memory: A lot of memory, say upto 1TB memory capacity
- Needs to support Multiple 4K Screens
- Run some Windows apps which don't have OSX or Linux alternatives
- Quiet and Compact would be Nice
Virtualisation Issues & Graphics Cards
On the face of it, it would seem possible to run a base Virtualisation Software e.g. Hyper-V or ESXi or something under Linux, and on top run the OS of your choice.
The big catch though is Graphics Cards.
To explain. My old workstation has multiple 4K monitors and needs has a Nvidia 9xx graphics card powering it.
None of the Hardware Virtualisation technologies support this well. With one exception actually, and that is ESXi using a feature called PCI Passthru.
So with this limiation the practical workstation choices reduce to ESXi or a native OS.
In a nutshell the OSX Gui to me is a 1990's joke. It is fat, inelegant and the crudeness of Finder drives me crackers
Next, Linux has some OK but not outstanding GUI's. The main issue is surely that there are so many of them. My usual Linux distribution, Mint, come with 3 different GUI options. And in general Linux distributions can be made to look fantasic. However:
- that look and feel may be totally obsoleted by the next great thing
- When you re-install or upgrade you may need to start from scratch and labouriously reapply the tailoring, if you can remember it.
So yes, The classic Windows Desktop, its Gui presentation is far superior IMHO.
Base OS Choices then:
+ Pretty cheap
+ Add ons like Virus protection are cheap
+ Support 2 Sockets and upto 256 Cores and 128GB plus. Enough
+ Can run VMware or an.other virtualisation software
I can't really fault it except to say that since Windows 2016 is the premiere product I think that would have a better reliability track record. After all most of the Web runs on Windows server, crashes are not acceptable.
Windows 2016 server
+ I bought it already
+ A more professional version of Windows 10
+ Able to run Windows programs natively. Example my scanner software must run under Windows. I have a few specific Windows editing and MP3 processing tools also.
Why not a guest? It's not a very timely solution to have a Windows Virtual Machine (VM) running under (say) your Linux Workstation. Because every time you want to scan, you need to load the VM and this is fiddly and time consuming.
+ Can run VMware or an.other virtualisation software
+ Native Hyper V support
- Anti virus and other addons cost a fortune
- Some W10 programs wont run, trivial example: Crucial Disk Suite
- Support of Audio and Bluetooth: I've had problems in the past. Definitely W10 is superior here
+ Native Bare Metal Backup
+ Windows Storage Spaces
+ Ethernet NIC teaming
ESXi as a base bare metal hypervisor
Vcenter 6.5 needs to be installed to perform management of the cluster
You would have one Main OS, example Windows 2016 server, which would be your default virtualisation
Using the PCI Passthru feature of ESXi you can dedicate the Physical PCIe graphics card to the Main OS and have that OS (and only that OS)
If I ran ESXi as a base level workstation supervisor, then a specific OS with PCI passthru as my daily driver I would get
+ blazing performance: I swear the VM performance seemed to be faster than native NVMe, which is impossible
+ Support of multiple Physical 4K screens using PCI Passthru. (I did it during testing)
- Costly setup. Although, its possible to join and pay VMUG and in this way get non commercial ESXi licenses that don't expire
(Without VMUG your cost will be thousands of GBP not hundreds)
- PCI passthru is very fiddly to setup and not practical for the addition of specific devices on an adhoc basis. I did get the Nvidia 1050 graphics card passed thru to a Windows 2016 server, but boy it was rather non trivial.
Overall ESXi is the ultimately professional setup, beating Native OS and VMware in many respects, just awfully complicated. Also I feel that frequently testing different Guest OS via VM's would mean going to the Vcenter or HTTP ESXi controlling console and this is simply a lot more effort than VMWare for setup but more importantly for maintenance. Example, how do you propose to backup these VM you use. How will you issue copy VM files from your workstation to (in our case) NAS? If you are running VM's in a regular OS environment like Windows or Linux you can at least setup some regular syncback or rsync task. But for ESXi this would be more complicated.
Microsoft has a free Hypervisor based on Windows 2016 server that you can download. However there is a catch or two. Like ESXi its not a GUI based Operating System or Hypervisor. Once installed you do a basic character based config as shown above. Then you need to manage this e.g. create a Virtual machine using a remote client.
And of course any Virtual machines you do install won't be able to run multiple 4K monitors, so not so good for our Workstation environment.
In general Hyper-V cannot do PCI passthru so any Virtual machine will only get virtualised perhiperals, so this will be slower and probably really only okay for USB devices.
I use Linux Mint and Centos flavours of Linux on a daily basis. In fact my laptop is Centos dual booting.
But can I use Linux as my base Desktop Workstation OS?
+ No need for any Anti Virus OSlevel software, saves money and improves performance
- No support found for my Nvidia 10xx cards, causing installation failure, Nvidia 9xx cards worked fine
- GUI vary between Linux. NO guarantee that yours will be supported mainstream in the future, and customisation is painful
+ Performance of the OS is fantastic, superior to Windows or Linux
- Poor Support for Backup, Bare Metal recovery or OS Upgrade
- Between Poor and non existent support for big hitters like Dropbox, Google Drive, Chrome Browser, Spotify.
- Filename conversion issues: Because Windows is case sensitive so migration of the millions of files I currently hold under NTFS could be problematic
In fact whilst testing neither Centos or Mint would not install using my latest Nvidia graphics card. I then used an older 9xx card and continued testing. I used VMware under Mint and I'm afraid it's just a poor solution compared to the Windows equivalent. Further under Linux the scanner software was compromised and there is no Garmin support.
OSX can't run native on my Hand Built Workstation. The best you can achieve is a Hackintosh OS image typically running inside a Virtual Machine (VM). If I had this as my principle Workstation environment
- That bloody awful GUI and finder application
- VM would allow stable OSX emulation because before any upgrade you can do a VM snapshot and rollback if a problem
- VM environment would not support my multiple screen requirement
I do like the fact that OSX contains a native UNIX underbelly, however in daily use a Linux system is clearly superior and a Cygwin addition to Windows provides just as much as OSX command line UNIX flexibility.
The Winner: Native Windows 2016 Server
I really wanted the winner to be ESXi for my next Workstation OS, however eventually I have had to go with Windows 2016. The only logical choice. And why?
+ Direct support for some Windows programs I want to frequently run, like Fujitsu scanner software
- Better support of programs I can't live without: Google Chrome, Dropbox, Google Drive, Spotify, Garmin Connect
+ VMware runs best here, very easy to use cf ESXi: snapshots, backup, reconfig
+ Superior support of ad hoc technology like Webcams and Printers
- I still think the GUI is best
+ I can have a UNIX command line, and indeed make daily use of this via Cygwin
+ Because of multi display requirement the only competition to native Windows was ESXi and this would provide a slightly fancier solution but with untold levels of greater complexity.
So Windows 2016 Server it is then!