I have been lately experimenting with server side development with Swift, primarily to explore the potential of near native performance in a high level language (I don't consider Go high level and find it too spartan to my taste in terms of language level features) as well as a strongly typed system with built in support for null safety. Lack of garbage collection and ARC is uniquely interesting in itself.
The primary problem at this point is the anaemic ecosystem around web development. While the ecosystem around Vapor etc. is rapidly growing, it is far from what you would expect from Node ecosystem or Rails.
Vapor (the only framework I have used so far) offers a really nice type-safe API that caters to most common use cases. However it is still not production ready and the issue tracker has many open issues (some filed by myself). Situation with most popular libraries is the same. The language itself keeps introducing low level changes between iterations - and many libraries have platform specific issues (eg. things that compile in OS X don't in linux etc.)
A sophisticated ORM like ActiveRecord does not exist at the moment but surprisingly GraphQL library is pretty good (I will blog about this in near future).
The prevalent concurrency model is GCD (Managed thread-pool) which is battle-tested in IOS/OS X applications. But there are also parallel efforts to incorporate Go style concurrency and some frameworks claim it to be a much better model for web servers. It is likely one of these approaches will emerge as a more popular solution but it is not obvious at the moment.
Build times are also not ideal and because of the way Swift package manager currently works, a fresh build of a vapor project builds all the dependencies too which can lead to fairly large compilation time (> 5-6 minutes for trivial projects).
Also, perhaps the only decent Swift IDE at the moment is Xcode, and Xcode integration with Swift Package Manager is work in progress. In general Xcode lags behind sophisticated IDEs like IntelliJ in terms of features. While syntactical error reporting, auto-completion etc. work quite well, automated complex refactorings etc. are a pipedream at the moment. Things like jump to definition across modules (managed by SPM) often inexplicably fail. There are on-going attempts to make editor agnostic language server (SourceKitten) but I have not been able to get them to work with my favorite editor Emacs.
Also very surprisingly getting the built in REPL or XCode playgrouds to work with third party packages installed through Swift Package Manager is a PITA.
So all in all, Swift is a really elegant language (not without few eccentricities though) with a strong emphasis on Type-safety and high-level programming while at the same time being very approachable to new comers. Playgrounds are a really nice concept, language level error reporting and feedback is great.
But the ecosystem around web development is not yet ready for primetime. If you are building hobby projects or small micro-services it is a great avenue to explore. But if you are thinking about a large scale commercial application - hold on, and consider better supported alternatives at the moment or be prepared to seriously invest time and effort in evolving the foundational ecosystem itself.