The current WebAPI, opens in a new tab has been in production serving the public for multiple years. It handles millions of requests for spatial information every month and is free to use with a simple to create API key, opens in a new tab. While it receives constant enhancements, opens in a new tab and bug fixes, opens in a new tab, there have been no breaking changes to the API, nor have there been any major architectural updates. That is about to change with a major update currently in the works.
The interesting and often wild office discussions about how to improve the WebAPI have not been backed by concrete efforts until now. Implementing these ideas is now possible since key technologies have made amazing advancements while others have matured. UGRC is ready to combine our ideas with new technologies to make the WebAPI even better. Work has already started on what will most likely be version 2, opens in a new tab of the WebAPI. And we hope to bring the same improvements to version 1 without interruption.
The History
The WebAPI proof of concept was built using ASP.NET WCF, opens in a new tab. After getting positive internal feedback (and usage in production, of course), we wrote version 1 of the WebAPI using ASP.NET WebAPI 3. The ASP.NET WebAPI then merged with ASP.NET MVC, and we are currently running on ASP.NET MVC 5.
RavenDB, opens in a new tab is the user and key store and Redis, opens in a new tab is being used for analytics and reporting. Redis and RavenDB are both running as Windows services and the WebAPI and developer website are hosted as separate sites in IIS. Supplementary data is cached from Google Sheets to allow for easy modifications.
Reading that last paragraph may have made you think that it would be hard to replicate or upgrade the entire WebAPI system. If that’s the case—you are correct, opens in a new tab. The list of installation instructions and documentation for the version 1 WebAPI system are as frustrating as they are long. A design goal of version 2 is to eliminate that friction as much as possible.
The Future
The new framework we plan to use, ASP.NET Core, opens in a new tab, is a ground-up redesign of ASP.NET 4 with the important design goals of being cross platform, lean, and modular. From our experience, this framework is fast, opens in a new tab! The WebAPI in development is running on ASP.NET Core 2.2 preview 2. From our testing, it can handle much more throughput than the current system. A huge benefit of this framework is that developers at the UGRC can run or develop this code using Visual Studio Code, opens in a new tab and Visual Studio IDE, opens in a new tab on Windows, Mac, and Linux. This creates a great developer experience by allowing any and all development environments. The cross platform abilities enables this project to enter the world of Docker, opens in a new tab and Kubernetes, opens in a new tab.
Containers
Version 2 of the WebAPI is being built for containers. It will run without them, but building the correct runtime environment is much simpler using them. The system is currently comprised of four containers that will likely expand to six or more.
- The ASP.NET Core WebAPI service
- The ASP.NET Core developer website
- The PostgreSQL database
- The Redis database
- The API Explorer website *
- The Google sheet synchronization service *
- The logging/reporting service (Kibana) *
*not yet built
ASP.NET Core runs very well in a container and will drive the WebAPI and developer websites. We chose to replace RavenDB with Postgres as the user and key store. This change is because PostgreSQL runs amazing in a Docker container for development. It isn’t quite durable enough for production, but since the major cloud providers offer cloud SQL equivalents to Postgres, the WebAPI can use a managed cloud database for production.
Redis remains our choice for in-memory caching and also runs very well in a container. The main goal of using containers is that docker compose, opens in a new tab can start the entire system with one command. Using Docker, anyone can play with the system without having to install additional software.
Orchestration
Our version 2 WebAPI will run many containers at the same time, and managing how those containers work together is what Kubernetes was designed to do. Kubernetes allows UGRC to scale the WebAPI automatically or using a CLI tool, opens in a new tab. Kubernetes will update our containers when UGRC makes changes to the code and can do so without any downtime. It can also monitor and manage the health of our containers to maintain our SLA.
To recap, UGRC is writing code and organizing it into separate containers. The containers are being managed by Kubernetes, but they still need compute infrastructure to run on, so why stop there?
We have taken it one step further with Terraform, opens in a new tab. Terraform allows UGRC to template and script the infrastructure that Kubernetes and the containers run on. To build on the magic of running the WebAPI with one command, with Terraform, we can have the WebAPI running in the Google Cloud Platform with one command—all of the nodes, networks, etc. We are very excited for this level of predictability and declarative configuration to run this important UGRC system.
The Minutia
There are a few notable changes to the WebAPI source code that happened during this transition that only a software developer would appreciate.
One major improvement is the replacement of our loose Command pattern, opens in a new tab implementation for the Mediator pattern, opens in a new tab. I have been thoroughly enjoying Jimmy Bogard, opens in a new tab’s MediatR, opens in a new tab implementation since it forces you to write better code with clean separation.
Like the command pattern, testing your handlers is straightforward because of that clean separation. Another highlight of MediatR are its behaviors. Behaviors allow handlers to be be chained together to create a pipeline where they operate on the input and output of other handlers. This allows for simple extensibility without modifying the original code. Registering the behavior with your dependency injection container will make sure it is used. Read more about the mediator pattern and its benefits on this great blog, opens in a new tab.
Given the WebAPI’s history, we were very happy to take advantage of new language features, like replacing all of the C# 5.0 TPL, opens in a new tab Task continuations with async and await. We are moving away from config files for our application secrets in favor of environment variables and the Options pattern, opens in a new tab. This allows for a seamless experience when running locally (with or without docker) and in Kubernetes.
Google Sheets is a lynchpin in the version 1 of the WebAPI since required data is loaded from a handful of sheets on application startup. If the Google Sheets API is unavailable, the WebAPI will be in an unhealthy state.
In new version 2, we imported a current snapshot of the sheets data into Postgres and are creating a microservice to keep the data synchronized. This way the application is guaranteed to start in a healthy state, and the semi-static reference data can be updated when it changes—either on a schedule or when notified via web hooks. This will be a major improvement over the version 1 WebAPI.
Version 1 of the WebAPI contains a canary endpoint to make sure the required services are available. ASP.NET Core makes these health checks, opens in a new tab a first class feature. We can now write health checks that not only let Kubernetes and other monitoring systems know that we are in a healthy state but when things go wrong, we will know exactly what dependency is not functioning properly.
It is a great time to be a software engineer. There are so many services and software to improve our effectiveness, make us more efficient, and help us deliver a better overall experience to our users. If you have not tried any of these software, you should strongly consider it.
It has been a fun and exciting experience developing the next version of the WebAPI. After all, if the work you are doing isn’t enticing and fun, maybe it’s time to make a change.