Recently, I realized that I wasn’t as hands-on with microservices as I needed to be, so over the weekend, I created a web app that uses microservices to get a better feel for this system architecture paradigm. The app does a pretty simple task: given a bunch of URLs of online job postings, it fetches the job description text and creates a word cloud out of them. It’s quasi-useful in trying to pick up the major skills or buzzwords in a particular set of job postings.
Microservices are supposed to be independent, stateless apps that each do one small thing. In this app, which I called CloudFun, the tasks are: 1) retrieve the contents from the pages specified by the URLs; 2) extract all the words from the job description; 3) create a wordcloud. I had no trouble creating the microservices themselves, which I will describe further below. However, thinking about it, it didn’t seem like in practice it would be a good idea to expose these services directly on the Web, with no authentication, protection against DOS attacks or runaway consumption. Moreover, I needed an app that presented a user interface. Therefore, I also threw in an MVC server that would create a presentation layer. As you probably know, this patter is called an API Gateway. The architectural diagram is pretty simple.
Although I didn’t draw the diagram until later, I had a mental picture of how this would work. I believe in getting an early win, and building on what you know, so since I’ve done a whole lot of web scraping I jumped in to build the Extract Words service. Since these microservices don’t need a lot of web application components like session or user management, using Flask or Tornado seemed like good options. I chose Tornado because I had never used it and it had come up in recent conversations, so I was curious.
There are plenty of examples of how to create a Web app with Tornado, including some that talk about microservices. Building on those examples, along with the knowledge that the most commonly used language is JSON, I came up with the following code.
This is as simple as it can get. When this file is executed, in response to an HTTP GET request on the port 8888, the service reads a local file, parses it using the html5lib and BeautifulSoup, and returns the words in a JSON wrapper.
The service can be tested from the command line using curl:
$curl http://localhost:8888/api/v1/words
That’s it, I’d built a microservice. I was elated. I was nearly done! Okay, fine, maybe it shouldn’t return the same response every time from a local file. That seemed easy enough to fix, let me carry on. That’s the benefit of an early win.
While I was at it, I figured I needed to take care of a few more things. Not only did the service need to accept and respond to input content, but as an HTTP service it should also return at least a status code. Also, there was no way to test the core logic of the service (extracting text), without running it and making requests through HTTP, which seemed cumbersome. Finally, while this wasn’t a lot of code, it seemed like it might be a good idea to isolate the functional code from the scaffolding, thereby setting up a convention for other services, some of which may involve more complex logic. I finally ended up with another file that looked like the following:
[pastacode lang=”python” manual=”def%20get_words(html)%3A%0A%20%20try%3A%0A%20%20%20%20%20%20%20%20soup%20%3D%20BeautifulSoup(html%2C%20%22html5lib%22)%23%20(1)%0A%20%20%20%20%20%20%20%20job_desc%20%3D%20soup.find(%22div%22%2C%20id%3D%22jobdescSec%22)%23%20(2)%0A%20%20%20%20%20%20%20%20if%20not%20job_desc%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20return%20None%0A%20%20%20%20%20%20%20%20else%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20job_text%20%3D%20job_desc.stripped_strings%23%20(3)%0A%20%20%20%20%20%20%20%20%20%20%20%20words%20%3D%20’%20′.join(job_text)%23%20(4)%0A%20%20%20%20%20%20%20%20%20%20%20%20json_response%20%3D%20json.dumps(%7B’data’%3Awords%7D)%23%20(5)%0A%20%20%20%20%20%20%20%20%20%20%20%20return%20json_response%0A%20%20%20%20except%20Exception%20as%20e%3A%0A%20%20%20%20%20%20%20%20return%20None%0Aclass%20WordsHandler(tornado.web.RequestHandler)%3A%0A%20%20%20%20def%20get(self)%3A%0A%20%20%20%20%20%20%20%20self.write(%22Breaking%20with%20all%20conventions%2C%20this%20API%20does%20not%20support%20GET%22)%0A%20%20%20%20def%20post(self)%3A%0A%20%20%20%20%20%20%20%20html%20%3D%20self.get_argument(%22html%22)%0A%20%20%20%20%20%20%20%20json_response%20%3D%20get_words(html)%0A%20%20%20%20%20%20%20%20if%20not%20json_response%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20self.set_status(HTTP_STATUS_NO_CONTENT%2C%20’There%20was%20no%20content’)%0A%20%20%20%20%20%20%20%20else%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20self.write(json_response)%0A%20%20%20%20%20%20%20%20%20%20%20%20self.set_header(‘Content-Type’%2C%20’application%2Fjson’)%0A%20%20%20%20%20%20%20%20%20%20%20%20self.set_status(HTTP_STATUS_OK)” message=”” highlight=”” provider=”manual”/]The lines numbered (1)-(5) are exactly the same as in the original version. They’re isolated in a function called get_words that can be independently unit-tested without running Tornado. In the handler itself, there are lines for returning a status code and setting other HTTP headers. More can be added if it becomes necessary. The code that sets up Tornado and launches it stayed in the original file.
The other two services, for fetching the contents of a page and generating a word cloud, are structured the same way.
Here’s the Fetch From URL code, without all the imports.
In case you’re wondering, the response.content is Base64 encoded because it comes out as a byte array, which is not directly JSON serializable.
Here’s the Make Wordcloud Image code. It uses the word_cloud project.
With the microservices pretty much out of the way, I just needed to create the view controller that would take URLs submitted by the user, build the response using these microservices, and send a response to the user. I used Django for building the app server because I just wanted to focus on the functionality I needed, while it could take care of most other web-app housekeeping.
My initial win looked something like this (some of this early code is pulled from the file history):
This initial version of the view class assumed the user had only entered one URL. The services were all hard-coded into the controller (more on that later). The response from one microservice is plugged directly into the next, and in this case, it works.
The Django form class is really simple, it has just two lines:
And the wordcloud.html template isn’t that complicated either:
I had now written enough code to take a URL submitted by the user and display a word cloud back to them. Time to put it to the test. I launched the three microservices as background python tasks: $ python microservices/cloud_creator/api_server.py &
$ python microservices/fetch_url/api_server.py &
$ python microservices/dice_scraper /api_server.py &
I started the Django server and the page http://localhost:8000/cloudfun in a browser, pasted a URL taken from Dice.com in the text area, and clicked on OK. It worked! I saw the following image in the browser.
You can try it out for real here. From this limited exercise, I see the appeal of microservices. It forces you to think about how to break down a larger functional goal into discrete tasks, what’s known as separation of concerns. I can see how if you’re building an e-commerce page or getting search results, you’d want to kick off a dozen asynchronous sub-requests that all returned various chunks of information that could be assembled into one page. In my mind I picture an F1 car at a pit stop, a whole bunch of workers pounce on it and quickly get it back in shape to continue on.
Putting all this together took an afternoon, but I could see that there were some anti-patterns that needed to be fixed. The biggest is that the locations of the services were hard-coded into the view controller. That’s the subject of the next article.
Of course, separation of concerns has been a software engineering preoccupation for a long time. Object-oriented programming was supposed to do that as well. Then came CORBA, which took a team of ten IBM engineers 6 months to make it work (I kid). Next came web services, and SOAP. When I worked for France Telecom in 2001, I did an evaluation of SOAP and it’s promise of interoperability. I got a Java service to talk to a .NET service. It was early days and I remember it was hell. I wrote (en français, même) a report of my findings. I think I said that it seemed that perhaps it could be a game changer like some people were predicting. People were spinning visions of a proliferation of Web Services, automatically discoverable, with service contracts written in WSDL. There would be flight-booking web services, financial services, and if one service were down the system could look up another. Heady stuff!
Fast forward 15 years and here we are, with microservices. Except, from what I have seen, their role is now limited to being a common service provider for clients within the organization, and not really to clients anywhere on the open Internet. Maybe that’ll happen too.
The full code for this app is available on GitHub.