This article will cover a quick side-project – a bot to automatically tweet Top Questions from the various Q&A StackExchange sites, such as StackOverflow, ServerFault, SuperUser, etc. We will build a simple client for the StackExchange API and then we’ll setup the interaction with the Twitter API using Spring Social – this first part will focus on the StackExchange Client only.
The initial purpose of this implementation is not to be a full fledged Client for the entire StackExchange API – that would be outside the scope of this project. The only reason the Client exists is that I couldn’t find one that would work against the 2.x version of the official API.
Note that this tutorial uses the Spring Social API, which is now deprecated. As a result, the supporting code shown here is no longer maintained.
1. The Maven Dependencies
To consume the StackExchange REST API, we will need very few dependencies – essentially just an HTTP client – the Apache HttpClient will do just fine for this purpose:
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3.3</version>
</dependency>
The Spring RestTemplate could also have been used to interact with the HTTP API, but that would have introduced quite a lot of other Spring related dependencies into the project, dependencies which are not strictly necessary, so HttpClient will keep things light and simple.
2. The Questions Client
The goal of this Client is to consume the /questions REST Service that StackExchange publishes, not to provide a general purpose client for the entire StackExchange APIs – so for the purpose of this article we will only look at that.
The actual HTTP communication using HTTPClient is relatively straightforward:
public String questions(int min, String questionsUri) {
HttpGet request = null;
try {
request = new HttpGet(questionsUri);
HttpResponse httpResponse = client.execute(request);
InputStream entityContentStream = httpResponse.getEntity().getContent();
return IOUtils.toString(entityContentStream, Charset.forName("utf-8"));
} catch (IOException ex) {
throw new IllegalStateException(ex);
} finally {
if (request != null) {
request.releaseConnection();
}
}
}
This simple interaction is perfectly adequate for obtaining the questions raw JSON that the API publishes – the next step will be processing that JSON.
There is one relevant detail here – and that is the questionsUri method argument – there are multiple StackExchange APIs that can publish questions (as the official documentation suggests), and this method needs to be flexible enough to consume all of them. It can consume for example the simplest API that returns questions by setting questionUri set to https://api.stackexchange.com/2.1/questions?site=stackoverflow or it may consume the tag based https://api.stackexchange.com/2.1/tags/{tags}/faq?site=stackoverflow API instead, depending on what the client needs.
A request to the StackExchange API is fully configured with query parameters, even for the more complex advanced search queries – there is no body being send. To construct the questionsUri, we’ll build a basic fluent RequestBuilder class that will make use of the URIBuilder from the HttpClient library. This will take care of correctly encoding the URI and generally making sure that the end result is valid:
public class RequestBuilder {
private Map<String, Object> parameters = new HashMap<>();
public RequestBuilder add(String paramName, Object paramValue) {
this.parameters.put(paramName, paramValue);
return this;
}
public String build() {
URIBuilder uriBuilder = new URIBuilder();
for (Entry<String, Object> param : this.parameters.entrySet()) {
uriBuilder.addParameter(param.getKey(), param.getValue().toString());
}
return uriBuilder.toString();
}
}
So now, to construct a valid URI for the StackExchange API:
String params = new RequestBuilder().
add("order", "desc").add("sort", "votes").add("min", min).add("site", site).build();
return "https://api.stackexchange.com/2.1/questions" + params;
3. Testing the Client
The Client will output raw JSON, but to test that, we will need a JSON processing library, specifically Jackson 2:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.3.3</version>
<scope>test</scope>
</dependency>
The tests we’ll look at will interact with the actual StackExchange API:
@Test
public void whenRequestIsPerformed_thenSuccess()
throws ClientProtocolException, IOException {
HttpResponse response = questionsApi.questionsAsResponse(50, Site.serverfault);
assertThat(response.getStatusLine().getStatusCode(), equalTo(200));
}
@Test
public void whenRequestIsPerformed_thenOutputIsJson()
throws ClientProtocolException, IOException {
HttpResponse response = questionsApi.questionsAsResponse(50, Site.serverfault);
String contentType = httpResponse.getHeaders(HttpHeaders.CONTENT_TYPE)[0].getValue();
assertThat(contentType, containsString("application/json"));
}
@Test
public void whenParsingOutputFromQuestionsApi_thenOutputContainsSomeQuestions()
throws ClientProtocolException, IOException {
String questionsAsJson = questionsApi.questions(50, Site.serverfault);
JsonNode rootNode = new ObjectMapper().readTree(questionsAsJson);
ArrayNode questionsArray = (ArrayNode) rootNode.get("items");
assertThat(questionsArray.size(), greaterThan(20));
}
The first test has verified that the response provided by the API was indeed a 200 OK, so the GET request to retrieve the questions was actually successful. After that basic condition is ensured, we moved on to the Representation – as specified by the Content-Type HTTP header – that needs to be JSON. Next, we actually parse the JSON and verify that there are actually Questions in that output – that parsing logic itself is low level and simple, which is enough for the purpose of the test.
Note that these requests count towards your rate limits specified by the API – for that reason, the Live tests are excluded from the standard Maven build:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<excludes>
<exclude>**/*LiveTest.java</exclude>
</excludes>
</configuration>
</plugin>
4. The Next Step
The current Client is only focused on a single type of Resource from the many available types published by the StackExchange APIs. This is because it’s initial purpose is limited – it only needs to enable a user to consume Questions from the various sites in the StackExchange portfolio. Consequently, the Client can be improved beyond the scope of this initial usecase to be able to consume the other types of the API.
The implementation is also very much raw – after consuming the Questions REST Service, it simply returns JSON output as a String – not any kind of Questions model out of that output. So, a potential next step would be to unmarshall this JSON into a proper domain DTO and return that back instead of raw JSON.
5. Conclusion
The purpose of this article was to show how to start building an integration with the StackExchange API, or really an HTTP based API out there. It covered how to write integration tests against the live API and make sure the end to end interaction actually works.
The second part of this article will show how to interact with the Twitter API by using the Spring Social library, and how to use the StackExchange Client we built here to tweet questions on a new twitter account.
I have already set up a few twitter accounts that are now tweeting the 2 Top Questions per day, for various disciplines:
- SpringTip – Two of the best Spring questions from StackOverflow each day
- JavaTopSO – Two of the best Java questions from StackOverflow each day
- AskUbuntuBest – Two of the best questions from AskUbuntu each day
- BestBash – Two of the best Bash questions from all StackExchange sites each day
- ServerFaultBest – Two of the best questions from ServerFault each day
The full implementation of this StackExchange Client is on github.
