Second Round of Improvements to the Reddit Application

1. Overview

Let’s continue our ongoing Reddit web app case study with a new round of improvements, with the goal of making the application more user friendly and easier to use.

2. Scheduled Posts Pagination

First – let’s list the scheduled posts with pagination, to make the whole thing easier to look at and understand.

2.1. The Paginated Operations

We’ll use Spring Data to generate the operation we need, making good use of the Pageable interface to retrieve user’s scheduled posts:

public interface PostRepository extends JpaRepository<Post, Long> {
    Page<Post> findByUser(User user, Pageable pageable);
}

And here is our controller method getScheduledPosts():

private static final int PAGE_SIZE = 10;

@RequestMapping("/scheduledPosts")
@ResponseBody
public List<Post> getScheduledPosts(
  @RequestParam(value = "page", required = false) int page) {
    User user = getCurrentUser();
    Page<Post> posts = 
      postReopsitory.findByUser(user, new PageRequest(page, PAGE_SIZE));
    
    return posts.getContent();
}

2.2. Display Paginated Posts

Now – let’s implement a simple pagination control in front end:

<table>
<thead><tr><th>Post title</th></thead>
</table>
<br/>
<button id="prev" onclick="loadPrev()">Previous</button> 
<button id="next" onclick="loadNext()">Next</button>

And here is how we load the pages with plain jQuery:

$(function(){ 
    loadPage(0); 
}); 

var currentPage = 0;
function loadNext(){ 
    loadPage(currentPage+1);
} 

function loadPrev(){ 
    loadPage(currentPage-1); 
}

function loadPage(page){
    currentPage = page;
    $('table').children().not(':first').remove();
    $.get("api/scheduledPosts?page="+page, function(data){
        $.each(data, function( index, post ) {
            $('.table').append('<tr><td>'+post.title+'</td><td></tr>');
        });
    });
}

As we move forward, this manual table will get quickly replaced with a more mature table plugin, but for now, this works just fine.

3. Show the Login Page to Non Logged in Users

When a user accesses the root, they should get different pages if they’re logged in or not.

If the user is logged in, they should see their homepage/dashboard. If they’re not logged in – they should see the login page:

@RequestMapping("/")
public String homePage() {
    if (SecurityContextHolder.getContext().getAuthentication() != null) {
        return "home";
    }
    return "index";
}

4. Advanced Options for Post Resubmit

Removing and resubmitting posts in Reddit is a useful, highly effective functionality. However, we want to be careful with it and have full control over when we should and when we shouldn’t do it.

For example – we might not want to remove a post if it already has comments. At the end of the day, comments are engagement and we want to respect the platform and the people commenting on the post.

So – that’s the first small yet highly useful feature we’ll add – a new option that’s going to allow us to only remove a post if it doesn’t have comments on it.

Another very interesting question to answer is – if the post is resubmitted for however many times but still doesn’t get the traction it needs – do we leave it on after the last attempt or not? Well, like all interesting questions, the answer here is – “it depends”. If it’s a normal post, we might just call it a day and leave it up. However, if it’s a super-important post and we really really want to make sure it gets some traction, we might delete it at the end.

So this is the second small but very handy feature we’ll build here.

Finally – what about controversial posts? A post can have 2 votes on reddit because there it has to positive votes, or because it has 100 positive and 98 negative votes. The first option means it’s not getting traction, while the second means that it’s getting a lot of traction and that the voting is split.

So – this is the third small feature we’re going to add – a new option to take this upvote to downvote ratio into account when determining if we need to remove the post or not.

4.1. The Post Entity

First, we need to modify our Post entity:

@Entity
public class Post {
    ...
    private int minUpvoteRatio;
    private boolean keepIfHasComments;
    private boolean deleteAfterLastAttempt;
}

Here are the 3 fields:

• minUpvoteRatio: The minimum upvote ratio the user wants his post to reach – the upvote ratio represents how % of total votes ara upvotes [max = 100, min =0]
• keepIfHasComments: Determine whether the user want to keep his post if it has comments despite not reaching required score.
• deleteAfterLastAttempt: Determine whether the user want to delete the post after the final attempt ends without reaching required score.

4.2. The Scheduler

Let’s now integrate these interesting new options into the scheduler:

@Scheduled(fixedRate = 3 * 60 * 1000)
public void checkAndDeleteAll() {
    List<Post> submitted = 
      postReopsitory.findByRedditIDNotNullAndNoOfAttemptsAndDeleteAfterLastAttemptTrue(0);
    
    for (Post post : submitted) {
        checkAndDelete(post);
    }
}

On the the more interesting part – the actual logic of checkAndDelete():

private void checkAndDelete(Post post) {
    if (didIntervalPass(post.getSubmissionDate(), post.getTimeInterval())) {
        if (didPostGoalFail(post)) {
            deletePost(post.getRedditID());
            post.setSubmissionResponse("Consumed Attempts without reaching score");
            post.setRedditID(null);
            postReopsitory.save(post);
        } else {
            post.setNoOfAttempts(0);
            post.setRedditID(null);
            postReopsitory.save(post);
        }
    }
}

And here’s the didPostGoalFail() implementation – checking if the post failed to reach the predefined goal/score:

private boolean didPostGoalFail(Post post) {
    PostScores postScores = getPostScores(post);
    int score = postScores.getScore();
    int upvoteRatio = postScores.getUpvoteRatio();
    int noOfComments = postScores.getNoOfComments();
    return (((score < post.getMinScoreRequired()) || 
             (upvoteRatio < post.getMinUpvoteRatio())) && 
           !((noOfComments > 0) && post.isKeepIfHasComments()));
}

We also need to modify the logic that retrieves the Post information from Reddit – to make sure we gather more data:

public PostScores getPostScores(Post post) {
    JsonNode node = restTemplate.getForObject(
      "http://www.reddit.com/r/" + post.getSubreddit() + 
      "/comments/" + post.getRedditID() + ".json", JsonNode.class);
    PostScores postScores = new PostScores();

    node = node.get(0).get("data").get("children").get(0).get("data");
    postScores.setScore(node.get("score").asInt());
    
    double ratio = node.get("upvote_ratio").asDouble();
    postScores.setUpvoteRatio((int) (ratio * 100));
    
    postScores.setNoOfComments(node.get("num_comments").asInt());
    
    return postScores;
}

We’re using a simple value object to represent the scores as we’re extracting them from the Reddit API:

public class PostScores {
    private int score;
    private int upvoteRatio;
    private int noOfComments;
}

Finally, we need to modify checkAndReSubmit() to set the successfully resubmitted post’s redditID to null:

private void checkAndReSubmit(Post post) {
    if (didIntervalPass(post.getSubmissionDate(), post.getTimeInterval())) {
        if (didPostGoalFail(post)) {
            deletePost(post.getRedditID());
            resetPost(post);
        } else {
            post.setNoOfAttempts(0);
            post.setRedditID(null);
            postReopsitory.save(post);
        }
    }
}

Note that:

• checkAndDeleteAll(): runs every 3 minutes through to see if any posts have consumed their attempts and can be deleted
• getPostScores(): return post’s {score, upvote ratio, number of comments}

4.3. Modify the Schedule Page

We need to add the new modifications to our schedulePostForm.html:

<input type="number" name="minUpvoteRatio"/>
<input type="checkbox" name="keepIfHasComments" value="true"/>
<input type="checkbox" name="deleteAfterLastAttempt" value="true"/>

5. Email Important Logs

Next, we’ll implement a quick but highly useful setting in our logback configuration – emailing of important logs (ERROR level). This is of course quite handy to easily track errors early on in the lifecycle of an application.

First, we’ll add a few required dependencies to our pom.xml:

<dependency>
    <groupId>jakarta.mail</groupId>
    <artifactId>jakarta.mail-api</artifactId>
    <version>2.1.2</version>
</dependency>

Then, we will add a SMTPAppender to our logback.xml:

<configuration>

    <appender name="STDOUT" ...

    <appender name="EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>ERROR</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>

        <smtpHost>smtp.example.com</smtpHost>
        <to>[email protected]</to>
        <from>[email protected]</from>
        <username>[email protected]</username>
        <password>password</password>
        <subject>%logger{20} - %m</subject>
        <layout class="ch.qos.logback.classic.html.HTMLLayout"/>
    </appender>

    <root level="INFO">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="EMAIL" />
    </root>

</configuration>

And that’s about it – now, the deployed application will email any problem as it happens.

6. Cache Subreddits

Turns out, auto-completing subreddits expensive. Every time a user starts typing in a subreddit when scheduling a post – we need to hit the Reddit API to get these subreddits and show the user some suggestions. Not ideal.

Instead of calling the Reddit API – we’ll simply cache the popular subreddits and use them to autocomplete.

6.1. Retrieve Subreddits

First, let’s retrieve the most popular subreddits and save them to a plain file:

public void getAllSubreddits() {
    JsonNode node;
    String srAfter = "";
    FileWriter writer = null;
    try {
        writer = new FileWriter("src/main/resources/subreddits.csv");
        for (int i = 0; i < 20; i++) {
            node = restTemplate.getForObject(
              "http://www.reddit.com/" + "subreddits/popular.json?limit=100&after=" + srAfter, 
              JsonNode.class);
            srAfter = node.get("data").get("after").asText();
            node = node.get("data").get("children");
            for (JsonNode child : node) {
                writer.append(child.get("data").get("display_name").asText() + ",");
            }
            try {
                Thread.sleep(3000);
            } catch (InterruptedException e) {
                logger.error("Error while getting subreddits", e);
            }
        }
        writer.close();
    } catch (Exception e) {
        logger.error("Error while getting subreddits", e);
    }
}

Is this a mature implementation? No. Do we need anything more? No we don’t. We need to move on.

6.2. Subbreddit Autocomplete

Next, let’s make sure the subreddits are loaded into memory on application startup – by having the service implement InitializingBean:

public void afterPropertiesSet() {
    loadSubreddits();
}
private void loadSubreddits() {
    subreddits = new ArrayList<String>();
    try {
        Resource resource = new ClassPathResource("subreddits.csv");
        Scanner scanner = new Scanner(resource.getFile());
        scanner.useDelimiter(",");
        while (scanner.hasNext()) {
            subreddits.add(scanner.next());
        }
        scanner.close();
    } catch (IOException e) {
        logger.error("error while loading subreddits", e);
    }
}

Now that the subreddit data is all loaded up into memory, we can search over the subreddits without hitting the Reddit API:

public List<String> searchSubreddit(String query) {
    return subreddits.stream().
      filter(sr -> sr.startsWith(query)).
      limit(9).
      collect(Collectors.toList());
}

The API exposing the subreddit suggestions of course remains the same:

@RequestMapping(value = "/subredditAutoComplete")
@ResponseBody
public List<String> subredditAutoComplete(@RequestParam("term") String term) {
    return service.searchSubreddit(term);
}

7. Metrics

Finally – we’ll integrate some simple metrics into the application. For a lot more on building out these kinds of metrics, I wrote about them in some detail here.

7.1. Servlet Filter

Here the simple MetricFilter:

@Component
public class MetricFilter implements Filter {

    @Autowired
    private IMetricService metricService;

    @Override
    public void doFilter(
      ServletRequest request, ServletResponse response, FilterChain chain) 
      throws IOException, ServletException {
        HttpServletRequest httpRequest = ((HttpServletRequest) request);
        String req = httpRequest.getMethod() + " " + httpRequest.getRequestURI();

        chain.doFilter(request, response);

        int status = ((HttpServletResponse) response).getStatus();
        metricService.increaseCount(req, status);
    }
}

We also need to add it in our ServletInitializer:

@Override
public void onStartup(ServletContext servletContext) throws ServletException {
    super.onStartup(servletContext);
    servletContext.addListener(new SessionListener());
    registerProxyFilter(servletContext, "oauth2ClientContextFilter");
    registerProxyFilter(servletContext, "springSecurityFilterChain");
    registerProxyFilter(servletContext, "metricFilter");
}

7.2. Metric Service

And here is our MetricService:

public interface IMetricService {
    void increaseCount(String request, int status);
    
    Map getFullMetric();
    Map getStatusMetric();
    
    Object[][] getGraphData();
}

7.3. Metric Controller

And her’s the basic controller responsible with exposing these metrics over HTTP:

@Controller
public class MetricController {
    
    @Autowired
    private IMetricService metricService;

    // 
    
    @RequestMapping(value = "/metric", method = RequestMethod.GET)
    @ResponseBody
    public Map getMetric() {
        return metricService.getFullMetric();
    }

    @RequestMapping(value = "/status-metric", method = RequestMethod.GET)
    @ResponseBody
    public Map getStatusMetric() {
        return metricService.getStatusMetric();
    }

    @RequestMapping(value = "/metric-graph-data", method = RequestMethod.GET)
    @ResponseBody
    public Object[][] getMetricGraphData() {
        Object[][] result = metricService.getGraphData();
        for (int i = 1; i < result[0].length; i++) {
            result[0][i] = result[0][i].toString();
        }
        return result;
    }
}

8. Conclusion

This case study is growing nicely. The app actually started as a simple tutorial on doing OAuth with the Reddit API; now, it’s evolving into a useful tool for the Reddit power-user – especially around the scheduling and re-submitting options.

Finally, since I’ve been using it, it looks like my own submissions to Reddit are generally picking up a lot more steam, so that’s always good to see.