1. Overview
Sometimes, we would like to remove all HTML tags and extract the text from an HTML document string.
The problem looks pretty straightforward. However, depending on the requirements, it can have different variants.
In this tutorial, we’ll discuss how to do that using Java.
2. Using Regex
Since we’ve already got the HTML as a String variable, we need to do a kind of text manipulation.
When facing text manipulation problems, regular expressions (Regex) could be the first idea coming up.
Removing HTML tags from a string won’t be a challenge for Regex since no matter the start or the end HTML elements, they follow the pattern “< … >”.
If we translate it into Regex, it would be “<[^>]*>” or “<.*?>”.
We should note that Regex does greedy matching by default. That is, the Regex “<.*>” won’t work for our problem since we want to match from ‘<‘ until the next ‘>‘ instead of the last ‘>‘ in a line.
Now, let’s test if it can remove tags from an HTML source.
Before we test removing HTML tags, first let’s create an HTML example, say example1.html:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>This is the page title</title>
</head>
<body>
<p>
If the application X doesn't start, the possible causes could be:<br/>
1. <a href="maven.com">Maven</a> is not installed.<br/>
2. Not enough disk space.<br/>
3. Not enough memory.
</p>
</body>
</html>
Now, let’s write a test and use String.replaceAll() to remove HTML tags:
String html = ... // load example1.html
String result = html.replaceAll("<[^>]*>", "");
System.out.println(result);
If we run the test method, we see the result:
This is the page title
If the application X doesn't start, the possible causes could be:
1. Maven is not installed.
2. Not enough disk space.
3. Not enough memory.
The output looks pretty good. This is because all HTML tags have been removed.
It preserves whitespaces from the stripped HTML. But we can easily remove or skip those empty lines or whitespaces when we process the extracted text. So far, so good.
As we’ve just seen, using Regex to remove HTML tags is pretty straightforward. However, this approach may have problems since we cannot predict what HTML source we’ll get.
For example, an HTML document may have <script> or <style> tags, and we may not want to have their content in the result.
Further, the text in the <script>, <style>, or even the <body> tags could contain “<” or “>” characters. If this is the case, our Regex approach may fail.
Now, let’s see another HTML example, say example2.html:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>This is the page title</title>
</head>
<script>
// some interesting script functions
</script>
<body>
<p>
If the application X doesn't start, the possible causes could be:<br/>
1. <a
id="link"
href="http://maven.apache.org/">
Maven
</a> is not installed.<br/>
2. Not enough (<1G) disk space.<br/>
3. Not enough (<64MB) memory.<br/>
</p>
</body>
</html>
This time, we have a <script> tag and “<” characters in the <body> tag.
If we use the same method on example2.html, we’ll get (empty lines have been removed):
This is the page title
// some interesting script functions
If the application X doesn't start, the possible causes could be:
1.
Maven
is not installed.
2. Not enough (
3. Not enough (
Apparently, we’ve lost some text due to the “<” characters.
Therefore, using Regex to process XML or HTML is fragile. Instead, we can choose an HTML parser to do the job.
Next, we’ll address a few easy-to-use HTML libraries to extract text.
3. Using Jsoup
Jsoup is a popular HTML parser. To extract text from an HTML document, we can simply call Jsoup.parse(htmlString).text().
First, we need to add the Jsoup library to the classpath. For example, let’s say we’re using Maven to manage project dependencies:
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.17.2</version>
</dependency>
Now, let’s test it with our example2.html:
String html = ... // load example2.html
System.out.println(Jsoup.parse(html).text());
If we give the method a run, it prints:
This is the page title If the application X doesn't start, the possible causes could be: 1. Maven is not installed. 2. Not enough (<1G) disk space. 3. Not enough (<64MB) memory.
As the output shows, Jsoup has successfully extracted texts from the HTML document. Also, the text in the <script> element has been ignored.
Additionally, by default, Jsoup will remove all text formatting and whitespaces, such as line breaks.
However, if it’s required, we can also ask Jsoup to preserve the line breaks.
4. Using HTMLCleaner
HTMLCleaner is another HTML parser. Its goal is to make “ill-formed and dirty” HTML from the Web suitable for further processing.
First, let’s add the HTMLCleaner dependency in our pom.xml:
<dependency>
<groupId>net.sourceforge.htmlcleaner</groupId>
<artifactId>htmlcleaner</artifactId>
<version>2.25</version>
</dependency>
We can set various options to control HTMLCleaner’s parsing behavior.
Here, as an example, let’s tell HTMLCleaner to skip the <script> element when parsing example2.html:
String html = ... // load example2.html
CleanerProperties props = new CleanerProperties();
props.setPruneTags("script");
String result = new HtmlCleaner(props).clean(html).getText().toString();
System.out.println(result);
HTMLCleaner will produce this output if we run the test:
This is the page title
If the application X doesn't start, the possible causes could be:
1.
Maven
is not installed.
2. Not enough (<1G) disk space.
3. Not enough (<64MB) memory.
As we can see, the content in the <script> element has been ignored.
Also, it converts <br/> tags into line breaks in the extracted text. This can be helpful if the format is significant.
On the other hand, HTMLCleaner preserves whitespace from the stripped HTML source. So, for example, the text “1. Maven is not installed” is broken into three lines.
5. Using Jericho
At last, we’ll see another HTML parser – Jericho. It has a nice feature: rendering HTML markup with simple text formatting. We’ll see it in action later.
As usual, let’s first add the Jericho dependency in the pom.xml:
<dependency>
<groupId>net.htmlparser.jericho</groupId>
<artifactId>jericho-html</artifactId>
<version>3.4</version>
</dependency>
In our example2.html, we have a hyperlink “Maven (http://maven.apache.org/)“. Now, let’s say we would like to have both the link URL and link text in the result.
To do that, we can create a Renderer object and use the includeHyperlinkURLs option:
String html = ... // load example2.html
Source htmlSource = new Source(html);
Segment segment = new Segment(htmlSource, 0, htmlSource.length());
Renderer htmlRender = new Renderer(segment).setIncludeHyperlinkURLs(true);
System.out.println(htmlRender);
Next, let’s execute the test and check the output:
If the application X doesn't start, the possible causes could be:
1. Maven <http://maven.apache.org/> is not installed.
2. Not enough (<1G) disk space.
3. Not enough (<64MB) memory.
As we can see in the result above, the text has been pretty-formatted. Also, the text in the <title> element is ignored by default.
The link URL is included as well. Apart from rendering links (<a>), Jericho supports rendering other HTML tags, for example <hr/>, <br/>, bullet-list (<ul> and <li>), and so on.
6. Conclusion
In this article, we’ve addressed different ways to remove HTML tags and extract HTML text.
We should note that it’s not a good practice to use Regex to process XML/HTML.
As always, the complete source code for this article can be found over on GitHub.
