How to Utilize Web Scraping API using Vert.x and Jsoup in Kotlin and Monetize it?

Adding Jsoup in Graddle Project

As is customary, we begin by creating our Gradle project. This time, we’ll include the Jsoup library as a dependency.

About Jsoup

Jsoup is a Java library for working with real-world HTML, according to its website — https://jsoup.org. It uses the finest of HTML5 DOM techniques and CSS selectors to create a highly easy API for requesting URLs, extracting, and modifying data.

Then there’s our finished product.

build.gradle will seems to be.

buildscript { ext.kotlin_version = '1.6.10' ext.vertx_version = '4.2.5' repositories { mavenCentral() } dependencies { classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version" } group = 'WebScraper' version = '1.0.0' } plugins { id 'org.jetbrains.kotlin.jvm' version '1.6.10' id 'java' } repositories { mavenCentral() } sourceCompatibility = JavaVersion.VERSION_11 targetCompatibility = JavaVersion.VERSION_11 dependencies { // Kotlin implementation 'org.jetbrains.kotlin:kotlin-stdlib' // Vertx Core implementation "io.vertx:vertx-core:$vertx_version" implementation "io.vertx:vertx-lang-kotlin:$vertx_version" // Vertx Web implementation "io.vertx:vertx-web:$vertx_version" // Vertx Rxjava implementation "io.vertx:vertx-rx-java3:$vertx_version" implementation "io.vertx:vertx-rx-java3-gen:$vertx_version" // Jsoup implementation 'org.jsoup:jsoup:1.14.3' } jar { duplicatesStrategy = DuplicatesStrategy.EXCLUDE manifest { attributes 'Implementation-Title': group attributes 'Implementation-Version': archiveVersion attributes 'Main-Class': 'AppKt' } from { configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) } } } task cleanAndJar { group = 'build' description = 'Clean and create jar' dependsOn clean dependsOn jar } compileKotlin { kotlinOptions { jvmTarget = JavaVersion.VERSION_11 } }

Source File Structures

This is how our source files will be organized.

- src | - main | | - java | | - kotlin | | | - verticle | | | | - HttpServerVerticle.kt | | | | - MainVerticle.kt | | | - App.kt

The main function that will operate the service is App.kt.

Our REST API code will be written in the class HttpServerVerticle.kt.

The primary Vert.x instance that will execute the HttpServerVerticle class is MainVerticle.kt.

Executing Web Scraping Services

In our HttpServerVerticle class, we’ll use Jsoup to construct the web scraper.

package verticle import io.vertx.core.Promise import io.vertx.core.impl.logging.LoggerFactory import io.vertx.core.json.JsonArray import io.vertx.core.json.JsonObject import io.vertx.rxjava3.core.AbstractVerticle import io.vertx.rxjava3.ext.web.Router import io.vertx.rxjava3.ext.web.RoutingContext import org.jsoup.Jsoup import java.text.SimpleDateFormat class HttpServerVerticle : AbstractVerticle() { private val logger = LoggerFactory.getLogger(HttpServerVerticle::class.java) override fun start(promise: Promise) { val router = Router.router(vertx).apply { get("/web-scraper/scrape").handler(this@HttpServerVerticle::scrape) } vertx .createHttpServer() .requestHandler(router) .rxListen(8282) .subscribe( { promise.complete() }, { failure -> promise.fail(failure.cause) }) } private fun scrape(context: RoutingContext) { val responses = JsonArray() responses.addAll(scrapeCnbc()) responses.addAll(scrapeGeekWire()) responses.addAll(scrapeTechStartups()) context.response().statusCode = 200 context.response().putHeader("Content-Type", "application/json") context.response().end(responses.encode()) } private fun scrapeCnbc(): JsonArray { val responses = JsonArray() val document = Jsoup.connect("https://www.cnbc.com/startups/").get() val pageRowElements = document.select("div.PageBuilder-pageRow") for (pageRowElement in pageRowElements) { try { val moduleHeaderTitle = pageRowElement.selectFirst("h2.ModuleHeader-title") if (moduleHeaderTitle?.text().equals("More In Start-ups")) { val elements = pageRowElement.select("div.Card-standardBreakerCard") for (element in elements) { val title = element.selectFirst("a.Card-title") val datePost = element.selectFirst("span.Card-time") val dateFormat = SimpleDateFormat("EEE, MMM d yyyy") val dateString = datePost?.text()?.replace("st|nd|rd|th".toRegex(), "") val date = dateFormat.parse(dateString) responses.add(JsonObject().apply { put("title", title?.text()) put("link", title?.attr("href")) put("timestamp", date.time) }) } } } catch (e: Exception) { logger.error(e.message) } } return responses } private fun scrapeGeekWire(): JsonArray { val responses = JsonArray() val document = Jsoup.connect("https://www.geekwire.com/startups/").get() val articleElements = document.select("article.type-post") for (articleElement in articleElements) { try { val title = articleElement.selectFirst("h2.entry-title > a") val datePost = articleElement.selectFirst("time.published") val dateFormat = SimpleDateFormat("MMM d, yyyy") val dateString = datePost?.text() val date = dateFormat.parse(dateString) responses.add(JsonObject().apply { put("title", title?.text()) put("link", title?.attr("href")) put("timestamp", date.time) }) } catch (e: Exception) { logger.error(e.message) } } return responses } private fun scrapeTechStartups(): JsonArray { val responses = JsonArray() val document = Jsoup.connect("https://techstartups.com/").get() val postElements = document.select("div.post.type-post") for (postElement in postElements) { try { val title = postElement.selectFirst("h5 > a") val datePost = postElement.selectFirst("span.post_info_date") val dateFormat = SimpleDateFormat("MMM d, yyyy") val dateString = datePost?.text()?.replace("Posted On ", "") val date = dateFormat.parse(dateString) responses.add(JsonObject().apply { put("title", title?.text()) put("link", title?.attr("href")) put("timestamp", date.time) }) } catch (e: Exception) { logger.error(e.message) } } return responses } }
val document = Jsoup.connect("https://www.geekwire.com/startups/").get()

The web address is sent to the connect() function. The entire HTML DOM is then obtained using the get() function.

Then, on line 83, we’ll use css selector to build a query.

val articleElements = document.select("article.type-post")

Selector in the JQuery library in Javascript is comparable to this.

The elements that match the selection will be found by the select () function. Yes, an array is returned. So, in the following line, we’ll iterate it.

The following code may be found on line 87.

val title = articleElement.selectFirst("h2.entry-title > a")

Only one element that meets the selection is returned using the selectFirst() function. As a result, we must verify that the selector exists only within one of the articleElements.

The data that has been scraped is then stored in a JsonArray object on line 94.

responses.add(JsonObject().apply { put("title", title?.text()) put("link", title?.attr("href")) put("timestamp", date.time) })

MainVerticle.kt & App.kt

These are the Vert.x standard classes and methods for deploying our service.

package verticle import io.vertx.core.Promise import io.vertx.rxjava3.core.AbstractVerticle import io.vertx.rxjava3.core.RxHelper class MainVerticle : AbstractVerticle() { override fun start(promise: Promise) { RxHelper.deployVerticle(vertx, HttpServerVerticle()) .subscribe( { promise.complete() }, promise::fail) } } import io.vertx.core.Launcher import verticle.MainVerticle fun main() { Launcher.executeCommand("run", MainVerticle::class.java.name) }

Merge and Execute the Web Scraper

After you’ve finished scripting, build it, and execute the cleanAndJar Gradle process.

Simply run: if you already have your jar file.

$ java -jar build/libs/WebScraper-1.0.0.jar

It’s time to put our web scraper API to the test. Postman is what you use to test the outcome of what we’ve constructed.

You can contact iWeb Scraping, for any web scraping service requirements and request for a quote!

Originally published at https://www.iwebscraping.com.